S3 — Use using block to get the stream

When you’re using the Ama­zon S3 client, have you come across the occa­sion­al excep­tion that says some­thing like one of these excep­tion mes­sages:

The request was abort­ed: The con­nec­tion was closed unex­pect­ed­ly”

Unable to read data from the trans­port con­nec­tion: A block­ing oper­a­tion was inter­rupt­ed by a call to WSACancel­Block­ing­Call”

Unable to read data from the trans­port con­nec­tion: An estab­lished con­nec­tion was abort­ed by the soft­ware in your host machine “

If you do, then you’re prob­a­bly attempt­ing to return the response stream direct­ly back to the rest of your appli­ca­tion with some­thing like this:

   1: var response = _s3Client.GetObject(request);

   2: return response.ResponseStream;

How­ev­er, because the stream is com­ing from the Ama­zon S3 ser­vice and is fed to your code in chunks, your code needs to ensure that the con­nec­tion to S3 stays open until all the data has been received. So as men­tioned in the S3 doc­u­men­ta­tion (which inci­den­tal­ly, most of us don’t read in great details…) here, you should be wrap­ping the response you get from the GetO­b­ject method in a using clause.

Depends on what it is you want to do with the stream, you might have to han­dle it dif­fer­ent­ly. For instance, if you just want to read the string con­tent of a text file, you might want to do this:

   1: using (var response = _s3Client.GetObject(request))

   2: {

   3:     using (var reader = new StreamReader(response.ResponseStream))

   4:     {

   5:         return reader.ReadToEnd();

   6:     }

   7: }

Alter­na­tive­ly, if you want to return the response stream itself, you’ll need to first load the stream in its entire­ty and return the loaded stream. Unfor­tu­nate­ly, at the time of this writ­ing, the AWSSDK library still hasn’t been migrat­ed to .Net 4 and there­fore doesn’t have the uber use­ful Copy­To method added in .Net 4, so you will most like­ly have to do the heavy lift­ing your­self and read the data out man­u­al­ly into a mem­o­ry stream:

   1: using (var response = _s3Client.GetObject(request))

   2: {

   3:     var binaryData = ReadFully(response.ResponseStream);

   4:     return new MemoryStream(binaryData);

   5: }

   6:

   7: /// <summary>

   8: /// See Jon Skeet's article on reading binary data:

   9: /// http://www.yoda.arachsys.com/csharp/readbinary.html

  10: /// </summary>

  11: public static byte[] ReadFully (Stream stream, int initialLength = -1)

  12: {

  13:     // If we've been passed an unhelpful initial length, just

  14:     // use 32K.

  15:     if (initialLength < 1)

  16:     {

  17:         initialLength = 32768;

  18:     }

  19:

  20:     byte[] buffer = new byte[initialLength];

  21:     int read=0;

  22:

  23:     int chunk;

  24:     while ( (chunk = stream.Read(buffer, read, buffer.Length-read)) > 0)

  25:     {

  26:         read += chunk;

  27:

  28:         // If we've reached the end of our buffer, check to see if there's

  29:         // any more information

  30:         if (read == buffer.Length)

  31:         {

  32:             int nextByte = stream.ReadByte();

  33:

  34:             // End of stream? If so, we're done

  35:             if (nextByte==-1)

  36:             {

  37:                 return buffer;

  38:             }

  39:

  40:             // Nope. Resize the buffer, put in the byte we've just

  41:             // read, and continue

  42:             byte[] newBuffer = new byte[buffer.Length*2];

  43:             Array.Copy(buffer, newBuffer, buffer.Length);

  44:             newBuffer[read]=(byte)nextByte;

  45:             buffer = newBuffer;

  46:             read++;

  47:         }

  48:     }

  49:     // Buffer is now too big. Shrink it.

  50:     byte[] ret = new byte[read];

  51:     Array.Copy(buffer, ret, read);

  52:     return ret;

  53: }