S3 – Use using block to get the stream

You can become a serverless blackbelt. Enrol to my 4-week online workshop Production-Ready Serverless and gain hands-on experience building something from scratch using serverless technologies. At the end of the workshop, you should have a broader view of the challenges you will face as your serverless architecture matures and expands. You should also have a firm grasp on when serverless is a good fit for your system as well as common pitfalls you need to avoid. Sign up now and get 15% discount with the code yanprs15!

When you’re using the Amazon S3 client, have you come across the occasional exception that says something like one of these exception messages:

“The request was aborted: The connection was closed unexpectedly”

“Unable to read data from the transport connection: A blocking operation was interrupted by a call to WSACancelBlockingCall”

“Unable to read data from the transport connection: An established connection was aborted by the software in your host machine “

If you do, then you’re probably attempting to return the response stream directly back to the rest of your application with something like this:

   1: var response = _s3Client.GetObject(request);

   2: return response.ResponseStream;

However, because the stream is coming from the Amazon S3 service and is fed to your code in chunks, your code needs to ensure that the connection to S3 stays open until all the data has been received. So as mentioned in the S3 documentation (which incidentally, most of us don’t read in great details…) here, you should be wrapping the response you get from the GetObject method in a using clause.

Depends on what it is you want to do with the stream, you might have to handle it differently. For instance, if you just want to read the string content of a text file, you might want to do this:

   1: using (var response = _s3Client.GetObject(request))

   2: {

   3:     using (var reader = new StreamReader(response.ResponseStream))

   4:     {

   5:         return reader.ReadToEnd();

   6:     }

   7: }

Alternatively, if you want to return the response stream itself, you’ll need to first load the stream in its entirety and return the loaded stream. Unfortunately, at the time of this writing, the AWSSDK library still hasn’t been migrated to .Net 4 and therefore doesn’t have the uber useful CopyTo method added in .Net 4, so you will most likely have to do the heavy lifting yourself and read the data out manually into a memory stream:

   1: using (var response = _s3Client.GetObject(request))

   2: {

   3:     var binaryData = ReadFully(response.ResponseStream);

   4:     return new MemoryStream(binaryData);

   5: }


   7: /// <summary>

   8: /// See Jon Skeet's article on reading binary data:

   9: /// http://www.yoda.arachsys.com/csharp/readbinary.html

  10: /// </summary>

  11: public static byte[] ReadFully (Stream stream, int initialLength = -1)

  12: {

  13:     // If we've been passed an unhelpful initial length, just

  14:     // use 32K.

  15:     if (initialLength < 1)

  16:     {

  17:         initialLength = 32768;

  18:     }


  20:     byte[] buffer = new byte[initialLength];

  21:     int read=0;


  23:     int chunk;

  24:     while ( (chunk = stream.Read(buffer, read, buffer.Length-read)) > 0)

  25:     {

  26:         read += chunk;


  28:         // If we've reached the end of our buffer, check to see if there's

  29:         // any more information

  30:         if (read == buffer.Length)

  31:         {

  32:             int nextByte = stream.ReadByte();


  34:             // End of stream? If so, we're done

  35:             if (nextByte==-1)

  36:             {

  37:                 return buffer;

  38:             }


  40:             // Nope. Resize the buffer, put in the byte we've just

  41:             // read, and continue

  42:             byte[] newBuffer = new byte[buffer.Length*2];

  43:             Array.Copy(buffer, newBuffer, buffer.Length);

  44:             newBuffer[read]=(byte)nextByte;

  45:             buffer = newBuffer;

  46:             read++;

  47:         }

  48:     }

  49:     // Buffer is now too big. Shrink it.

  50:     byte[] ret = new byte[read];

  51:     Array.Copy(buffer, ret, read);

  52:     return ret;

  53: }

Liked this article? Support me on Patreon and get direct help from me via a private Slack channel or 1-2-1 mentoring.
Subscribe to my newsletter

Hi, I’m Yan. I’m an AWS Serverless Hero and I help companies go faster for less by adopting serverless technologies successfully.

Are you struggling with serverless or need guidance on best practices? Do you want someone to review your architecture and help you avoid costly mistakes down the line? Whatever the case, I’m here to help.

Hire me.

Skill up your serverless game with this hands-on workshop.

My 4-week Production-Ready Serverless online workshop is back!

This course takes you through building a production-ready serverless web application from testing, deployment, security, all the way through to observability. The motivation for this course is to give you hands-on experience building something with serverless technologies while giving you a broader view of the challenges you will face as the architecture matures and expands.

We will start at the basics and give you a firm introduction to Lambda and all the relevant concepts and service features (including the latest announcements in 2020). And then gradually ramping up and cover a wide array of topics such as API security, testing strategies, CI/CD, secret management, and operational best practices for monitoring and troubleshooting.

If you enrol now you can also get 15% OFF with the promo code “yanprs15”.

Enrol now and SAVE 15%.

Check out my new podcast Real-World Serverless where I talk with engineers who are building amazing things with serverless technologies and discuss the real-world use cases and challenges they face. If you’re interested in what people are actually doing with serverless and what it’s really like to be working with serverless day-to-day, then this is the podcast for you.

Check out my new course, Learn you some Lambda best practice for great good! In this course, you will learn best practices for working with AWS Lambda in terms of performance, cost, security, scalability, resilience and observability. We will also cover latest features from re:Invent 2019 such as Provisioned Concurrency and Lambda Destinations. Enrol now and start learning!

Check out my video course, Complete Guide to AWS Step Functions. In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. There is something for everyone from beginners to more advanced users looking for design patterns and best practices. Enrol now and start learning!