Should you pack the AWS SDK in your deployment artefact?

Update 06/04/2023:

It’s been 3 years since I originally wrote this blog post and I want to share some updates to my thinking because this blog post was referenced in a recent PR for the serverless-esbuild project. Thanks, Fredrick for messaging me about this.

Let me start by saying that the risks I outlined in this post are still there.

1. The built-in AWS SDK can become outdated and miss security patches and bug fixes.
2. It invalidates integration tests since the runtime uses a different version of the AWS SDK to what was tested.
3. AWS can update the built-in AWS SDK without notice

But these risks have rarely, if ever, manifested into actual problems. For instance, I haven’t heard any reports of people’s functions suddenly breaking when the version of the built-in AWS SDK is updated under the cover. Since the incident I reported below, I have not heard any reports of people’s functions breaking because of changes to the built-in AWS SDK. In general, I think the AWS SDKs have become more stable based on empirical evidence.

I have also observed that the version of the built-in AWS SDK is updated more frequently these days. So there is less chance of them getting out-of-date and missing critical security updates.

Having said that, AWS still recommends including the AWS SDK in your deployment package (see the official best practices guide). And in some environments, especially large enterprises, security teams would mandate that all your application dependencies need to be scanned by the approved vendor. So points 2 & 3 above are non-starters.

So once again, the tradeoff is between convenience and safety.

And since I initially wrote this post, my default stance has shifted towards convenience. Because I have built up a lot of faith in how AWS manages these Lambda execution environments and keeping their stable. So unless the situation demands that I prioritize safety over convenience, I won’t be including the AWS SDK in my deployment artefact.

But you know what? One incident is all it’d take to swing me back the other way!

So if you’re building libraries or services for someone else to use, such as the serverless-esbuild project. Then I think it’s better to err on the side of caution. Because all it’ll take is one incident for you to lose the trust and goodwill you spend years building with your user base.

ps. When I do include the AWS SDK as part of my deployment artefact, I do so using the serverless-layers plugin. Because it knows whether my dependencies have changed and only creates new layer versions when they have. To me, this is the best way to use Lambda Layers because Lambda Layers is not a suitable replacement for package managers like NPM, but it’s a good deployment optimization.

 

Update 01/10/2019: a few of you have mentioned layers in the comments or on social media. Scroll to the bottom to see my thoughts on layers as a solution to the problems I mentioned in this post.

A version of the AWS SDK is always bundled into the Lambda runtime for your language. So the conventional wisdom says you don’t need to include it in your deployment artefact.

I’m here to tell you otherwise.

But first, let’s examine the argument for not packing the AWS SDK.

Deployment artefacts are smaller

This means deployment is faster. Although the bulk of the deployment time is with CloudFormation, not uploading the artefact. So the actual time saving depends on your internet speed.

It also means you are less likely to hit the 75GB code storage limit. This is fair, but I feel this risk can be mitigated with the lambda-janitor.

Lastly, you can actually see and edit your code in the Cloud9 editor! Alright, this is pretty useful when you just wanna quickly check something out. Although when you’re building a production service, you wouldn’t want to rely on the AWS console, ever. You certainly wouldn’t be making code changes directly in the console!

So what’s wrong with relying on the built-in version of the AWS SDK?

Old versions

The version of the built-in AWS SDK is often much older than the latest. This means it doesn’t have the latest security patches, bug fixes as well as support for new features. Known vulnerabilities can exist in both the AWS SDK itself as well as its transient dependencies.

This can also manifest itself in subtle differences in the behaviour or performance of your application. Or as bugs that you can’t replicate when you invoke the function locally. Both are problems that are difficult to track down.

Invalidates tests

If you rely on tests that execute the Lambda code locally then the fact the runtime has a different version of AWS SDK would invalidate your tests. Personally, I think you should always have end-to-end tests as well. Which runs against the deployed system and checks everything works as expected when running inside AWS.

No immutable infrastructure

AWS can update the version of the built-in AWS SDK without notice. This can introduce subtle differences in your application’s behaviour without any action from you.

A while back, AWS updated the version of boto in the Python runtime. Unfortunately, the version they upgraded to had a bug. It caused many people’s functions to suddenly break. This affected Bustle amongst others and many hours were wasted on debugging the issue. As you can imagine, it wasn’t the easiest problem to track down!

Summary

In summary, my arguments against using the built-in AWS SDK are:

1. The built-in AWS SDK can become outdated and miss security patches and bug fixes.
2. It invalidates integration tests since the runtime uses a different version of the AWS SDK to what was tested.
3. AWS can update the built-in AWS SDK without notice

Ultimately, I think the tradeoff is between convenience and safety. And I for one, am happy to sacrifice small conveniences for immutable infrastructure.

What about layers?

Quite a few of you have mentioned Lambda layers as a solution. Yes, you can ship the AWS SDK via layers instead. It would address point 1 and 3 above, but it introduces other challenges. I discussed those challenges in this post, so please give that a read.

Granted, some of those challenges have been tackled – e.g. the Serverless framework now supports layers when you invoke functions locally, and you can introduce semantic versioning to layers by wrapping them in SAR apps. However, it still remains a tooling problem if you want to run unit or integration tests before deploying your functions to AWS. In both cases, you’d still need a copy of the AWS SDK locally. For a dependency that is relatively small (a few MBs) and should be updated frequently, such as the AWS SDK, I don’t feel layers is worth the hassle it brings.

p.s. AWS also recommends including your own copy of the AWS SDK in the official documentation.

 

I hope you’ve found this post useful. If you want to learn more about running serverless in production and what it takes to build production-ready serverless applications then check out my upcoming workshop, Production-Ready Serverless!

In the workshop, I will give you a quick introduction to AWS Lambda and the Serverless framework, and take you through topics such as:

• testing strategies
• how to secure your APIs
• API Gateway best practices
• CI/CD
• configuration management
• security best practices
• event-driven architectures
• how to build observability into serverless applications

and much more!