Check out my new course Learn you some Lambda best practice for great good! and learn the best practices for performance, cost, security, resilience, observability and scalability.
Hi, welcome to another weekly update! I’ve been busy since the last time we were here, and I have lots of cool updates to share with you.
New blog posts
How to use the power of CloudFormation custom resources for great good. In this post, I described an approach for invoking a Lambda function as part of a CloudFormation deployment using a reusable custom resource. This was useful for a number of SAR apps I maintain, including the new propagate-cfn-tags SAR app below. Where I’d like to process all existing CloudFormation stacks in the region during deployment.
Step Functions course update + 50% off offer
I added lots of new content to my Step Functions course to cover all the recent updates to the service, including:
- callback patterns – a better approach for pausing workflows while you wait for an external event, no more activities and no need to run pollers.
- nested workflows – to start another workflow inside a workflow
- blue-green deployments
Lots of videos were also updated to reflect improvements to the serverless-step-functions plugin.
To celebrate the update, I’m also running a special promotion till the end of September where you can buy the course for £9.99 instead of the usual £19.99.
DAZN Lambda powertools supports Firehose + Lambda layers
The DAZN Lambda powertools now supports Kinesis Firehose as an event source. This means we are able to:
- propagate correlation IDs through Firehose events so they are automatically included in Lambda logs
- support sampling debug logs
- track the number of invocation hops on a call-chain (to help prevent infinite recursions)
You can now also deploy all 16 packages from the DAZN Lambda powertools as a single Lambda layer. You can deploy the layer as a SAR application, using this technique by Gojko Adzic.
serverless-apigateway-service-proxy adds customization for S3 proxies
The serverless-apigateway-service-proxy plugin now lets you customize the S3 proxy, which is useful in many situations. For instance, when you cannot parameter the object key using path or query string parameters. Such is the challenge that one of my clients faced, where we subscribed an S3 service proxy endpoint to a third-party webhook. The service proxy endpoint needs to be a static URL, therefore we needed more control of the request mapping to use the request ID as object key.
Kudos to Erez Rokah for implementing this awesome feature.
serverless-lumigo supports Python functions
The serverless-lumigo plugin now officially supports Python functions! If you work with Python and would like to give Lumigo a try, then it’s easier to do so than ever before with the help of this Serverless framework plugin.
[new!] propgate-cfn-tags SAR
I published another SAR app called propagate-cfn-tags. This app addresses the problem that CloudFormation tags do not propagate to some resources such as CloudWatch log groups. By deploying this app to your account, you ensure that all existing CloudFormation stacks, as well as any future CloudFormation stacks, would propagate their tags to CloudWatch Log Groups that belong to the stacks.
async-custom-metrics SAR supports Lambda usage metrics and estimated cost metric
The async-custom-metrics SAR can now parse usage reports from Lambda logs and turn them into custom metrics:
- BilledDuration: Lambda invocations are billed in 100ms blocks, see how much time you’re actually billed for.
- MemorySize: the memory size (in MBs) of the function at the time it was invoked.
- MemoryUsed: the max amount of memory (in MBs) that was used by the invocation.
- EstimatedCost: the estimated cost for the function, calculated using the BilledDuration and MemorySize.
These custom metrics incur extra cost, so you don’t have to enable them. When you deploy the latest version of the app, you can decide whether or not to enable them.
If they are both enabled, then you will find the new metrics under the AWS/Lambda namespace.
Better DX for auto-subscribe-log-group-to-arn and auto-set-log-group-retention SAR apps
When you deploy the auto-subscribe-log-group-to-arn or auto-set-log-group-retention SAR apps, I want to process all existing log groups in the region. Previously, this was done via a cron job that runs once an hour. Which means when you first deploy the app, you won’t see any changes to existing log groups. It was confusing for users, and I wanted to give them a better developer experience.
Using the technique I outlined in this blog post, we can now invoke the relevant Lambda functions during deployment to process all existing log groups. So when the deployment is done, the log groups would have been updated. Giving you the feedback and causality that you’d expect.
[new!] cloudwatch-alarms-macro SAR
I have saved the best for last. A lot of clients have me “what alarms do I need for my serverless application?”. I wrote about CloudWatch metrics and discussed common alerts you should use. Many of these are repeatable, and to help you automate the process of creating these alarms I published another SAR app called cloudwatch-alarms-macro.
This app creates a macro in your account, which you can include in any CloudFormation stack, including AWS SAM or Serverless framework templates. You can customize the behaviour of the macro with a default configuration that controls what alarms are created by default, and what thresholds to use.
For example, as a convention, every Lambda function should have an alarm for when the error rate reaches 1% or above.
For a particular CloudFormation stack, you can also override this default configuration for particular functions. For example, you may choose to disable certain alarms or to use a different threshold.
Right now, the macro can auto-generate alarms for the following resources:
- API Gateway
- Step Functions
A longer post would follow in the next few days and explain the inner-workings of this macro and how you can make the best use of it.
Announcements from the community
In case you missed it, big VPC improvements are coming to Lambda! Once rolled out to your account, you will no longer experience the 10s cold start penalty for giving Lambda functions access to your VPC. Which is a showstopper for many companies looking to adopt serverless. AWS has been teasing this for a while now, and I’m really glad to hear that it’s finally being rolled out!
I specialise in rapidly transitioning teams to serverless and building production-ready services on AWS.
Are you struggling with serverless or need guidance on best practices? Do you want someone to review your architecture and help you avoid costly mistakes down the line? Whatever the case, I’m here to help.
Check out my new podcast Real-World Serverless where I talk with engineers who are building amazing things with serverless technologies and discuss the real-world use cases and challenges they face. If you’re interested in what people are actually doing with serverless and what it’s really like to be working with serverless day-to-day, then this is the podcast for you.
Check out my new course, Learn you some Lambda best practice for great good! In this course, you will learn best practices for working with AWS Lambda in terms of performance, cost, security, scalability, resilience and observability. We will also cover latest features from re:Invent 2019 such as Provisioned Concurrency and Lambda Destinations. Enrol now and start learning!
Check out my video course, Complete Guide to AWS Step Functions. In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. There is something for everyone from beginners to more advanced users looking for design patterns and best practices. Enrol now and start learning!
Are you working with Serverless and looking for expert training to level-up your skills? Or are you looking for a solid foundation to start from? Look no further, register for my Production-Ready Serverless workshop to learn how to build production-grade Serverless applications!
Here is a complete list of all my posts on serverless and AWS Lambda. In the meantime, here are a few of my most popular blog posts.
- Lambda optimization tip – enable HTTP keep-alive
- You are wrong about serverless and vendor lock-in
- You are thinking about serverless costs all wrong
- Just how expensive is the full AWS SDK?
- Many faced threats to Serverless security
- We can do better than percentile latencies
- Yubl’s road to Serverless
- AWS Lambda – should you have few monolithic functions or many single-purposed functions?
- AWS Lambda – compare coldstart time with different languages, memory and code sizes
- Guys, we’re doing pagination wrong
- Top 10 Serverless framework best practices