You can become a serverless blackbelt. Enrol to my 4-week online workshop Production-Ready Serverless and gain hands-on experience building something from scratch using serverless technologies. At the end of the workshop, you should have a broader view of the challenges you will face as your serverless architecture matures and expands. You should also have a firm grasp on when serverless is a good fit for your system as well as common pitfalls you need to avoid. Sign up now and get 15% discount with the code yanprs15!
Hi, welcome to another weekly update! I’ve been busy since the last time we were here, and I have lots of cool updates to share with you.
New blog posts
How to use the power of CloudFormation custom resources for great good. In this post, I described an approach for invoking a Lambda function as part of a CloudFormation deployment using a reusable custom resource. This was useful for a number of SAR apps I maintain, including the new propagate-cfn-tags SAR app below. Where I’d like to process all existing CloudFormation stacks in the region during deployment.
Step Functions course update + 50% off offer
I added lots of new content to my Step Functions course to cover all the recent updates to the service, including:
- callback patterns – a better approach for pausing workflows while you wait for an external event, no more activities and no need to run pollers.
- nested workflows – to start another workflow inside a workflow
- blue-green deployments
Lots of videos were also updated to reflect improvements to the serverless-step-functions plugin.
To celebrate the update, I’m also running a special promotion till the end of September where you can buy the course for £9.99 instead of the usual £19.99.
DAZN Lambda powertools supports Firehose + Lambda layers
The DAZN Lambda powertools now supports Kinesis Firehose as an event source. This means we are able to:
- propagate correlation IDs through Firehose events so they are automatically included in Lambda logs
- support sampling debug logs
- track the number of invocation hops on a call-chain (to help prevent infinite recursions)
You can now also deploy all 16 packages from the DAZN Lambda powertools as a single Lambda layer. You can deploy the layer as a SAR application, using this technique by Gojko Adzic.
serverless-apigateway-service-proxy adds customization for S3 proxies
The serverless-apigateway-service-proxy plugin now lets you customize the S3 proxy, which is useful in many situations. For instance, when you cannot parameter the object key using path or query string parameters. Such is the challenge that one of my clients faced, where we subscribed an S3 service proxy endpoint to a third-party webhook. The service proxy endpoint needs to be a static URL, therefore we needed more control of the request mapping to use the request ID as object key.
Kudos to Erez Rokah for implementing this awesome feature.
serverless-lumigo supports Python functions
The serverless-lumigo plugin now officially supports Python functions! If you work with Python and would like to give Lumigo a try, then it’s easier to do so than ever before with the help of this Serverless framework plugin.
[new!] propgate-cfn-tags SAR
I published another SAR app called propagate-cfn-tags. This app addresses the problem that CloudFormation tags do not propagate to some resources such as CloudWatch log groups. By deploying this app to your account, you ensure that all existing CloudFormation stacks, as well as any future CloudFormation stacks, would propagate their tags to CloudWatch Log Groups that belong to the stacks.
async-custom-metrics SAR supports Lambda usage metrics and estimated cost metric
The async-custom-metrics SAR can now parse usage reports from Lambda logs and turn them into custom metrics:
- BilledDuration: Lambda invocations are billed in 100ms blocks, see how much time you’re actually billed for.
- MemorySize: the memory size (in MBs) of the function at the time it was invoked.
- MemoryUsed: the max amount of memory (in MBs) that was used by the invocation.
- EstimatedCost: the estimated cost for the function, calculated using the BilledDuration and MemorySize.
These custom metrics incur extra cost, so you don’t have to enable them. When you deploy the latest version of the app, you can decide whether or not to enable them.
If they are both enabled, then you will find the new metrics under the AWS/Lambda namespace.
Better DX for auto-subscribe-log-group-to-arn and auto-set-log-group-retention SAR apps
When you deploy the auto-subscribe-log-group-to-arn or auto-set-log-group-retention SAR apps, I want to process all existing log groups in the region. Previously, this was done via a cron job that runs once an hour. Which means when you first deploy the app, you won’t see any changes to existing log groups. It was confusing for users, and I wanted to give them a better developer experience.
Using the technique I outlined in this blog post, we can now invoke the relevant Lambda functions during deployment to process all existing log groups. So when the deployment is done, the log groups would have been updated. Giving you the feedback and causality that you’d expect.
[new!] cloudwatch-alarms-macro SAR
I have saved the best for last. A lot of clients have me “what alarms do I need for my serverless application?”. I wrote about CloudWatch metrics and discussed common alerts you should use. Many of these are repeatable, and to help you automate the process of creating these alarms I published another SAR app called cloudwatch-alarms-macro.
This app creates a macro in your account, which you can include in any CloudFormation stack, including AWS SAM or Serverless framework templates. You can customize the behaviour of the macro with a default configuration that controls what alarms are created by default, and what thresholds to use.
For example, as a convention, every Lambda function should have an alarm for when the error rate reaches 1% or above.
For a particular CloudFormation stack, you can also override this default configuration for particular functions. For example, you may choose to disable certain alarms or to use a different threshold.
Right now, the macro can auto-generate alarms for the following resources:
- API Gateway
- Step Functions
A longer post would follow in the next few days and explain the inner-workings of this macro and how you can make the best use of it.
Announcements from the community
In case you missed it, big VPC improvements are coming to Lambda! Once rolled out to your account, you will no longer experience the 10s cold start penalty for giving Lambda functions access to your VPC. Which is a showstopper for many companies looking to adopt serverless. AWS has been teasing this for a while now, and I’m really glad to hear that it’s finally being rolled out!
Hi, I’m Yan. I’m an AWS Serverless Hero and I help companies go faster for less by adopting serverless technologies successfully.
Are you struggling with serverless or need guidance on best practices? Do you want someone to review your architecture and help you avoid costly mistakes down the line? Whatever the case, I’m here to help.
Skill up your serverless game with this hands-on workshop.
My 4-week Production-Ready Serverless online workshop is back!
This course takes you through building a production-ready serverless web application from testing, deployment, security, all the way through to observability. The motivation for this course is to give you hands-on experience building something with serverless technologies while giving you a broader view of the challenges you will face as the architecture matures and expands.
We will start at the basics and give you a firm introduction to Lambda and all the relevant concepts and service features (including the latest announcements in 2020). And then gradually ramping up and cover a wide array of topics such as API security, testing strategies, CI/CD, secret management, and operational best practices for monitoring and troubleshooting.
If you enrol now you can also get 15% OFF with the promo code “yanprs15”.
Check out my new podcast Real-World Serverless where I talk with engineers who are building amazing things with serverless technologies and discuss the real-world use cases and challenges they face. If you’re interested in what people are actually doing with serverless and what it’s really like to be working with serverless day-to-day, then this is the podcast for you.
Check out my new course, Learn you some Lambda best practice for great good! In this course, you will learn best practices for working with AWS Lambda in terms of performance, cost, security, scalability, resilience and observability. We will also cover latest features from re:Invent 2019 such as Provisioned Concurrency and Lambda Destinations. Enrol now and start learning!
Check out my video course, Complete Guide to AWS Step Functions. In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. There is something for everyone from beginners to more advanced users looking for design patterns and best practices. Enrol now and start learning!
Here is a complete list of all my posts on serverless and AWS Lambda. In the meantime, here are a few of my most popular blog posts.
- All you need to know about caching for serverless applications
- Lambda optimization tip – enable HTTP keep-alive
- You are wrong about serverless and vendor lock-in
- You are thinking about serverless costs all wrong
- Just how expensive is the full AWS SDK?
- Check-list for going live with API Gateway and Lambda
- How to choose the right API Gateway auth method
- CloudFormation protip: use !Sub instead of !Join
- AWS Lambda – should you have few monolithic functions or many single-purposed functions?
- Guys, we’re doing pagination wrong
- Top 10 Serverless framework best practices
- How to break the “senior engineer” career ceiling
- My advice to junior developers