How to use the power of CloudFormation custom resources for great good

You can become a serverless blackbelt. Enrol to my 4-week online workshop Production-Ready Serverless and gain hands-on experience building something from scratch using serverless technologies. At the end of the workshop, you should have a broader view of the challenges you will face as your serverless architecture matures and expands. You should also have a firm grasp on when serverless is a good fit for your system as well as common pitfalls you need to avoid. Sign up now and get 15% discount with the code yanprs15!

Disclaimer: if you’re new to CloudFormation custom resources, then I recommend you start by reading this excellent post by Alex Debrie first.

Custom resources bring a whole new dimension to CloudFormation and enable some fascinating use cases. For example:

  • provision DataDog dashboards as part of your CloudFormation stack
  • run a load test every time you deploy a CloudFormation stack
  • provision/modify AWS resources that are not natively supported by CloudFormation (e.g. EventBridge)

You can use custom resources to run one-off tasks as part of every CloudFormation deployment. Which is something I wanted to do in many of my SAR apps such as this and this. However, I had always assumed that this would require a two-step process:

  1. deploy a SAR/CloudFormation stack to provision the custom resource Lambda function
  2. deploy the intended SAR app which uses the custom resource

Turns out, I was wrong. You can include the custom resource Lambda function in the same CloudFormation stack that uses it. In hindsight, I was overthinking things – in essence, all we are doing is referencing a Lambda function by ARN. You’ll see what this means later.

My latest SAR app, propagate-cfn-tags, tackles the issue that CloudFormation does not propagate its tags to some resources such as CloudWatch log groups. This means you can’t include CloudWatch log groups in your cost monitoring (via cost allocation tags), amongst other things.

When you deploy the app, I want to run a one-off script to process all existing stacks in the region and propagate their tags. This is where custom resources come in.

With the help of custom resources, I can:

  • create a PropagateAll Lambda function to process existing CloudFormation stacks

  • create a Lambda function to support a Custom::LambdaInvocation custom resource type

  • create a Custom::LambdaInvocation resource to trigger the PropagateAll function during the CloudFormation deployment

I can do all these in the same SAR app. This gives the user a seamless experience. As far as they’re concerned, you just deploy the app and everything happens as advertised. No two-step deployment process, no need to trigger functions manually.

This simple pattern combines very well with CloudWatch events (or EventBridge):

  • a custom resource to update existing resources (e.g. propagate tags for existing CloudFormation stacks)
  • CloudTrail + event pattern to process future changes (e.g. propagate tags for future CloudFormation stack deployments)

I find this combination especially useful when building “platform” features. That is, features that span across all my applications running in AWS. For example, making sure all CloudWatch log groups are subscribed to a Kinesis stream. These are not decision points that should be taken at the project level. Which is why I’m not a fan of using Serverless framework plugins to configure these settings.

Liked this article? Support me on Patreon and get direct help from me via a private Slack channel or 1-2-1 mentoring.
Subscribe to my newsletter

Hi, I’m Yan. I’m an AWS Serverless Hero and I help companies go faster for less by adopting serverless technologies successfully.

Are you struggling with serverless or need guidance on best practices? Do you want someone to review your architecture and help you avoid costly mistakes down the line? Whatever the case, I’m here to help.

Hire me.

Skill up your serverless game with this hands-on workshop.

My 4-week Production-Ready Serverless online workshop is back!

This course takes you through building a production-ready serverless web application from testing, deployment, security, all the way through to observability. The motivation for this course is to give you hands-on experience building something with serverless technologies while giving you a broader view of the challenges you will face as the architecture matures and expands.

We will start at the basics and give you a firm introduction to Lambda and all the relevant concepts and service features (including the latest announcements in 2020). And then gradually ramping up and cover a wide array of topics such as API security, testing strategies, CI/CD, secret management, and operational best practices for monitoring and troubleshooting.

If you enrol now you can also get 15% OFF with the promo code “yanprs15”.

Enrol now and SAVE 15%.

Check out my new podcast Real-World Serverless where I talk with engineers who are building amazing things with serverless technologies and discuss the real-world use cases and challenges they face. If you’re interested in what people are actually doing with serverless and what it’s really like to be working with serverless day-to-day, then this is the podcast for you.

Check out my new course, Learn you some Lambda best practice for great good! In this course, you will learn best practices for working with AWS Lambda in terms of performance, cost, security, scalability, resilience and observability. We will also cover latest features from re:Invent 2019 such as Provisioned Concurrency and Lambda Destinations. Enrol now and start learning!

Check out my video course, Complete Guide to AWS Step Functions. In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. There is something for everyone from beginners to more advanced users looking for design patterns and best practices. Enrol now and start learning!