Check out my new course Learn you some Lambda best practice for great good! and learn the best practices for performance, cost, security, resilience, observability and scalability.
The Serverless framework is the most popular deployment framework for serverless applications. It gives you a convenient abstraction over CloudFormation and some best practices out-of-the-box:
- Filters out dev dependencies for Node.js function.
- Update deployment packages to S3, which lets you work around the default 50MB limit on deployment packages.
- Enforces a consistent naming convention for functions and APIs.
But our serverless applications is not only about Lambda functions. We often have to deal with share resources such as VPCs, SQS queues and RDS databases. For example, you might have a centralised Kinesis stream to capture all applications events in the system. In this case, the stream doesn’t belong to any one project and shouldn’t be tied to their deployment cycles.
You still need to follow the principle of Infrastructure as Code:
- version control changes to these shared resources, and
- ensure they can be deployed in a consistent way to different environments
You can still use the Serverless framework to manage these shared resources. It is an abstraction layer over CloudFormation after all. Even without Lambda functions, you can configure AWS resources using normal CloudFormation syntax in YAML.
But this is often frowned upon by DevOps/infrastructure teams. Perhaps the name “Serverless” makes one assume it’s only for deploying serverless applications. On the other hand, Terraform is immensely popular in the DevOps space and enjoys a cult-like following.
I see many teams use both Terraform and Serverless framework in their stack:
- Serverless framework to deploy Lambda functions and their event sources (API Gateway, etc.).
- Terraform to deploy shared dependencies such as VPCs and RDS databases.
The Serverless framework translates your
serverless.yml into a CloudFormation stack during deployment. It also lets you reference outputs from another CloudFormation stack. But there’s no built-in support to reference Terraform state. So there is no easy way to reference the shared resources that are managed by Terraform.
Here at DAZN we have used a simple trick to make Serverless framework and Terraform work together. Reading the Terraform state from the Serverless framework is tricky. So, we cheat ;-)
We would create a CloudFormation stack as part of every Terraform script. This CloudFormation stack would hold the output from the resources that Terraform creates?—?ARNs, etc. We would then be able to reference them from our
Let’s look at a simple example.
Here’s a simple Terraform script that provisions a SQS queue.
To export the ARN and URL of this queue, we need to add a CloudFormation Stack to our script. Notice that the stack specifies the outputs
MyQueueUrl. This is all we wanted to do here. But unfortunately CloudFormation requires you to specify at least one resource…
Since the stack is here to provide outputs for others to reference, let’s stay with that theme. Let’s expose the SQS attributes as SSM parameters as well.
After you run
terraform plan and
terraform apply you will be able to find the
my-terraform-outputs stack in CloudFormation. You will find the URL and ARN for the SQS queue in this stack’s output.
From here, we can reference these outputs from a
Since our stack also created SSM parameters for these outputs, we can also reference them from SSM Parameter Store too.
The Serverless framework lets you reference variables from a number of AWS services:
- Another CloudFormation stack’s output.
- A JSON file in S3.
- SSM Parameter Store.
- Secrets Manager.
So you don’t have to use CloudFormation as a way to store outputs from Terraform. Which as you can see, forces you to also provision some resources via CloudFormation…
Assuming we’re not talking about application secrets (which, is a whole separate topic) you should consider outputting them to SSM parameters instead.
I specialise in rapidly transitioning teams to serverless and building production-ready services on AWS.
Are you struggling with serverless or need guidance on best practices? Do you want someone to review your architecture and help you avoid costly mistakes down the line? Whatever the case, I’m here to help.
Check out my new podcast Real-World Serverless where I talk with engineers who are building amazing things with serverless technologies and discuss the real-world use cases and challenges they face. If you’re interested in what people are actually doing with serverless and what it’s really like to be working with serverless day-to-day, then this is the podcast for you.
Check out my new course, Learn you some Lambda best practice for great good! In this course, you will learn best practices for working with AWS Lambda in terms of performance, cost, security, scalability, resilience and observability. We will also cover latest features from re:Invent 2019 such as Provisioned Concurrency and Lambda Destinations. Enrol now and start learning!
Check out my video course, Complete Guide to AWS Step Functions. In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. There is something for everyone from beginners to more advanced users looking for design patterns and best practices. Enrol now and start learning!
Are you working with Serverless and looking for expert training to level-up your skills? Or are you looking for a solid foundation to start from? Look no further, register for my Production-Ready Serverless workshop to learn how to build production-grade Serverless applications!
Here is a complete list of all my posts on serverless and AWS Lambda. In the meantime, here are a few of my most popular blog posts.
- Lambda optimization tip – enable HTTP keep-alive
- You are wrong about serverless and vendor lock-in
- You are thinking about serverless costs all wrong
- Just how expensive is the full AWS SDK?
- Many faced threats to Serverless security
- We can do better than percentile latencies
- Yubl’s road to Serverless
- AWS Lambda – should you have few monolithic functions or many single-purposed functions?
- AWS Lambda – compare coldstart time with different languages, memory and code sizes
- Guys, we’re doing pagination wrong
- Top 10 Serverless framework best practices