You can become a serverless blackbelt. Enrol to my 4-week online workshop Production-Ready Serverless and gain hands-on experience building something from scratch using serverless technologies. At the end of the workshop, you should have a broader view of the challenges you will face as your serverless architecture matures and expands. You should also have a firm grasp on when serverless is a good fit for your system as well as common pitfalls you need to avoid. Sign up now and get 15% discount with the code yanprs15!
The Serverless framework is the most popular deployment framework for serverless applications. It gives you a convenient abstraction over CloudFormation and some best practices out-of-the-box:
- Filters out dev dependencies for Node.js function.
- Update deployment packages to S3, which lets you work around the default 50MB limit on deployment packages.
- Enforces a consistent naming convention for functions and APIs.
But our serverless applications is not only about Lambda functions. We often have to deal with share resources such as VPCs, SQS queues and RDS databases. For example, you might have a centralised Kinesis stream to capture all applications events in the system. In this case, the stream doesn’t belong to any one project and shouldn’t be tied to their deployment cycles.
You still need to follow the principle of Infrastructure as Code:
- version control changes to these shared resources, and
- ensure they can be deployed in a consistent way to different environments
You can still use the Serverless framework to manage these shared resources. It is an abstraction layer over CloudFormation after all. Even without Lambda functions, you can configure AWS resources using normal CloudFormation syntax in YAML.
But this is often frowned upon by DevOps/infrastructure teams. Perhaps the name “Serverless” makes one assume it’s only for deploying serverless applications. On the other hand, Terraform is immensely popular in the DevOps space and enjoys a cult-like following.
I see many teams use both Terraform and Serverless framework in their stack:
- Serverless framework to deploy Lambda functions and their event sources (API Gateway, etc.).
- Terraform to deploy shared dependencies such as VPCs and RDS databases.
The Serverless framework translates your
serverless.yml into a CloudFormation stack during deployment. It also lets you reference outputs from another CloudFormation stack. But there’s no built-in support to reference Terraform state. So there is no easy way to reference the shared resources that are managed by Terraform.
Here at DAZN we have used a simple trick to make Serverless framework and Terraform work together. Reading the Terraform state from the Serverless framework is tricky. So, we cheat ;-)
We would create a CloudFormation stack as part of every Terraform script. This CloudFormation stack would hold the output from the resources that Terraform creates?—?ARNs, etc. We would then be able to reference them from our
Let’s look at a simple example.
Here’s a simple Terraform script that provisions a SQS queue.
To export the ARN and URL of this queue, we need to add a CloudFormation Stack to our script. Notice that the stack specifies the outputs
MyQueueUrl. This is all we wanted to do here. But unfortunately CloudFormation requires you to specify at least one resource…
Since the stack is here to provide outputs for others to reference, let’s stay with that theme. Let’s expose the SQS attributes as SSM parameters as well.
After you run
terraform plan and
terraform apply you will be able to find the
my-terraform-outputs stack in CloudFormation. You will find the URL and ARN for the SQS queue in this stack’s output.
From here, we can reference these outputs from a
Since our stack also created SSM parameters for these outputs, we can also reference them from SSM Parameter Store too.
The Serverless framework lets you reference variables from a number of AWS services:
- Another CloudFormation stack’s output.
- A JSON file in S3.
- SSM Parameter Store.
- Secrets Manager.
So you don’t have to use CloudFormation as a way to store outputs from Terraform. Which as you can see, forces you to also provision some resources via CloudFormation…
Assuming we’re not talking about application secrets (which, is a whole separate topic) you should consider outputting them to SSM parameters instead.
Hi, I’m Yan. I’m an AWS Serverless Hero and I help companies go faster for less by adopting serverless technologies successfully.
Are you struggling with serverless or need guidance on best practices? Do you want someone to review your architecture and help you avoid costly mistakes down the line? Whatever the case, I’m here to help.
Skill up your serverless game with this hands-on workshop.
My 4-week Production-Ready Serverless online workshop is back!
This course takes you through building a production-ready serverless web application from testing, deployment, security, all the way through to observability. The motivation for this course is to give you hands-on experience building something with serverless technologies while giving you a broader view of the challenges you will face as the architecture matures and expands.
We will start at the basics and give you a firm introduction to Lambda and all the relevant concepts and service features (including the latest announcements in 2020). And then gradually ramping up and cover a wide array of topics such as API security, testing strategies, CI/CD, secret management, and operational best practices for monitoring and troubleshooting.
If you enrol now you can also get 15% OFF with the promo code “yanprs15”.
Check out my new podcast Real-World Serverless where I talk with engineers who are building amazing things with serverless technologies and discuss the real-world use cases and challenges they face. If you’re interested in what people are actually doing with serverless and what it’s really like to be working with serverless day-to-day, then this is the podcast for you.
Check out my new course, Learn you some Lambda best practice for great good! In this course, you will learn best practices for working with AWS Lambda in terms of performance, cost, security, scalability, resilience and observability. We will also cover latest features from re:Invent 2019 such as Provisioned Concurrency and Lambda Destinations. Enrol now and start learning!
Check out my video course, Complete Guide to AWS Step Functions. In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. There is something for everyone from beginners to more advanced users looking for design patterns and best practices. Enrol now and start learning!
Here is a complete list of all my posts on serverless and AWS Lambda. In the meantime, here are a few of my most popular blog posts.
- All you need to know about caching for serverless applications
- Lambda optimization tip – enable HTTP keep-alive
- You are wrong about serverless and vendor lock-in
- You are thinking about serverless costs all wrong
- Just how expensive is the full AWS SDK?
- Check-list for going live with API Gateway and Lambda
- How to choose the right API Gateway auth method
- CloudFormation protip: use !Sub instead of !Join
- AWS Lambda – should you have few monolithic functions or many single-purposed functions?
- Guys, we’re doing pagination wrong
- Top 10 Serverless framework best practices
- How to break the “senior engineer” career ceiling
- My advice to junior developers