Introducing a better way to record custom metrics

You can become a serverless blackbelt. Enrol to my 4-week online workshop Production-Ready Serverless and gain hands-on experience building something from scratch using serverless technologies. At the end of the workshop, you should have a broader view of the challenges you will face as your serverless architecture matures and expands. You should also have a firm grasp on when serverless is a good fit for your system as well as common pitfalls you need to avoid. Sign up now and get 15% discount with the code yanprs15!

Many clients have asked me “how do I record custom metrics from Lambda?”.

Generally speaking, you can either:

  • Publish custom metrics synchronously – e.g. send them at the end of an invocation.
  • Publish custom metrics asynchronously by writing them to stdout first and then extracting them from CloudWatch Logs.

Problems with sending custom metrics synchronously

The synchronous approach adds latency to invocations. This can be especially problematic when those extra milliseconds are experienced by our users. For example, if the user is waiting for an API response.

Individually, the delay might be negligible. CloudWatch metrics typically respond within tens of milliseconds. That is acceptable to most. But they can quickly compound when functions call one another via API Gateway.

Moreover, services are most fragile around their integration points – i.e. when they make network calls to other services. Publishing metrics to CloudWatch introduces another integration point that you need to harden.

If CloudWatch experiences an outage, surely you would still want your system to stay up, right? Similarly, if CloudWatch experiences elevated response time then you wouldn’t want your functions to timeout as a result!

Hence why I generally prefer to record custom metrics asynchronously, even though this approach also has its drawbacks:

  • There is an additional delay in seeing the most recent metric data.
  • It’s sending more data to CloudWatch Logs, which has a cost implication

  • It introduces complexity because you need something to process the logs and turn them into metrics.

Sending custom metrics asynchronously

In simple cases, where you have few custom metrics, CloudWatch metric filters are the way to go. However, this approach does not scale with complexity – when you have lots of functions and custom metrics.

Instead, you can use a Lambda function.

To make it really easy for you to record custom metrics asynchronously, I have published a new application to the Serverless Application Repository. You can also check out the source code on GitHub here.

Getting started

You can deploy the app via the AWS console here, by clicking the Deploy button and follow the instructions.

Or you can deploy it as part of a CloudFormation stack with AWS SAM:

You can do the same via CloudFormation or the Serverless framework. You need to first add the following Transform though. For more details on how to do this with the Serverless framework, read this post.

Transform: AWS::Serverless-2016-10-31

We announced this new app live on Twitch yesterday. You can go to the 21:00 mark and see how you can configure everything via CloudFormation, including subscribing all CloudWatch log groups to a Kinesis stream first, before subscribing this app to that stream.

Recording custom metrics

Once deployed, you would be able to record custom metrics by writing to stdout in this format:

MONITORING|<value>|<unit>|<metric_name>|<namespace>|<dimensions>

where:

  • dimensions: comma-separated key-value pairs, e.g. service=content-item,region=eu-west-1

These messages would be processed and published as custom metrics in CloudWatch metrics. All without adding latency to your invocations!

Future works

I hope you enjoy this new app, and please feel free to suggest improvements via GitHub issues. Here are some ideas I have for making it more useful:

  • Parse the REPORT messages at the end of an invocation and turn Billed Duration, Memory Size and Memory Used into metrics:

  • Since AppSync doesn’t report resolver metrics to CloudWatch, we can parse the resolver logs and report resolver duration as metrics.
Liked this article? Support me on Patreon and get direct help from me via a private Slack channel or 1-2-1 mentoring.
Subscribe to my newsletter


Hi, I’m Yan. I’m an AWS Serverless Hero and I help companies go faster for less by adopting serverless technologies successfully.

Are you struggling with serverless or need guidance on best practices? Do you want someone to review your architecture and help you avoid costly mistakes down the line? Whatever the case, I’m here to help.

Hire me.


Skill up your serverless game with this hands-on workshop.

My 4-week Production-Ready Serverless online workshop is back!

This course takes you through building a production-ready serverless web application from testing, deployment, security, all the way through to observability. The motivation for this course is to give you hands-on experience building something with serverless technologies while giving you a broader view of the challenges you will face as the architecture matures and expands.

We will start at the basics and give you a firm introduction to Lambda and all the relevant concepts and service features (including the latest announcements in 2020). And then gradually ramping up and cover a wide array of topics such as API security, testing strategies, CI/CD, secret management, and operational best practices for monitoring and troubleshooting.

If you enrol now you can also get 15% OFF with the promo code “yanprs15”.

Enrol now and SAVE 15%.


Check out my new podcast Real-World Serverless where I talk with engineers who are building amazing things with serverless technologies and discuss the real-world use cases and challenges they face. If you’re interested in what people are actually doing with serverless and what it’s really like to be working with serverless day-to-day, then this is the podcast for you.


Check out my new course, Learn you some Lambda best practice for great good! In this course, you will learn best practices for working with AWS Lambda in terms of performance, cost, security, scalability, resilience and observability. We will also cover latest features from re:Invent 2019 such as Provisioned Concurrency and Lambda Destinations. Enrol now and start learning!


Check out my video course, Complete Guide to AWS Step Functions. In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. There is something for everyone from beginners to more advanced users looking for design patterns and best practices. Enrol now and start learning!