A simple event-sourcing example with snapshots using Lambda and DynamoDB

You can become a serverless blackbelt. Enrol to my 4-week online workshop Production-Ready Serverless and gain hands-on experience building something from scratch using serverless technologies. At the end of the workshop, you should have a broader view of the challenges you will face as your serverless architecture matures and expands. You should also have a firm grasp on when serverless is a good fit for your system as well as common pitfalls you need to avoid. Sign up now and get 15% discount with the code yanprs15!

Recently, I have been helping a client implement an event-sourced system. In the process, I put together a very simple demo app to illustrate how one could build such a system using Lambda and DynamoDB. The source code is available on GitHub here.


Before you go ahead and read all about the demo app, I want to give the client in question, InDebted, a quick shout out. They are disrupting the debt collection industry which has been riddled with malpractices and horror stories, and looking to protect the most vulnerable of us in society. They are also doing it by leveraging modern technologies and building with a serverless-first mentality. I have been working with the team for about 4 months and I have nothing but good things to say about them. If you’re looking for opportunities in the Sydney area, or are looking to relocate there, then please get in touch with Wagner. They’re looking for good people.


This demo app uses the banking example where a user can:

  • create an account
  • check his/her balance
  • withdraw money
  • credit the account

DynamoDB is the datastore.

Events

Every time the account holder withdraws from or credits the account, I will record an event.

It means that when I need to work out the current balance of the account I will have to build up its current state from these events.

Snapshots

A common question people ask about event-sourced systems is “how do you avoiding reading lots of data on every request?”

The solution is to create snapshots from time to time. In this demo app, I ensure that there are regular snapshots of the current state. One snapshot for every 10 rows in the table, to be precise.

These snapshots allow me to limit the number of rows I need to fetch on every request. In this case, I have a constant cost of fetching 10 items every time.

Rebuilding the current state

To rebuild the current state, I find the most recent snapshot and apply the events since the snapshot was taken.

For example, given the below:

The most recent snapshot is Version 22, with a Balance of 60. There have been 3 events since then. So the current balance is 60–10–10+10 = 50.

Here’s what it looks like in code:

Optimistic locking

To protect against concurrent updates to the account, the Version attribute is configured as the RANGE key. Whenever I add an event to the DynamoDB table, I will check that the version doesn’t exist already.

Optimizations

To bring down the cold start as well as warmed performance of the endpoints. I applied a number of basic optimization:

Streaming events to other consumers

It wasn’t included in the demo app, but you can also stream these events to other systems by:

a) letting other services subscribe to the DynamoDB table’s stream

b) create another Kinesis stream, and convert these DynamoDB INSERT events into domain events such as AccountCreated and BalanceWithdrawn.

My personal preference would be option b. It lets other consumers work with domain events and decouples them from implementation details in your service.

From here, you can also connect the Kinesis stream to Kinesis Firehose to persist the data to S3 as the data lake. You can then use Athena to run complex, ad-hoc queries over ALL the historical data, or to generate daily reports, or to feed a BI dashboard hosted in QuickSight.

Further reading

If you want to learn more about event-sourcing in the real-world (and at scale!), I recommend following this series by Rob Gruhl. Part 2 has some delightful patterns that you can use. You should also check out their Hello-Retail demo app.

Liked this article? Support me on Patreon and get direct help from me via a private Slack channel or 1-2-1 mentoring.
Subscribe to my newsletter


Hi, I’m Yan. I’m an AWS Serverless Hero and I help companies go faster for less by adopting serverless technologies successfully.

Are you struggling with serverless or need guidance on best practices? Do you want someone to review your architecture and help you avoid costly mistakes down the line? Whatever the case, I’m here to help.

Hire me.


Skill up your serverless game with this hands-on workshop.

My 4-week Production-Ready Serverless online workshop is back!

This course takes you through building a production-ready serverless web application from testing, deployment, security, all the way through to observability. The motivation for this course is to give you hands-on experience building something with serverless technologies while giving you a broader view of the challenges you will face as the architecture matures and expands.

We will start at the basics and give you a firm introduction to Lambda and all the relevant concepts and service features (including the latest announcements in 2020). And then gradually ramping up and cover a wide array of topics such as API security, testing strategies, CI/CD, secret management, and operational best practices for monitoring and troubleshooting.

If you enrol now you can also get 15% OFF with the promo code “yanprs15”.

Enrol now and SAVE 15%.


Check out my new podcast Real-World Serverless where I talk with engineers who are building amazing things with serverless technologies and discuss the real-world use cases and challenges they face. If you’re interested in what people are actually doing with serverless and what it’s really like to be working with serverless day-to-day, then this is the podcast for you.


Check out my new course, Learn you some Lambda best practice for great good! In this course, you will learn best practices for working with AWS Lambda in terms of performance, cost, security, scalability, resilience and observability. We will also cover latest features from re:Invent 2019 such as Provisioned Concurrency and Lambda Destinations. Enrol now and start learning!


Check out my video course, Complete Guide to AWS Step Functions. In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. There is something for everyone from beginners to more advanced users looking for design patterns and best practices. Enrol now and start learning!