A self-healing Kinesis function that adapts its throughput based on performance

You can become a serverless blackbelt. Enrol to my 4-week online workshop Production-Ready Serverless and gain hands-on experience building something from scratch using serverless technologies. At the end of the workshop, you should have a broader view of the challenges you will face as your serverless architecture matures and expands. You should also have a firm grasp on when serverless is a good fit for your system as well as common pitfalls you need to avoid. Sign up now and get 15% discount with the code yanprs15!

I spent time with a client this week to solve an interesting problem – to adjust the number of concurrent requests to a downstream service dynamically based on response time and error rate. This is a common challenge when integrating with third parties, so we decided to share our approach so others might benefit from it.

The problem

My client is in the financial services sector and they integrate with over 50 service providers. They have API access to the providers’ data and they need to fetch them from every provider on a daily basis.

However, many of these providers place restrictions and rate limits on their APIs, for example:

  • You can only access the API during off-peak hours.
  • You cannot access the API during the weekly maintenance window.
  • You can only make X concurrent requests to the API.

My client has implemented a scheduler service to manage the timing constraints. The scheduler knows when to kick off the ingestion process for each provider. But there’s also a secondary process in which their users (financial advisers) can schedule ad-hoc ingestions themselves.

And there are other additional contexts that we had to keep in mind:

  • The APIs follow an established industry standard. Unfortunately, they can only return one record at a time.
  • The API response times are highly variable – anything from 5ms to 70s.
  • The amount of data the client needs to fetch varies from provider to provider. Some just a few thousand records, others would have hundreds of thousands of records.
  • We should not fetch the same record from a provider more than once per day. This owes to the usage limits.
  • This ingestion process is one of the most business-critical parts of their system. The migration to this new system needs to be managed carefully to minimise the potential impact to their customers.

The solution

To solve the aforementioned challenge, we need a mechanism to:

  • Fetch a large number of records from a provider’s API slowly and steadily. The process needs to be reliable but can take several hours if need be.
  • Only fetch a maximum of X records concurrently.
  • Gradually reduce the concurrency when the provider API’s response time or error rate starts to escalate.

Rather than implementing such a mechanism from scratch, we chose to leverage the built-in batching support with Kinesis and Lambda. In our proposed setup:

  • Every provider has a Kinesis stream.
  • The scheduler service enqueues the records that we need to fetch into the stream.
  • A ventilator Lambda function would receive records in batches and fan them out to a worker function (via direct function invocation).
  • There are many different worker functions, one for every provider. This is because most providers have provider-specific business logic.
  • To avoid processing the same record twice in the same day, the ventilator tracks records that have been processed today in a DynamoDB table.
  • To support retrying failed records, the ventilator also tracks the number of attempts at processing each record in the same DynamoDB table. When it encounters partial failures, the ventilator would increment the attempts count for the failed records and except. Kinesis would automatically retry the same batch of records for us. On subsequent attempts, previously successful records are ignored.

From a high-level, the system looks like this:

This design does not deliver maximum throughput, which is by design! As discussed earlier, this is due to the daily usage limits and scalability concerns of the provider APIs.

Every provider has its own CloudFormation stack with the Kinesis stream, the ventilator and worker Lambda functions, as well as the DynamoDB table. This set up makes it easy to migrate to the new system gradually, one provider at a time.

Self-healing function

The one stream per provider configuration lets us control concurrency using the batch size of the Kinesis event source mapping. The ventilator function can, therefore, self-adjust its batch size by updating the Kinesis event source mapping.

As the provider API’s response time goes up, the ventilator can respond by reducing its batch size. This gives the API a chance to catch its breath and recover. When the response time returns to acceptable levels, then the ventilator can gradually increase its batch size back to previous levels.

We went even a step further.

If the provider API still struggles at a batch size of 1, then something is clearly not right. You can’t change the batch size to 0, but you can disable the Kinesis event source mapping altogether. When you re-enable it later, Kinesis will push events to the ventilator from where it stopped before, as if nothing’s happened.

However, if the Kinesis event source is disabled, then something else needs to trigger wake the ventilator up later. For this, we have a CloudWatch schedule to trigger the ventilator function every 10 minutes.

If the Kinesis event source is already enabled then these CloudWatch-triggered invocations simply do nothing. When Kinesis is disabled, however, the ventilator can re-enable it at a batch size of 1. This allows one request to the provider API on the next invocation by Kinesis. If the response time is still above the threshold, then we will disable Kinesis again and the cycle repeats. Until such a time that the provider API is able to respond to requests in a timely fashion. Then the Kinesis stream stays enabled and the ventilator function would gradually restore its previous batch size.

This is the classic circuit breaker pattern applied to stream processing.

Show me how it works!

If you want to see how this can work in practice, I put together a demo app for illustration purpose. You can find the source code here. Follow the instructions in the README to deploy the demo app and pump data in using the provided feed-stream script.

With this demo app, I can adjust the ratio of slow and erroneous responses from the worker function via its environment variables.

When I do, you will soon see the ventilator take actions to reduce its batch size. It will adjust the batch size, incrementing and decrementing accordingly.

You can also create a metric filter to extract the batch size from the logs, and turn it into a metric in CloudWatch. This lets you observe how the batch size changes over time.

If the elevated response time and error rate continue, then eventually the ventilator function would disable its Kinesis event source.

The cron job would then re-enable the stream every 10 minutes.

And if the worker function has recovered.

Then the ventilator function would gradually increase its batch size.

Until it eventually returns to its original batch size of 10. If you monitor how the batch size changes along the way, you will probably see something along the lines of:


In this post, we discussed a design for a self-healing Kinesis function that can adapt its throughput based on performance. In doing so we’re able to carefully tune the number of concurrent requests our system makes against third-party APIs and stay within their operational limits. We also discussed how such design can implement the circuit breaker pattern and take the foot off the paddle entirely when the third-party API is struggling.

I hope you have enjoyed this post, don’t forget to let us know if you have any suggestions or comments.

Liked this article? Support me on Patreon and get direct help from me via a private Slack channel or 1-2-1 mentoring.
Subscribe to my newsletter

Hi, I’m Yan. I’m an AWS Serverless Hero and I help companies go faster for less by adopting serverless technologies successfully.

Are you struggling with serverless or need guidance on best practices? Do you want someone to review your architecture and help you avoid costly mistakes down the line? Whatever the case, I’m here to help.

Hire me.

Skill up your serverless game with this hands-on workshop.

My 4-week Production-Ready Serverless online workshop is back!

This course takes you through building a production-ready serverless web application from testing, deployment, security, all the way through to observability. The motivation for this course is to give you hands-on experience building something with serverless technologies while giving you a broader view of the challenges you will face as the architecture matures and expands.

We will start at the basics and give you a firm introduction to Lambda and all the relevant concepts and service features (including the latest announcements in 2020). And then gradually ramping up and cover a wide array of topics such as API security, testing strategies, CI/CD, secret management, and operational best practices for monitoring and troubleshooting.

If you enrol now you can also get 15% OFF with the promo code “yanprs15”.

Enrol now and SAVE 15%.

Check out my new podcast Real-World Serverless where I talk with engineers who are building amazing things with serverless technologies and discuss the real-world use cases and challenges they face. If you’re interested in what people are actually doing with serverless and what it’s really like to be working with serverless day-to-day, then this is the podcast for you.

Check out my new course, Learn you some Lambda best practice for great good! In this course, you will learn best practices for working with AWS Lambda in terms of performance, cost, security, scalability, resilience and observability. We will also cover latest features from re:Invent 2019 such as Provisioned Concurrency and Lambda Destinations. Enrol now and start learning!

Check out my video course, Complete Guide to AWS Step Functions. In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. There is something for everyone from beginners to more advanced users looking for design patterns and best practices. Enrol now and start learning!