Kinesis

The one mistake everyone makes when using Kinesis with Lambda

AWS Kinesis and Lambda are a great combo for processing large amounts of data in real-time. However, there’s a common oversight that many developers make when integrating these two services. There are established best practices for configuring Lambda’s EventSourceMapping [1] for Kinesis: Configure an OnFailure destination for failed records. Enable BisectBatchOnFunctionError. Override MaximumRetryAttempts. Choose a …

The one mistake everyone makes when using Kinesis with Lambda Read More »

3 ways to manage concurrency in serverless applications

Many software engineering concepts show up in different contexts. Modularity, the single-responsibility principle and separation of concerns are just a few examples that come to mind. They are equally applicable to how we write code, architect our systems and organize our teams. Similarly, there are many parallels between multithreaded programming and event-driven architectures. For example, …

3 ways to manage concurrency in serverless applications Read More »

How to include SNS and Kinesis in your e2e tests

Update 17/09/2019: based on feedback, I added a few more alternatives that do not require you to deploy additional Lambda functions to the e2e-test stage. Scroll to the end to find out more. Being event-driven is a key characteristic of a serverless architecture. Often our Lambda functions would publish events to SNS topics or Kinesis …

How to include SNS and Kinesis in your e2e tests Read More »

How to connect SNS to Kinesis for cross-account delivery via API Gateway

At DAZN (where I no longer work), the teams work with a number of third-party providers. They often have to synchronize data between different AWS accounts. SNS to SQS is the primary mechanism for these cross-account deliveries because: it was an established pattern within the organization DAZN engineers and third-party engineers are both familiar with …

How to connect SNS to Kinesis for cross-account delivery via API Gateway Read More »

A self-healing Lambda function that adapts its throughput based on performance

I spent time with a client this week to solve an interesting problem – to adjust the number of concurrent requests to a downstream service dynamically based on response time and error rate. This is a common challenge when integrating with third parties, so we decided to share our approach so others might benefit from …

A self-healing Lambda function that adapts its throughput based on performance Read More »

how to do fan-out and fan-in with AWS Lambda

In the last post, we look at how you can implement pub-sub with AWS Lambda. We compared several event sources you can use, SNS, Kinesis streams and DynamoDB streams, and the tradeoffs available to you. Let’s look at another messaging pattern today, push-pull, which is often referred to as fan-out/fan-in. It’s really two separate patterns …

how to do fan-out and fan-in with AWS Lambda Read More »

What is the best event source for doing pub-sub with AWS Lambda?

AWS offers a wealth of options for implementing messaging patterns such as pub-sub with Lambda, let’s compare and contrast some of these options. The pub-sub pattern Publish-Subscribe (often shortened to pub-sub) is a messaging pattern where publishers and subscribers are decoupled through an intermediary broker (ZeroMQ, RabbitMQ, SNS, etc.). SNS + Lambda In the AWS ecosystem, the obvious …

What is the best event source for doing pub-sub with AWS Lambda? Read More »

Capture and forward correlation IDs through different Lambda event sources

Serverless architectures are microservices by default, you need correlation IDs to help debug issues that spans across multiple functions, and possibly different event source types – asynchronous, synchronous and streams. This is the last of a 3-part mini series on managing your AWS Lambda logs. If you haven’t read part 1 yet, please give it a read now. …

Capture and forward correlation IDs through different Lambda event sources Read More »

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close