Kinesis

The one mistake everyone makes when using Kinesis with Lambda

AWS, Kinesis, Lambda, Serverless

AWS Kinesis and Lambda are a great combo for processing large amounts of data in real-time. However, there’s a common oversight that many developers make when integrating these two services. There are established best practices for configuring Lambda’s EventSourceMapping [1] for Kinesis: Configure an OnFailure destination for failed records. Enable BisectBatchOnFunctionError. Override MaximumRetryAttempts. Choose a …

The one mistake everyone makes when using Kinesis with Lambda Read More »

3 ways to control concurrency in serverless applications

AWS, EventBridge, Kinesis, Lambda, Serverless, Step Functions

Many software engineering concepts appear in different contexts. Modularity, the single-responsibility principle and separation of concerns are just a few examples that come to mind. They apply equally to how we write code, architect our systems and organize our teams. In this post, let’s discuss three ways we can control the concurrency of a serverless …

3 ways to control concurrency in serverless applications Read More »

How to include SNS and Kinesis in your e2e tests

API Gateway, AWS, DynamoDB, Kinesis, Lambda, Programming, Serverless, SNS, Testing

Update 17/09/2019: based on feedback, I added a few more alternatives that do not require you to deploy additional Lambda functions to the e2e-test stage. Scroll to the end to find out more. Being event-driven is a key characteristic of a serverless architecture. Often our Lambda functions would publish events to SNS topics or Kinesis …

How to include SNS and Kinesis in your e2e tests Read More »

How to connect SNS to Kinesis for cross-account delivery via API Gateway

API Gateway, AWS, Kinesis, Lambda, Programming, Serverless, SNS

At DAZN (where I no longer work), the teams work with a number of third-party providers. They often have to synchronize data between different AWS accounts. SNS to SQS is the primary mechanism for these cross-account deliveries because: it was an established pattern within the organization DAZN engineers and third-party engineers are both familiar with …

How to connect SNS to Kinesis for cross-account delivery via API Gateway Read More »

how to do fan-out and fan-in with AWS Lambda

API Gateway, AWS, Design Patterns, DynamoDB, Kinesis, Lambda, Programming, Serverless, SNS, SQS

In the last post, we look at how you can implement pub-sub with AWS Lambda. We compared several event sources you can use, SNS, Kinesis streams and DynamoDB streams, and the tradeoffs available to you. Let’s look at another messaging pattern today, push-pull, which is often referred to as fan-out/fan-in. It’s really two separate patterns …

how to do fan-out and fan-in with AWS Lambda Read More »

What is the best event source for doing pub-sub with AWS Lambda?

AWS, Design Patterns, DynamoDB, Kinesis, Lambda, Programming, Serverless, SNS

AWS offers a wealth of options for implementing messaging patterns such as pub-sub with Lambda, let’s compare and contrast some of these options. The pub-sub pattern Publish-Subscribe (often shortened to pub-sub) is a messaging pattern where publishers and subscribers are decoupled through an intermediary broker (ZeroMQ, RabbitMQ, SNS, etc.). SNS + Lambda In the AWS ecosystem, the obvious …

What is the best event source for doing pub-sub with AWS Lambda? Read More »

Capture and forward correlation IDs through different Lambda event sources

API Gateway, AWS, Kinesis, Lambda, Serverless

Serverless architectures are microservices by default, you need correlation IDs to help debug issues that spans across multiple functions, and possibly different event source types – asynchronous, synchronous and streams. This is the last of a 3-part mini series on managing your AWS Lambda logs. If you haven’t read part 1 yet, please give it a read now. …

Capture and forward correlation IDs through different Lambda event sources Read More »

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close