Kinesis

How to include SNS and Kinesis in your e2e tests

Update 17/09/2019: based on feedback, I added a few more alternatives that do not require you to deploy additional Lambda functions to the e2e-test stage. Scroll to the end to find out more. Being event-driven is a key characteristic of a serverless architecture. Often our Lambda functions would publish events to SNS topics or Kinesis …

How to include SNS and Kinesis in your e2e tests Read More »

How to connect SNS to Kinesis for cross-account delivery via API Gateway

At DAZN (where I no longer work), the teams work with a number of third-party providers. They often have to synchronize data between different AWS accounts. SNS to SQS is the primary mechanism for these cross-account deliveries because: it was an established pattern within the organization DAZN engineers and third-party engineers are both familiar with …

How to connect SNS to Kinesis for cross-account delivery via API Gateway Read More »

A self-healing Kinesis function that adapts its throughput based on performance

I spent time with a client this week to solve an interesting problem – to adjust the number of concurrent requests to a downstream service dynamically based on response time and error rate. This is a common challenge when integrating with third parties, so we decided to share our approach so others might benefit from …

A self-healing Kinesis function that adapts its throughput based on performance Read More »

Centralised logging for AWS Lambda, REVISED (2018)

First of all, I would like to thank all of you for following and reading my content. My post on centralised logging for AWS Lambda has been viewed more than 20K times by now, so it is clearly a challenge that many of you have run into. In the post, I outlined an approach of …

Centralised logging for AWS Lambda, REVISED (2018) Read More »

What is the best event source for doing pub-sub with AWS Lambda?

AWS offers a wealth of options for implementing messaging patterns such as pub-sub with Lambda, let’s compare and contrast some of these options. The pub-sub pattern Publish-Subscribe (often shortened to pub-sub) is a messaging pattern where publishers and subscribers are decoupled through an intermediary broker (ZeroMQ, RabbitMQ, SNS, etc.). SNS + Lambda In the AWS ecosystem, the obvious …

What is the best event source for doing pub-sub with AWS Lambda? Read More »

Capture and forward correlation IDs through different Lambda event sources

Serverless architectures are microservices by default, you need correlation IDs to help debug issues that spans across multiple functions, and possibly different event source types – asynchronous, synchronous and streams. This is the last of a 3-part mini series on managing your AWS Lambda logs. If you haven’t read part 1 yet, please give it a read now. …

Capture and forward correlation IDs through different Lambda event sources Read More »

Auto-scaling Kinesis streams with AWS Lambda

Following on from the last post where we discussed 3 useful tips for working effectively with Lambda and Kinesis, let’s look at how you can use Lambda to help you auto scale Kinesis streams. Auto-scaling for DynamoDB and Kinesis are two of the most frequently requested features for AWS, as I write this post I’m sure the folks …

Auto-scaling Kinesis streams with AWS Lambda Read More »

AWS Lambda —3 pro tips for working with Kinesis streams

At Yubl, we arrived at a non-trivial serverless architecture where Lambda and Kinesis became a prominent feature of this architecture. Whilst our experience using Lambda with Kinesis was great in general, there was a couple of lessons that we had to learn along the way. Here are 3 useful tips to help you avoid some …

AWS Lambda —3 pro tips for working with Kinesis streams Read More »

New releases – DynamoDB.SQL and Darkseid

Hi, just a quick update on two of my libraries aimed at making AWS easier to work with from .Net.   DynamoDB.SQL DynamoDB.SQL is a SQL-like external DSL for querying & scanning data in Amazon DynamoDB. Version 3.0.0 has been released, which moves away from the monolithic .Net AWSSDK (v2.x.x), and onto the DynamoDB specific package. You can …

New releases – DynamoDB.SQL and Darkseid Read More »

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close