admin Yan Cui

All articles by admin

 

Applying principles of chaos engineering to AWS Lambda with latency injection

Tweet This is part 2 of a multipart series that explores ideas on how we could apply the principles of chaos engineering to serverless architectures built around Lambda functions. part 1: how can we apply principles of chaos engineering to Lambda? part 2: latency injection for APIs part 3: fault injection for Lambda functions The...
Continue reading...

 

How can we apply the principles of chaos engineering to AWS Lambda?

Tweet This is the first of a multipart series that explores ideas on how we could apply the principles of chaos engineering to serverless architectures built around Lambda functions. part 1: how can we apply principles of chaos engineering to Lambda? part 2: latency injection for APIs part 3: fault injection for Lambda functions There’s...
Continue reading...

 

Mind the 75GB limit on AWS Lambda deployment packages

Tweet Gotta clean up those old Lambda deployment packages! With AWS Lambda and the Serverless framework, deploying your code has become so simple and frictionless. As you move more and more of your architecture to run on Lambda, you might find that, in addition to getting things done faster you are also deploying your code more frequently. That’s awesome!...
Continue reading...

 

You should use SSM Parameter Store over Lambda env variables

Tweet AWS Lambda announced native support for environment variables at the end of 2016. But even before that, the Serverless framework had supported environment variables and I was using them happily as me and my team at the time migrated our monolithic Node.js backend to serverless. However, as our architecture expanded we found several drawbacks with managing configurations with environment...
Continue reading...

 

Using Protocol Buffers with API Gateway and AWS Lambda

Tweet AWS announced binary support for API Gateway in late 2016, which opened up the door for you to use more efficient binary formats such as Google’s Protocol Buffers and Apache Thrift. Why? Compared to JSON – which is the bread and butter for APIs built with API Gateway and Lambda – these binary formats can produce significantly smaller payloads....
Continue reading...

 

Capture and forward correlation IDs through different Lambda event sources

Tweet Serverless architectures are microservices by default, you need correlation IDs to help debug issues that spans across multiple functions, and possibly different event source types – asynchronous, synchronous and streams. This is the last of a 3-part mini series on managing your AWS Lambda logs. If you haven’t read part 1 yet, please give it a read...
Continue reading...

 

Tips and tricks for logging and monitoring AWS Lambda functions

Tweet The common practice of using agents/daemons to buffer and batch send logs and metrics are no longer applicable in the world of serverless. Here are some tips to help you get the most out of your logging and monitoring infrastructure for your functions. This is part 2 of a 3-part mini series on managing...
Continue reading...

 

Centralised logging for AWS Lambda

Tweet CloudWatch Logs is hardly the ideal fit for all your logging needs, fortunately you can easily stream the logs to your preferred log aggregation service with AWS Lambda functions. This is the first of a 3-part mini series on managing your AWS Lambda logs. In part 1 we will look at how you can...
Continue reading...

 

Write recursive AWS Lambda functions the right way

Tweet You may not realise that you can write AWS Lambda functions in a recursive manner to perform long-running tasks. Here’s two tips to help you do it right. AWS Lambda limits the maximum execution time of a single invocation to 5 minutes. Whilst this limit might be raised in the future, it’s likely that you’ll...
Continue reading...

 

Many-faced threats to Serverless security

Tweet Threats to the security of our serverless applications take many forms, some are the same old foes we have faced before; some are new; and some have taken on new forms in the serverless world. As we adopt the serverless paradigm for building cloud-hosted applications, we delegate even more of the operational responsibilities to...
Continue reading...

 

The problems with DynamoDB Auto Scaling and how it might be improved

Tweet TL;DR – AWS announced the long awaited auto scaling capability for DynamoDB, but we found it takes too long to scale up and doesn’t scale up aggressively enough as it’s held back by using consumed capacity as scaling metric rather than actual request count. Here at Space Ape Games we developed an in-house tech to auto scale...
Continue reading...

 

Running and debugging AWS Lambda functions locally with the Serverless framework and VS Code

Tweet One of the complaints developers often have for AWS Lambda is the inability to run and debug functions locally. For Node.js at least, the Serverless framework and VS Code provides a good solution for doing just that. An often underused feature of the Serverless framework is the invoke local command, which runs your code locally...
Continue reading...

 

I’m running a live course on designing serverless architecture with AWS Lambda

Tweet Hi everyone, just a quick note to let you know that I’m running a live online course with O’Reilly on designing serverless architectures with AWS Lambda. It’s a 2-day course on September 11-12th with 6 hours in total, and it’s available for free if you have a subscription with SafariBooksOnline. Registration for the course is open till...
Continue reading...

 

Slides for my serverless security talk

Tweet Security in serverless world from Yan Cui Tweet...
Continue reading...

 

Applying the Saga pattern with AWS Lambda and Step Functions

Tweet The Saga pattern is a pattern for managing failures, where each action has a compensating action for rollback. In Hector Garcia-Molina’s 1987 paper, it is described as an approach to handling system failures in a long-running transactions. It has become increasingly relevant in the world of microservices as application logic often needs to transact...
Continue reading...

 

Yubl’s road to Serverless – Part 5 – building better recommendations with Lambda, BigQuery and GrapheneDB

Tweet Note: see here for the rest of the series.   When I joined Yubl in April 2016, it had launched just 2 months earlier, after a long and chaotic development cycle that lasted more than 2 years – all the while there was a fully armed sales team before there was even a product!...
Continue reading...

 

AWS X-Ray and Lambda : the good, the bad and the ugly

Tweet AWS announced general availability of AWS Lambda support for AWS X-Ray back in May. It’s taken me a while to try it out, and whilst I see plenty of values I think its current limitations significantly restricts its usefulness in a complex system. I found Lambda-specific documentations to be fragmented and I had to...
Continue reading...

 

Finding coldstarts : how long does AWS Lambda keep your idle functions around?

Tweet In the last post I compared the coldstart time for Lambda functions with different language, memory and code size. One of the things I learnt was that idle functions are no longer terminated after 5 minutes of inactivity. AWS Lambda – compare coldstart time with different languages, memory and code sizes It is a fantastic...
Continue reading...

 

AWS Lambda – compare coldstart time with different languages, memory and code sizes

Tweet A while back we looked at the performance difference between the language runtimes AWS Lambda supports natively. AWS Lambda – comparing platform performances We intentionally omitted coldstart time from that experiment as we were interested in performance differences when a function is “warm”. However, coldstart is still an important performance consideration, so let’s take...
Continue reading...

 

Beware of dilution of DynamoDB throughput due to excessive scaling

Tweet TL;DR – The no. of partitions in a DynamoDB table goes up in response to increased load or storage size, but it never come back down, ever. DynamoDB is pretty great, but as I have seen this particular problem at 3 different companies – Gamesys, JUST EAT, and now Space Ape Games – I think...
Continue reading...

 

From F# to Scala – implicits

Tweet Note: read the whole series here.   Having looked at case class and extractors recently, the next logical thing would be partial functions. Since Andrea pointed me to a really well article on the subject I don’t think there’s anything else for me to add, so instead, let’s look at Scala’s implicits, which is a very powerful...
Continue reading...

 

Serverless 1.X – enable API Gateway caching on request parameters

Tweet Having previously blogged about the untrodden path to enable caching on API Gateway request parameters in the Serverless framework 0.5.X, it’s a little disappointing that it’s still not officially fixed in the 1.X versions… The Problem The problem is two-fold: there’s currently no way to specify caching should be enabled for path & query...
Continue reading...

 

Yubl’s road to Serverless architecture – Part 4 – building a scalable push notification system

Tweet The Road So Far part 1 : overview part 2 : testing and continuous delivery strategies part 3 : ops   Just before Yubl’s untimely demise we did an interesting piece of work to redesign the system for sending targeted push notifications to our users to improve retention. The old system relied on MixPanel for both selecting users as...
Continue reading...

 

Auto-scaling Kinesis streams with AWS Lambda

Tweet Following on from the last post where we discussed 3 useful tips for working effectively with Lambda and Kinesis, let’s look at how you can use Lambda to help you auto scale Kinesis streams. Auto-scaling for DynamoDB and Kinesis are two of the most frequently requested features for AWS, as I write this post I’m sure the...
Continue reading...

 

AWS Lambda —3 pro tips for working with Kinesis streams

Tweet At Yubl, we arrived at a non-trivial serverless architecture where Lambda and Kinesis became a prominent feature of this architecture. Whilst our experience using Lambda with Kinesis was great in general, there was a couple of lessons that we had to learn along the way. Here are 3 useful tips to help you avoid...
Continue reading...

 

AWS Lambda – build yourself a URL shortener in 2 hours

Tweet An interesting requirement came up at work this week where we discussed potentially having to run our own URL Shortener because the Universal Links mechanism (in iOS 9 and above) requires a JSON manifest at https://domain.com/apple-app-site-association Since the OS doesn’t follow redirects this manifest has to be hosted on the URL shortener’s root domain. Owing to a limitation...
Continue reading...

 

Slides for my AWS user group talk – AWS Lambda from the trenches

Tweet Hello, just a quick note to say thanks to everyone who attended the AWS user group meetup last night, was a pleasure to talk to many of you afterwards. As promised, here are the slides for my talk from last night. AWS Lambda from the trenches from Yan Cui Tweet...
Continue reading...

 

AWS Lambda – comparing platform performances

Tweet As Lambda adds nodejs 6.10 to its supported platforms I wondered if there’s any performance differences between the platforms. Thankfully the templates in the Serverless framework make it a relative breeze to test it out with a simple HelloWorld function.   The Test see the test code here. I created a simple Lambda function for each platform that...
Continue reading...

 

Yubl’s road to Serverless architecture – Part 3 – ops

Tweet Note: see here for the rest of the series.   A couple of folks asked me about our strategy for monitoring, logging, etc. after part 2, and having watched Chris Swan talk about “Serverless Operations is not a Solved Problem” at the Serverless meetup it’s a good time for us to talk about how we approached ops with...
Continue reading...

 

Yubl’s road to Serverless architecture – Part 2 – Testing and CI/CD

Tweet Note: see here for the rest of the series.   Having spoken to quite a few people about using AWS Lambda in production, testing and CI/CD are always high up the list of questions, so I’d like to use this post to discuss the approaches that we took at Yubl. Please keep in mind that this...
Continue reading...

 

From F# to Scala – apply & unapply functions

Tweet Note: read the whole series here.   Last time around we looked at Scala’s Case Class in depth and how it compares to F#’s Discriminated Unions. F# also has Active Patterns, which is a very powerful language feature in its own right. Unsurprisingly, Scala also has something similar in the shape of extractors (via the unapply...
Continue reading...

 

From F# to Scala – case class/object (ADTs)

Tweet Note: read the whole series here.   Continuing on from where we left off with traits last time around, let’s look at Scala’s case class/object which can be used to create Algebraic Data Types (ADTs) in Scala.   Case Class You can declare an ADT in F# using Discriminated Unions (DUs). For example, a binary tree might be represented...
Continue reading...

 

From F# to Scala – traits

Tweet Note: read the whole series here.   Continuing on from where we left off with type inference last time around, let’s look at a language feature in Scala that doesn’t exist in F# – traits. Scala has both abstract classes and traits (think of them as interfaces, but we’ll get into the differences shortly) to support...
Continue reading...

 

From F# to Scala – type inference

Tweet It’s a new year, a new job, a new language (Scala) and runtime (JVM) to work with. After a week of learning about my new surroundings at Space Ape Games and getting up to speed with Scala I have learnt a fair bit. In my own journey (aided by Twitter’s Scala School, Odersky’s Coursera course and...
Continue reading...

 

Year in Review, 2016

Tweet Changes, Changes, Changes On a personal front, 2016 has been a year of great highs and lows. I left JUST EAT in March, and started working at Yubl where I had the pleasure of working with an amazing group of people and had lots of fun building things with AWS Lambda and Node.js. We took...
Continue reading...

 

Advent of Code F# – Day 25

Tweet ps. look out for all my other solutions for Advent of Code challenges here.   Day 25 See details of the challenge here. Today’s input looks like this: cpy a d cpy 7 c cpy 362 b inc d dec b jnz b -2 … At last, we’re at the end of this year’s Advent...
Continue reading...

 

Advent of Code F# – Day 24

Tweet ps. look out for all my other solutions for Advent of Code challenges here.   Day 24 See details of the challenge here. Today’s challenge is a mix of Breadth-First Search (or Level Order Tree Traversal) and the Travelling Salesman Problem. First, let’s parse the input file into a 2D array to make it easier...
Continue reading...

 

Advent of Code F# – Day 23

Tweet ps. look out for all my other solutions for Advent of Code challenges here.   Day 23 See details of the challenge here. Today’s input looks like this: cpy a b dec b cpy a d cpy 0 a cpy b c inc a .. Today’s challenge is an extension to Day 12, where we...
Continue reading...

 

Advent of Code F# – Day 22

Tweet ps. look out for all my other solutions for Advent of Code challenges here.   Day 22 See details of the challenge here. Today’s input looks like this: root@ebhq-gridcenter# df -h Filesystem                           Size    Used     Avail    Use% /dev/grid/node-x0-y0  ...
Continue reading...

 

Advent of Code F# – Day 21

Tweet ps. look out for all my other solutions for Advent of Code challenges here.   Day 21 See details of the challenge here. Today’s input looks like this: swap position 2 with position 7 swap letter f with letter a swap letter c with letter a rotate based on position of letter g rotate based...
Continue reading...

 

Advent of Code F# – Day 20

Tweet ps. look out for all my other solutions for Advent of Code challenges here.   Day 20 See details of the challenge here. Today’s input looks like this: 1873924193-1879728099 2042754084-2076891856 4112318198-4113899685 1039794493-1057170966 3791841434-3797494664 1518668516-1518748952 1946127596-1953926346 4058215215-4086224696 3429681642-3455096313 2599576643-2604275147 1800210010-1801990849 1761160149-1766904471 2774395403-2774748831 1520470679-1542287000 2343327790-2346083217 … First, let’s read the input into a list of tuples representing...
Continue reading...

 

Advent of Code F# – Day 19

Tweet ps. look out for all my other solutions for Advent of Code challenges here.   Day 19 See details of the challenge here. I initially approached today’s challenge as a dynamic programming exercise, but it quickly transpired that there’s a much better way to do it once I realised that part 1 is in fact...
Continue reading...

 

Advent of Code F# – Day 18

Tweet ps. look out for all my other solutions for Advent of Code challenges here.   Day 18 See details of the challenge here. To solve both part 1 and 2:   Links Day 18 challenge description Advent of Code 2015 Solution for Day 17 All my F# solutions for Advent of Code Github repo Tweet...
Continue reading...

 

Advent of Code F# – Day 17

Tweet ps. look out for all my other solutions for Advent of Code challenges here.   Day 17 See details of the challenge here. First, let’s add a hash function (that returns the MD5 as a hexadecimal string as we have done so often this year). Then, add a step function that’ll take a (x, y) position and the...
Continue reading...

 

Advent of Code F# – Day 16

Tweet ps. look out for all my other solutions for Advent of Code challenges here.   Day 16 See details of the challenge here. I also tried an implememtation using string instead of bool[] and there is no distinguishable difference in terms of performance. Now that we have the solve function we can answer both part...
Continue reading...

 

Advent of Code F# – Day 15

Tweet ps. look out for all my other solutions for Advent of Code challenges here.   Day 15 See details of the challenge here. Today’s challenge is pretty straight forward, and a brute force approach would have suffice – try every t from 0 to inifinity and return the first t that satisfies this equation for...
Continue reading...

 

Advent of Code F# – Day 14

Tweet ps. look out for all my other solutions for Advent of Code challenges here.   Day 14 See details of the challenge here. Today’s challenge is very similar to that of Day 5, but the requirements for a valid hash value is different this time. As before, we’ll start by defining a hash function that will accept a string and...
Continue reading...

 

Advent of Code F# – Day 13

Tweet ps. look out for all my other solutions for Advent of Code challenges here.   Day 13 See details of the challenge here. Today’s challenge involves solving two separate problems: counting set bits in an integer level order tree traversal For the first problem, I recommend reading through this blog post which covers several approaches and optimizations for...
Continue reading...

 

Equilibrium Index problem in F#

Tweet A good friend pointed me to a nice coding challenge she encountered – finding the Equilibrium Index of an array in O(n) time. After a quick discussion we ended up with 2 solutions, and naturally I had to code them up in F#   Solution 1 This is the same approach to the one you saw...
Continue reading...

 

Advent of Code F# – Day 12

Tweet ps. look out for all my other solutions for Advent of Code challenges here.   Day 12 See details of the challenge here. The input for today’s challenge looks like this: cpy 1 a cpy 1 b cpy 26 d jnz c 2 jnz 1 5 cpy 7 c … This is very similar to Day...
Continue reading...