Yubl’s road to Serverless architecture – Part 1

Since Yubl’s closure quite a few people have asked about the serverless architecture we ended up with and some of the things we have learnt along the way.

As such, this is the first of a series of posts where I’d share some of the lessons we learnt. However, bear in mind the pace of change in this particular space so some of the challenges/problems we encountered might have been solved by the time you read this.

ps. many aspects of this series is already covered in a talk I gave on Amazon Lambda at Leetspeak this year, you can find the slides and recording of the talk here.

 

From A Monolithic Beginning

Back when I joined Yubl in April I inherited a monolithic Node.js backend running on EC2 instances, with MongoLab (hosted MongoDB) and CloudAMQP (hosted RabbitMQ) thrown into the mix.

yubl-monolith

There were numerous problems with the legacy system, some could be rectified with incremental changes (eg. blue-green deployment) but others required a rethink at an architectural level. Although things look really simple on paper (at the architecture diagram level), all the complexities are hidden inside each of these 3 services and boy, there were complexities!

My first tasks were to work with the ops team to improve the existing deployment pipeline and to draw up a list of characteristics we’d want from our architecture:

  • able to do small, incremental deployments
  • deployments should be fast, and requires no downtime
  • no lock-step deployments
  • features can be deployed independently
  • features are loosely coupled through messages
  • minimise cost for unused resources
  • minimise ops effort

From here we decided on a service-oriented architecture, and Amazon Lambda seemed the perfect tool for the job given the workloads we had:

  • lots of APIs, all HTTPS, no ultra-low latency requirement
  • lots of background tasks, many of which has soft-realtime requirement (eg. distributing post to follower’s timeline)

 

To a Serverless End

It’s suffice to say that we knew the migration was going to be a long road with many challenges along the way, and we wanted to do it incrementally and gradually increase the speed of delivery as we go.

“The lead time to someone saying thank you is the only reputation metric that matters”

– Dan North

The first step of the migration was to make the legacy systems publish state changes in the system (eg. user joined, user A followed user B, etc.) so that we can start building new features on top of the legacy systems.

To do this, we updated the legacy systems to publish events to Kinesis streams.

 

Our general strategy is:

  • build new features on top of these events, which usually have their own data stores (eg. DynamoDB, CloudSearch, S3, BigQuery, etc.) together with background processing pipelines and APIs
  • extract existing features/concepts from the legacy system into services that will run side-by-side
    • these new services will initially be backed by the same shared MongoLab database
    • other services (including the legacy ones) are updated to use hand-crafted API clients to access the encapsulated resources via the new APIs rather than hitting the shared MongoLab database directly
    • once all access to these resources are done via the new APIs, data migration (usually to DynamoDB tables) will commence behind the scenes
  • wherever possible, requests to existing API endpoints are forwarded to the new APIs so that we don’t have to wait for the iOS and Android apps to be updated (which can take weeks) and can start reaping the benefits earlier

 

After 6 months of hard work, my team of 6 backend engineers (including myself) have drastically transformed our backend infrastructure. Amazon was very impressed by the work we were doing with Lambda and in the process of writing up a case study of our work when Yubl was shut down at the whim of our major shareholder.

Here’s an almost complete picture of the architecture we ended up with (some details are omitted for brevity and clarity).

overall

Some interesting stats:

  • 170 Lambda functions running in production
  • roughly 1GB of total deployment package size (after Janitor Lambda cleans up unreferenced versions)
  • Lambda cost was around 5% of what we pay for EC2 for a comparable amount of compute
  • the no. of production deployments increased from 9/month in April to 155 in September

 

For the rest of the series I’ll drill down into specific features, how we utilised various AWS services, and how we tackled the challenges of:

  • centralised logging
  • centralised configuration management
  • distributed tracing with correlation IDs for Lambda functions
  • keeping Lambda functions warm to avoid coldstart penalty
  • auto-scaling AWS resources that do not scale dynamically
  • automatically clean up old Lambda function versions
  • securing sensitive data (eg. mongodb connection string, service credentials, etc.)

I can also explain our strategy for testing, and running/debugging functions locally, and so on. If there’s anything you’d like me to cover in particular, please leave a comment and let me know.

 

Links

Slides and recording of my Lambda talk at LeetSpeak 2016

Serverless – enable caching on query string parameters in API Gateway

Since I started working at Yubl less than 2 weeks ago, I have been doing a lot of work with Amazon API Gateway & Lambda with the help of the Serverless framework. So far that experience has been really great.

One little caveat I ran into was that, it wasn’t clear on how to enable caching on query string and request path parameters. For instance, if I declare in my s-function.json file that my API Gateway endpoint has a query string parameter called query:

"requestParameters": {
    "integration.request.querystring.query": "method.request.querystring.query"
},

When the endpoint is deployed, you will see in the API Gateway console that ‘Caching’ is not enabled:

f0c8f61c-03fc-11e6-8aef-3f2b197f09b6

Which means when I enable caching on my deployed stage, the query string parameter would not be taken into account.

api-gateway-dev-stage

Users visiting the following URLs would get the same cached response back:

    https://my-awesome-service.com/yubl?query=one-awesome-yubl

    https://my-awesome-service.com/yubl?query=another-awesome-yubl

That’s obviously not OK.

Sadly, the same applied to request path parameters too:

"path": "yubl/{query}",
"method": "GET",
"type": "AWS",

07d58dfa-03ff-11e6-9869-46d4de5c13a2

Thankfully, the guys working on Serverless has been very actively in responding to their issues page and one of the team members picked up my ticket and offered some insight on how to make this work.

API Gateway’s own documentation eludes to two Integration parameters called cacheNamespace and cacheKeyParameters. Although, to say they’re poorly documented would be an understatement…

api-gateway-cache-params

So, after some fiddling around:

  1. create an API with cached query string param
  2. deploy it
  3. export as Swagger + API Gateway Extensions
  4. inspect the JSON to see what value is outputted for cacheNamespace and cacheKeyParameters

api-gateway-cache-params-example

Interestingly, when I put the above value – “method.input.params.query” – in my s-function.json I get the following error during deployment:

Invalid cache key parameter specified

what did work though, is the same value I had specified in my requestParameters dictionary, which is “method.request.querystring.query” in this case. ie

"requestParameters": {
    "integration.request.querystring.query": "method.request.querystring.query"
},
"cacheKeyParametes" [
    "method.request.querystring.query"
]

However, applying the same approaches to request path parameters proved fruitless.. I guess we’ll just have to wait for official documentation to be updated to see how it should work, which should be soon by the sound of things!

UPDATE 17/08/2016 : shortly after I posted this I did find a way to get request path parameters working too (which might have been a result of a Serverless update, though I can’t remember now). For request path parameters you need something along the lines of:

"requestParameters": {
    "integration.request.path.otherUserId": "method.request.path.id"
},
"cacheKeyParametes" [
    "method.request.path.id"
]