Mind the 75GB limit on AWS Lambda deployment packages

Got­ta clean up those old Lamb­da deploy­ment pack­ages!

With AWS Lamb­da and the Server­less frame­work, deploy­ing your code has become so sim­ple and fric­tion­less.

As you move more and more of your archi­tec­ture to run on Lamb­da, you might find that, in addi­tion to get­ting things done faster you are also deploy­ing your code more fre­quent­ly.

That’s awe­some!

But, as you rejoice in this new found super­pow­er to make your users and stake­hold­ers hap­py, you need to keep an eye out for that region­al lim­it of 75GB for all the uploaded deploy­ment pack­ages.

http://docs.aws.amazon.com/lambda/latest/dg/limits.html

At Yubl, me and a small team of 6 serv­er engi­neers man­aged to rack up near­ly 20GB of deploy­ment pack­ages in 3 months.

We wrote all of our Lamb­da func­tions in Node­js, and deploy­ment pack­ages were typ­i­cal­ly less than 2MB. But the fre­quen­cy of deploy­ments made sure that the over­all size of deploy­ment pack­ages went up steadi­ly.

Now that I’m writ­ing most of my Lamb­da func­tions in Scala (it’s the weapon of choice for the Space Ape Games serv­er team), I’m deal­ing with deploy­ment pack­ages that are sig­nif­i­cant­ly big­ger!

When author­ing Lamb­da func­tions in Java, be pre­pared to sig­nif­i­cant­ly big­ger deploy­ment pack­ages.

Serverless framework: disable versionFunctions

By default, the Server­less frame­work would cre­ate a new ver­sion of your func­tion every time you deploy.

In Server­less 0.X, this is (kin­da) need­ed because it used func­tion alias. For exam­ple, I can have mul­ti­ple deploy­ment stages for the same func­tion?—?devstaging and production. But in the Lamb­da con­sole there is only one func­tion, and each stage is sim­ply an alias point­ing to a dif­fer­ent ver­sion of the same func­tion.

Unfor­tu­nate­ly this behav­iour also made it dif­fi­cult to man­age the IAM per­mis­sions because mul­ti­ple ver­sions of the same func­tion share the same IAM role. Since you can’t ver­sion the IAM role with the func­tion, this makes it hard for you to add or remove per­mis­sions with­out break­ing old­er ver­sions.

For­tu­nate­ly, the devel­op­ers lis­tened to the com­mu­ni­ty and since the 1.0 release each stage is deployed as a sep­a­rate func­tion.

Essen­tial­ly, this allows you to “ver­sion” IAM roles with deploy­ment stages since each stage gets a sep­a­rate IAM role. So there’s tech­ni­cal­ly no need for you to cre­ate a new ver­sion for every deploy­ment any­more. But, that is still the default behav­iour, unless you explic­it­ly dis­able it in your serverless.ymlby set­ting versionFunctions to false.

You might argue that hav­ing old ver­sions of the func­tion in pro­duc­tion makes it quick­er to roll­back.

In that case, enable it for the pro­duc­tion stage only. To do that, here’s a handy trick to allow a default con­fig­u­ra­tion in your serverless.yml to be over­rid­able by deploy­ment stage.

In my per­son­al expe­ri­ence though, unless you have tak­en great care and used alias­es to tag the pro­duc­tion releas­es it’s actu­al­ly quite hard to know which ver­sion cor­re­lates to what. Assum­ing that you have repro­ducible builds, I would have much more con­fi­dence if we roll­back by deploy­ing from a hotfixor support branch of our code.


Clean up old versions with janitor-lambda

If dis­abling versionFunctions in the serverless.yml for all of your projects is hard to enforce, anoth­er approach would be to retroac­tive­ly delete old ver­sions of func­tions that are no longer ref­er­enced by an alias.

To do that, you can cre­ate a cron job (ie. sched­uled Cloud­Watch event + Lamb­da) that will scan through your func­tions and look for ver­sions that are not ref­er­enced and delete them.

I took some inspi­ra­tion from Netflix’s Jan­i­tor Mon­key and cre­at­ed a Jan­i­tor Lamb­da func­tion that you can deploy to your AWS envi­ron­ment to clean unused ver­sions of your func­tions.

After we employed this Jan­i­tor Lamb­da func­tion, our total deploy­ment pack­age went from 20GB to ~1GB (we had a lot of func­tions…).