AWS Lambda — janitor-lambda function to clean up old deployment packages

When work­ing with AWS Lamb­da, one of the things to keep in mind is that there’s a per region lim­it of 75GB total size for all deploy­ment pack­ages. Whilst that sounds a lot at first glance, our small team of serv­er engi­neers man­aged to rack up near­ly 20GB of deploy­ment pack­ages in just over 3 months!

Whilst we have been mind­ful of deploy­ment pack­age size (because it affects cold start time) and heav­i­ly using Server­less’s built-in mech­a­nism to exclude npm pack­ages that are not used by each of the func­tions, the sim­ple fact that deploy­ment is sim­ple and fast means we’re doing A LOT OF DEPLOYMENTS.

Indi­vid­u­al­ly, most of our func­tions are sub-2MB, but many func­tions are deployed so often that in some cas­es there are more than 300 deployed ver­sions! This is down to how the Server­less frame­work deploy func­tions — by pub­lish­ing a new ver­sion each time. On its own, it’s not a prob­lem, but unless you clean up the old deploy­ment pack­ages you’ll even­tu­al­ly run into the 75GB lim­it.

 

Some read­ers might have heard of Netflix’s Jan­i­tor Mon­key, which cleans up unused resources in your envi­ron­ment — instance, ASG,  EBS vol­umes, EBS snap­shots, etc.

Tak­ing a leaf out of Netflix’s book, we wrote a Lamb­da func­tion which finds and deletes old ver­sions of your func­tions that are not ref­er­enced by an alias — remem­ber, Server­less uses alias­es to imple­ment the con­cept of stages in Lamb­da, so not being ref­er­enced by an alias essen­tial­ly equates to an orphaned ver­sion.

janitor-lambda

At the time of writ­ing, we have just over 100 Lamb­da func­tions in our devel­op­ment envi­ron­ment and around 50 run­ning in pro­duc­tion. After deploy­ing the jan­i­tor-lamb­da func­tion, we have cut the code stor­age size down to 1.1GB, which include only the cur­rent ver­sion of deploy­ments for all our stages (we have 4 non-prod stages in this account).

lambda-console

side­bar: if you’d like to hear more about our expe­ri­ence with Lamb­da thus far and what we have been doing, then check out the slides from my talk on the mat­ter, I’d be hap­py to write them up in more details too when I have more free time.

 

Janitor-Lambda

With­out fur­ther ado, here’s the bulk of our jan­i­tor func­tion:

Since AWS Lamb­da throt­tles you on the no. of APIs calls per minute, we had to store the list of func­tions in the func­tions vari­able so that it car­ries over mul­ti­ple invo­ca­tions.

When we hit the (almost) inevitable throt­tle excep­tion, the cur­rent invo­ca­tion will end, and any func­tions that haven’t been com­plete­ly cleaned will be cleaned the next time the func­tion is invoked.

Anoth­er thing to keep in mind is that, when using a Cloud­Watch Event as the source of your func­tion, Ama­zon will retry your func­tion up to 2 more times on fail­ure. In this case, if the func­tion is retried straight away, it’ll just get throt­tled again. Hence why in the han­dler we log and swal­low any excep­tions:

I hope you have found this post use­ful, let me know in the com­ments if you have any Lamb­da/Server­less relat­ed ques­tions.

Like what you’re read­ing? Check out my video course Pro­duc­tion-Ready Server­less and learn the essen­tials of how to run a server­less appli­ca­tion in pro­duc­tion.

We will cov­er top­ics includ­ing:

  • authen­ti­ca­tion & autho­riza­tion with API Gate­way & Cog­ni­to
  • test­ing & run­ning func­tions local­ly
  • CI/CD
  • log aggre­ga­tion
  • mon­i­tor­ing best prac­tices
  • dis­trib­uted trac­ing with X-Ray
  • track­ing cor­re­la­tion IDs
  • per­for­mance & cost opti­miza­tion
  • error han­dling
  • con­fig man­age­ment
  • canary deploy­ment
  • VPC
  • secu­ri­ty
  • lead­ing prac­tices for Lamb­da, Kine­sis, and API Gate­way

You can also get 40% off the face price with the code ytcui. Hur­ry though, this dis­count is only avail­able while we’re in Manning’s Ear­ly Access Pro­gram (MEAP).