Write recursive AWS Lambda functions the right way

You may not realise that you can write AWS Lamb­da func­tions in a recur­sive man­ner to per­form long-run­ning tasks. Here’s two tips to help you do it right.

AWS Lamb­da lim­its the max­i­mum exe­cu­tion time of a sin­gle invo­ca­tion to 5 min­utes. Whilst this lim­it might be raised in the future, it’s like­ly that you’ll still have to con­sid­er time­outs for any long-run­ning tasks. For this rea­son, I per­son­al­ly think it’s a good thing that the cur­rent lim­it is too low for many long run­ning tasks?—?it forces you to con­sid­er edge cas­es ear­ly and avoid the trap of think­ing “it should be long enough to do X” with­out con­sid­er­ing pos­si­ble fail­ure modes.

Instead, you should write Lamb­da func­tions that per­form long-run­ning tasks as recur­sive func­tions?—?eg. pro­cess­ing a large S3 file.

Here’s 2 tips to help you do it right.

use context.getRemainingTimeInMillis()

When your func­tion is invoked, the context object allows you to find out how much time is left in the cur­rent invo­ca­tion.

Sup­pose you have an expen­sive task that can be bro­ken into small tasks that can be processed in batch­es. At the end of each batch, use context.getRemainingTimeInMillis() to check if there’s still enough time to keep pro­cess­ing. Oth­er­wise, recurse and pass along the cur­rent posi­tion so the next invo­ca­tion can con­tin­ue from where it left off.

use local state for optimization

Whilst Lamb­da func­tions are ephemer­al by design, con­tain­ers are still reused for opti­miza­tion which means you can still lever­age in-mem­o­ry states that are per­sist­ed through invo­ca­tions.

You should use this oppor­tu­ni­ty to avoid load­ing the same data on each recursion?—?eg. you could be pro­cess­ing a large S3 file and it’s more effi­cient (and cheap­er) to cache the con­tent of the S3 file.

I notice that AWS has also updat­ed their Lamb­da best prac­tices page to advise you to take advan­tage of con­tain­er reuse:

How­ev­er, as Lamb­da can recy­cle the con­tain­er between recur­sions, it’s pos­si­ble for you to lose the cached state from one invo­ca­tion to anoth­er. There­fore, you shouldn’t assume the cached state to always be avail­able dur­ing a recur­sion, and always check if there’s cached state first.

Also, when deal­ing with S3 objects, you need to pro­tect your­self against con­tent changes?—?ie. S3 object is replaced, but con­tain­er instance is still reused so the cache data is still avail­able. When you call S3’s GetO­b­ject oper­a­tion, you should set the option­al If-None-Match para­me­ter with the ETag of the cached data.

Here’s how you can apply this tech­nique.

Have a look at this exam­ple Lamb­da func­tion that recur­sive­ly process­es a S3 file, using the approach out­lined in this post.

Like what you’re read­ing? Check out my video course Pro­duc­tion-Ready Server­less and learn the essen­tials of how to run a server­less appli­ca­tion in pro­duc­tion.

We will cov­er top­ics includ­ing:

  • authen­ti­ca­tion & autho­riza­tion with API Gate­way & Cog­ni­to
  • test­ing & run­ning func­tions local­ly
  • CI/CD
  • log aggre­ga­tion
  • mon­i­tor­ing best prac­tices
  • dis­trib­uted trac­ing with X-Ray
  • track­ing cor­re­la­tion IDs
  • per­for­mance & cost opti­miza­tion
  • error han­dling
  • con­fig man­age­ment
  • canary deploy­ment
  • VPC
  • secu­ri­ty
  • lead­ing prac­tices for Lamb­da, Kine­sis, and API Gate­way

You can also get 40% off the face price with the code ytcui. Hur­ry though, this dis­count is only avail­able while we’re in Manning’s Ear­ly Access Pro­gram (MEAP).