Centralised logging for AWS Lambda

[update 23/07/2018]: The approach out­lined in this post works well when you start out, but you can run into con­cur­ren­cy relat­ed issues at scale. To see a bet­ter approach when run­ning at scale, please see the revised ver­sion of this post here.


Cloud­Watch Logs is hard­ly the ide­al fit for all your log­ging needs, for­tu­nate­ly you can eas­i­ly stream the logs to your pre­ferred log aggre­ga­tion ser­vice with AWS Lamb­da func­tions.

This is the first of a 3-part mini series on man­ag­ing your AWS Lamb­da logs. In part 1 we will look at how you can get all of your logs off Cloud­Watch.

Part 2 will help you bet­ter under­stand the trade­offs with dif­fer­ent approach­es to log­ging & mon­i­tor­ing, with some help­ful tips and tricks that I have come across.

Part 3 will demon­strate how to cap­ture and for­ward cor­re­la­tion IDs through var­i­ous event sources — eg. API Gate­way, SNS and Kine­sis.

part 2 : tips and tricks

part 3 : track­ing cor­re­la­tion IDs

AWS Lambda logging basics

Dur­ing the exe­cu­tion of a Lamb­da func­tion, what­ev­er you write to std­out (eg. using console.log in Node.js) will be cap­tured by Lamb­da and sent to Cloud­Watch Logs asyn­chro­nous­ly in the back­ground, with­out adding any over­head to your func­tion exe­cu­tion time.

You can find all the logs for your Lamb­da func­tions in Cloud­Watch Logs, organ­ised into log groups (one log group per func­tion) and then log streams (one log stream per con­tain­er instance).

You could, of course, send these logs to Cloud­Watch Logs your­self via the Put­Lo­gEvents oper­a­tion, or send them to your pre­ferred log aggre­ga­tion ser­vice such as Splunk or Elas­tic­search. But, remem­ber that every­thing has to be done dur­ing a function’s invo­ca­tion. If you’re mak­ing addi­tion­al net­work calls dur­ing the invo­ca­tion then you’ll pay for those addi­tion­al exe­cu­tion time, and your users would have to wait that much longer for the API to respond.

So, don’t do that!

Instead, process the logs from Cloud­Watch Logs after the fact.

Streaming CloudWatch Logs

In the Cloud­Watch Logs con­sole, you can select a log group (one for each Lamb­da func­tion) and choose to stream the data direct­ly to Amazon’s host­ed Elas­tic­search ser­vice.

This is very use­ful if you’re using the host­ed Elas­tic­search ser­vice already. But if you’re still eval­u­at­ing your options, then give this post a read before you decide on the AWS-host­ed Elas­tic­search.

As you can see from the screen­shot above, you can also choose to stream the logs to a Lamb­da func­tion instead. In fact, when you cre­ate a new func­tion from the Lamb­da con­sole, there’s a num­ber of blue­prints for push­ing Cloud­Watch Logs to oth­er log aggre­ga­tion ser­vices already.

Clear­ly this is some­thing a lot of AWS’s cus­tomers have asked for.

You can find blue­prints for ship­ping Cloud­Watch Logs to Sumo­log­ic, Splunk and Log­gly out of the box.

So that’s great, now you can use these blue­prints to help you write a Lamb­da func­tion that’ll ship Cloud­Watch Logs to your pre­ferred log aggre­ga­tion ser­vice. But here are a few things to keep in mind.

Auto-subscribe new log groups

When­ev­er you cre­ate a new Lamb­da func­tion, it’ll cre­ate a new log group in Cloud­Watch logs. You want to avoid a man­u­al process for sub­scrib­ing log groups to your ship-logs func­tion above.

Instead, enable Cloud­Trail, and then set­up an event pat­tern in Cloud­Watch Events to invoke anoth­er Lamb­da func­tion when­ev­er a log group is cre­at­ed.

You can do this one-off set­up in the Cloud­Watch con­sole man­u­al­ly.

Match the Cre­ateL­og­Group API call in Cloud­Watch Logs and trig­ger a sub­scribe-log-group Lamb­da func­tion to sub­scribe the new­ly cre­at­ed log group to the ship-logs func­tion you cre­at­ed ear­li­er.

If you’re work­ing with mul­ti­ple AWS accounts, then you should avoid mak­ing the set­up a man­u­al process. With the Server­less frame­work, you can set­up the event source for this subscribe-log-group func­tion in the serverless.yml file.

Anoth­er thing to keep in mind is that, you need to avoid sub­scrib­ing the log group for the ship-logs func­tion to itself — it’ll cre­ate an infi­nite invo­ca­tion loop and that’s a painful les­son that you want to avoid.

Auto-setting the log retention policy

By default, when Lamb­da cre­ates a new log group for your func­tion the reten­tion pol­i­cy is to keep them for­ev­er. Under­stand­ably this is overkill and the cost of stor­ing all these logs can add up over time. 

By default, logs for your Lamb­da func­tions are kept for­ev­er

For­tu­nate­ly, using the same tech­nique above we can add anoth­er Lamb­da func­tion to auto­mat­i­cal­ly update the reten­tion pol­i­cy to some­thing more rea­son­able.

Here’s a Lamb­da func­tion for auto-updat­ing the log reten­tion pol­i­cy to 30 days.

Taking care of existing log groups

If you already have lots of exist­ing log groups, then con­sid­er wrap­ping the demo code (below) for auto-sub­scrib­ing log groups and auto-updat­ing log reten­tion pol­i­cy into a one-off script to update them all.

You can do this by recurs­ing through all log groups with the DescribeL­og­Groups API call, and then invoke the cor­re­spond­ing func­tions for each log group.

You can find exam­ple code in this repo.

Like what you’re read­ing? Check out my video course Pro­duc­tion-Ready Server­less and learn the essen­tials of how to run a server­less appli­ca­tion in pro­duc­tion.

We will cov­er top­ics includ­ing:

  • authen­ti­ca­tion & autho­riza­tion with API Gate­way & Cog­ni­to
  • test­ing & run­ning func­tions local­ly
  • CI/CD
  • log aggre­ga­tion
  • mon­i­tor­ing best prac­tices
  • dis­trib­uted trac­ing with X-Ray
  • track­ing cor­re­la­tion IDs
  • per­for­mance & cost opti­miza­tion
  • error han­dling
  • con­fig man­age­ment
  • canary deploy­ment
  • VPC
  • secu­ri­ty
  • lead­ing prac­tices for Lamb­da, Kine­sis, and API Gate­way

You can also get 40% off the face price with the code ytcui. Hur­ry though, this dis­count is only avail­able while we’re in Manning’s Ear­ly Access Pro­gram (MEAP).