AWS Lambda — comparing platform performances

As Lamb­da adds node­js 6.10 to its sup­port­ed plat­forms I won­dered if there’s any per­for­mance dif­fer­ences between the plat­forms. Thank­ful­ly the tem­plates in the Server­less frame­work make it a rel­a­tive breeze to test it out with a sim­ple Hel­loWorld func­tion.

 

The Test

see the test code here.

I cre­at­ed a sim­ple Lamb­da func­tion for each plat­form that will respond to an API Gate­way event and return “hel­lo”. This is the node­js ver­sion.

I decid­ed to use API Gate­way as the trig­ger as it allows me to invoke the func­tion and apply a con­stant load using stan­dard load test­ing tools for HTTP. I chose Artillery because you can get going with min­i­mal fuzz and I had used it before.

For each plat­form, I ran a test with 10 vir­tu­al users send­ing 1 request per sec­ond (ie. a total of 10 req/s) for an hour.

artillery quick dura­tion 3600 rate 10 -n 1 http://my.lambda.backed.api/

Since we’re inter­est­ed in the per­for­mance char­ac­ter­is­tics of the dif­fer­ent Lamb­da plat­forms, we’ll only be look­ing at the func­tion Dura­tion met­ric, and we’ll ignore the ini­tial cold start times.

 

Observation 1 — C# is slower?

Unsur­pris­ing­ly the invo­ca­tion dura­tion is fair­ly con­sis­tent across the func­tions, although C# is stick­ing out like a sore thumb.

Take this 10 mins win­dow for instance — where there were no spikes that looked like cold starts — the C# plat­form is con­sis­tent­ly high­er than the rest.

 

Observation 2 — Java has very consistent performance

If you look at the max dura­tion for the same 10 mins win­dow — for what­ev­er rea­son, I didn’t get any per­centile met­rics from Cloud­Watch for the entire dura­tion of the test so had to set­tle for max instead — the Java plat­form was both low­er and had less vari­ance, by some dis­tance.

If you com­pare the aver­age and max dura­tion for the Java plat­form over a longer time win­dow, you’ll also see that there’s very lit­tle dif­fer­ence between the two (if you ignore the spike at 01:38 which might be down to GC pause as opposed to cold start) which sug­gests the per­for­mance of the Java plat­form is very con­sis­tent.

 

Observation 3 — static languages has more consistent performance?

Fol­low­ing on from the pre­vi­ous obser­va­tion, it seems that both C# and Java shows less vari­ance when it comes to max dura­tion, so per­haps it’s because both are com­piled lan­guages?

 

Observation 4 — Java packages are big…

One of the ben­e­fits with using node­js and Python to write Lamb­da func­tions is that they pro­duce much small­er pack­ages, which we know trans­lates to low­er code start time. Now, the fault might lie with the Server­less tem­plate for aws-java-maven, but my Hel­loWorld Java exam­ple pro­duces a whoop­ing 2MB pack­age, which is orders of mag­ni­tude big­ger than the node­js and Python func­tions. I expect­ed it to be big­ger than node­js, but per­haps clos­er to the size of the C# pack­age.

 

Conclusions

Take these results with a pinch of salt. Things are evolv­ing at an incred­i­ble pace and what­ev­er per­for­mance dis­crep­an­cies we’re see­ing today can change quick­ly as AWS improves all the plat­forms behind the scenes.

Even as I observe that the C# plat­form appears to be slow­er in this test, we’re talk­ing about sub-mil­lisec­ond dif­fer­ence for a Hel­loWorld exam­ple, hard­ly rep­re­sen­ta­tive of a real world appli­ca­tion. The Dot­Net­Core plat­form itself (which C# Lamb­da func­tions run on) is also evolv­ing quick­ly, and any future per­for­mance improve­ments in that under­ly­ing plat­form will be trans­ferred to you at no cost, so don’t let this post dis­suade you from writ­ing Lamb­da func­tions in C#.