Many-faced threats to Serverless security

Threats to the secu­ri­ty of our server­less appli­ca­tions take many forms, some are the same old foes we have faced before; some are new; and some have tak­en on new forms in the server­less world.

As we adopt the server­less par­a­digm for build­ing cloud-host­ed appli­ca­tions, we del­e­gate even more of the oper­a­tional respon­si­bil­i­ties to our cloud providers. When you build a server­less archi­tec­ture around AWS Lamb­da you no longer have to con­fig­ure AMIs, patch the OS, and install dae­mons to col­lect and dis­trib­ute logs and met­rics for your appli­ca­tion. AWS takes care all of that for you.

What does this mean for the Shared Respon­si­bil­i­ty Mod­el that has long been the cor­ner­stone of secu­ri­ty in the AWS cloud?

Protection from attacks against the OS

AWS takes over the respon­si­bil­i­ty for main­tain­ing the host OS as part of their core com­pe­ten­cy. Hence alle­vi­at­ing you from the rig­or­ous task of apply­ing all the lat­est secu­ri­ty patch­es — some­thing most of us just don’t do a good enough job of as it’s not our pri­ma­ry focus.

In doing so, it pro­tects us from attacks against known vul­ner­a­bil­i­ties in the OS and pre­vents attacks such as Wan­naCry.

Also, by remov­ing long lived servers from the pic­ture we also removed the threats posed by com­pro­mised servers that live in our envi­ron­ment for a long time.

WannaCry happened because the MS17–017 security patch was not applied to the affected hosts.
Wan­naCry hap­pened because the MS17–017 secu­ri­ty patch was not applied to the affect­ed hosts.

How­ev­er, it is still our respon­si­bil­i­ty to patch our appli­ca­tion and address vul­ner­a­bil­i­ties that exist in our code and our depen­den­cies.

OWASP top 10 is still as relevant as ever

Aside from a few reclassifications the OWASP top 10 list has largely stayed the same in 7 years.
Aside from a few reclas­si­fi­ca­tions the OWASP top 10 list has large­ly stayed the same in 7 years.

A glance at the OWASP top 10 list for 2017 shows us some famil­iar threats — Injec­tion, Cross-Site Script­ing, Sen­si­tive Data Expo­sure, and so on.

A9 — Components with Known Vulnerabilities

When the folks at Snyk looked at a dataset of 1792 data breach­es in 2016 they found that 12 of the top 50 data breach­es were caused by appli­ca­tions using com­po­nents with known vul­ner­a­bil­i­ties.

Fur­ther­more, 77% of the top 5000 URLs from Alexa include at least one vul­ner­a­ble library. This is less sur­pris­ing than it first sound when you con­sid­er that some of the most pop­u­lar front-end js frame­works — eg. jquery, Angu­lar and React — all had known vul­ner­a­bil­i­ties. It high­lights the need to con­tin­u­ous­ly update and patch your depen­den­cies.

How­ev­er, unlike OS patch­es which are stand­alone, trust­ed and easy to apply, secu­ri­ty updates to these 3rd par­ty depen­den­cies are usu­al­ly bun­dled with fea­ture and API changes that need to be inte­grat­ed and test­ed. It makes our life as devel­op­ers dif­fi­cult and it’s yet anoth­er thing we got­ta do when we’re work­ing over­time to ship new fea­tures.

And then there’s the mat­ter of tran­sient depen­den­cies, and boy there are so many of them… If these tran­sient depen­den­cies have vul­ner­a­bil­i­ties then you too are vul­ner­a­ble through your direct depen­den­cies.

https://david-dm.org/request/request?view=tree
https://david-dm.org/request/request?view=tree

Find­ing vul­ner­a­bil­i­ties in our depen­den­cies is hard work and requires con­stant dili­gence, which is why ser­vices such as Snyk is so use­ful. It even comes with a built-in inte­gra­tion with Lamb­da too!

Attacks against NPM publishers

What if the author/publisher of your 3rd party dependency is not who you think they are?
What if the author/publisher of your 3rd par­ty depen­den­cy is not who you think they are?

Just a few weeks ago, a secu­ri­ty boun­ty hunter post­ed this amaz­ing thread on how he man­aged to gain direct push rights to 14% of NPM pack­ages. The list of affect­ed pack­ages include some big names too: debug, request, react, co, express, moment, gulp, mongoose, mysql, bower, browserify, electron, jasmine, cheerio, modernizr, redux and many more. In total, these pack­ages account for 20% of the total num­ber of month­ly down­loads from NPM.

Let that sink in for a moment.


Did he use high­ly sophis­ti­cat­ed meth­ods to cir­cum­vent NPM’s secu­ri­ty?

Nope, it was a com­bi­na­tion of brute force and using known account & cre­den­tial leaks from a num­ber of sources includ­ing Github. In oth­er words, any­one could have pulled these off with very lit­tle research.

It’s hard not to feel let down by these pack­age authors when so many dis­play such a cav­a­lier atti­tude towards secur­ing access to their NPM accounts. I feel my trust in these 3rd par­ty depen­den­cies have been betrayed.

662 users had pass­word «123456», 174 — «123», 124 — «password».

1409 users (1%) used their user­name as their pass­word, in its orig­i­nal form, with­out any mod­i­fi­ca­tions.

11% of users reused their leaked pass­words: 10.6% — direct­ly, and 0.7% — with very minor mod­i­fi­ca­tions.

As I demon­strat­ed in my recent talk on Server­less secu­ri­ty, one can eas­i­ly steal tem­po­rary AWS cre­den­tials from affect­ed Lamb­da func­tions (or EC2-host­ed Node.js appli­ca­tions) with a few lines of code.

Imag­ine then, a sce­nario where an attack­er had man­aged to gain push rights to 14% of all NPM pack­ages. He could pub­lish a patch update to all these pack­ages and steal AWS cre­den­tials at a mas­sive scale.

The stakes are high and it’s quite pos­si­bly the biggest secu­ri­ty threat we face in the server­less world; and it’s equal­ly threat­en­ing to appli­ca­tions host­ed in con­tain­ers or VMs.

The prob­lems and risks with pack­age man­age­ment is not spe­cif­ic to the Node.js ecosys­tem. I have spent most of my career work­ing with .Net tech­nolo­gies and am now work­ing with Scala at Space Ape Games, pack­age man­age­ment has been a chal­lenge every­where. Whether or not you’re talk­ing about Nuget or Maven, or what­ev­er pack­age repos­i­to­ry, you’re always at risk if the authors of your depen­den­cies do not exer­cise the same due dili­gence to secure their accounts as they would their own appli­ca­tions.

Or, per­haps they do…

A1 — Injection & A3 — XSS

SQL injec­tion and oth­er forms of injec­tion attacks are still pos­si­ble in the server­less world, as are cross-site script­ing attacks.

Even if you’re using NoSQL data­bas­es you might not be exempt from injec­tion attacks either. Mon­goDB for instance, expos­es a num­ber of attack vec­tors through its query APIs.

Arguably DynamoDB’s API makes it hard (at least I haven’t heard of a way yet) for an attack­er to orches­trate an injec­tion attack, but you’re still open to oth­er forms of exploits — eg. XSS, and leaked cre­den­tials which grants attack­er access to DynamoDB tables.

Nonethe­less, you should always san­i­tize user inputs, as well as the out­put from your Lamb­da func­tions.

A6 — Sensitive Data Exposure

Along with servers, web frame­works also dis­ap­peared when one migrates to the server­less par­a­digm. These web frame­works have served us well for many years, but they also hand­ed us a loaded gun we can shot our­selves in the foot with.

As Troy Hunt demon­strat­ed at a recent talk at the LDNUG, we can acci­den­tal­ly expose all kinds of sen­si­tive data by acci­den­tal­ly leav­ing direc­to­ry list­ing options ON. From web.config con­tain­ing cre­den­tials (at 35:28) to SQL back­ups files (at 1:17:28)!

With API Gate­way and Lamb­da, acci­den­tal expo­sures like this become very unlike­ly — direc­to­ry list­ing is a “fea­ture” you’d have to imple­ment your­self. It forces you to make very con­scious deci­sions about when to sup­port direc­to­ry list­ing and the answer is prob­a­bly nev­er.

IAM

If your Lamb­da func­tions are com­pro­mised, then the next line of defence is to restrict what these com­pro­mised func­tions can do.

This is why you need to apply the Least Priv­i­lege Prin­ci­ple when con­fig­ur­ing Lamb­da per­mis­sions.

In the Serverless frame­work, the default behav­iour is to use the same IAM role for all func­tions in the ser­vice.

How­ev­er, the serverless.yml spec allows you to spec­i­fy a dif­fer­ent IAM role per func­tion. Although as you can see from the exam­ples it involves a lot more devel­op­ment effort and (from my expe­ri­ence) adds enough fric­tion that almost no one does this…

Apply per-function IAM policies.
Apply per-func­tion IAM poli­cies.

IAM policy not versioned with Lambda

A short­com­ing with the cur­rent Lamb­da + IAM con­fig­u­ra­tion is that IAM poli­cies are not ver­sioned along with the Lamb­da func­tion.

In the sce­nario where you have mul­ti­ple ver­sions of the same func­tion in active use (per­haps with dif­fer­ent alias­es), then it becomes prob­lem­at­ic to add or remove per­mis­sions:

  • adding a new per­mis­sion to a new ver­sion of the func­tion allows old ver­sions of the func­tion addi­tion­al access that they don’t require (and pos­es a vul­ner­a­bil­i­ty)
  • remov­ing an exist­ing per­mis­sion from a new ver­sion of the func­tion can break old ver­sions of the func­tion that still require that per­mis­sion

Since the 1.0 release of the Serverless frame­work this has become less a prob­lem as it no longer use alias­es for stages — instead, each stage is deployed as a sep­a­rate func­tion, eg.

  • service-function-dev
  • service-function-staging
  • service-function-prod

which means it’s far less like­ly that you’ll need to have mul­ti­ple ver­sions of the same func­tion in active use.

I also found (from per­son­al expe­ri­ence) account lev­el iso­la­tion can help mit­i­gate the prob­lems of adding/removing per­mis­sions, and cru­cial­ly, the iso­la­tion also helps com­part­men­talise secu­ri­ty breach­es — eg. a com­pro­mised func­tion run­ning in a non-pro­duc­tion account can­not be used to cause harm in the pro­duc­tion account and impact your users.

We can apply the same idea of bulkheads (which has been popularised in the microservices world by Michael Nygard’s “Release It”) and compartmentalise security breaches at an account level.
We can apply the same idea of bulk­heads (which has been pop­u­larised in the microser­vices world by Michael Nygard’s “Release It”) and com­part­men­talise secu­ri­ty breach­es at an account lev­el.

Delete unused functions

One of the ben­e­fits of the server­less par­a­digm is that you don’t pay for func­tions when they’re not used.

The flip side of this prop­er­ty is that you have less need to remove unused func­tions since they don’t show up on your bill. How­ev­er, these func­tions still exist as attack sur­face, even more so than active­ly used func­tions because they’re less like­ly to be updat­ed and patched. Over time, these unused func­tions can become a hotbed for com­po­nents with known vul­ner­a­bil­i­ties that attack­ers can exploit.

Lambda’s doc­u­men­ta­tions also cites this as one of the best prac­tices.

Delete old Lamb­da func­tions that you are no longer using.

The changing face of DoS attacks

With AWS Lamb­da you are far more like­ly to scale your way out of a DoS attack. How­ev­er, scal­ing your server­less archi­tec­ture aggres­sive­ly to fight a DoS attack with brute force has a sig­nif­i­cant cost impli­ca­tion.

No won­der peo­ple start­ed call­ing DoS attacks against server­less appli­ca­tions Denial of Wal­let (DoW) attacks!

But you can just throt­tle the no. of con­cur­rent invo­ca­tions, right?”

Sure, and you end up with a DoS prob­lem instead… it’s a lose-lose sit­u­a­tion.

AWS recent­ly intro­duced AWS Shield but at the time of writ­ing the pay­ment pro­tec­tion (only if you pay a month­ly flat fee for AWS Shield Advanced) does not cov­er Lamb­da costs incurred dur­ing a DoS attack.

For a monthly flat fee, AWS Shield Advanced gives you cost protection in the event of a DoS attack, but that protection does not cover Lambda yet.
For a month­ly flat fee, AWS Shield Advanced gives you cost pro­tec­tion in the event of a DoS attack, but that pro­tec­tion does not cov­er Lamb­da yet.

Also, Lamb­da has an at-least-once invo­ca­tion pol­i­cy. Accord­ing to the folks at Sun­Gard, this can result in up to 3 (suc­cess­ful) invo­ca­tions. From the arti­cle, the report­ed rate of mul­ti­ple invo­ca­tions is extreme­ly low — 0.02% — but one won­ders if the rate is tied to the load and might man­i­fest itself at a much high­er rate dur­ing a DoS attack.

Taken from the “Run, Lambda, Run” article below.
Tak­en from the “Run, Lamb­da, Run” arti­cle men­tioned above.

Fur­ther­more, you need to con­sid­er how Lamb­da retries failed invo­ca­tions by an asyn­chro­nous source — eg. S3, SNS, SES, Cloud­Watch Events, etc. Offi­cial­ly, these invo­ca­tions are retried twice before they’re sent to the assigned DLQ (if any).

How­ev­er, an analy­sis by the Ops­Ge­nie guys showed that the no. of retries are not carved in stone and can go up to as many as 6 before the invo­ca­tion is sent to the DLQ.

If the DoS attack­er is able to trig­ger failed async invo­ca­tions (per­haps by upload­ing files to S3 that will cause your func­tion to except when attempt­ing to process) then they can sig­nif­i­cant­ly mag­ni­fy the impact of their attack.

All these add up to the poten­tial for the actu­al no. of Lamb­da invo­ca­tions to explode dur­ing a DoS attack. As we dis­cussed ear­li­er, whilst your infra­struc­ture might be able to han­dle the attack, can your wal­let stretch to the same extend? Should you allow it to?

Securing external data

Just a handful of the places you could be storing state outside of your stateless Lambda function.
Just a hand­ful of the places you could be stor­ing state out­side of your state­less Lamb­da func­tion.

Due to the ephemer­al nature of Lamb­da func­tions, chances are all of your func­tions are state­less. More than ever, states are stored in exter­nal sys­tems and we need to secure them both at rest and in-tran­sit.

Com­mu­ni­ca­tion to all AWS ser­vices are via HTTPS and every request needs to be signed and authen­ti­cat­ed. A hand­ful of AWS ser­vices also offer serv­er-side encryp­tion for your data at rest — S3, RDS and Kine­sis streams springs to mind, and Lamb­da has built-in inte­gra­tion with KMS to encrypt your func­tions’ envi­ron­ment vari­ables.

The same dili­gence needs to be applied when stor­ing sen­si­tive data in services/DBs that do not offer built-in encryp­tion — eg. DynamoDB, Elas­tic­search, etc. — and ensure they’re pro­tect­ed at rest. In the case of a data breach, it pro­vides anoth­er lay­er of pro­tec­tion for our users’ data.

We owe our users that much.

Use secure trans­port when trans­mit­ting data to and from ser­vices (both exter­nal and inter­nal ones). If you’re build­ing APIs with API Gate­way and Lamb­da then you’re forced to use HTTPS by default, which is a good thing. How­ev­er, API Gate­way is always pub­licly acces­si­ble and you need to take the nec­es­sary pre­cau­tions to secure access to inter­nal APIs.

You can use API keys but I think it’s bet­ter to use IAM roles. It gives you fine grained con­trol over who can invoke which actions on which resources. Also, using IAM roles spares you from awk­ward con­ver­sa­tions like this:

It’s X’s last day, he prob­a­bly has our API keys on his lap­top some­where, should we rotate the API keys just in case?”

mm.. that’d be a lot of work, X is trust­wor­thy, he’s not gonna do any­thing.”

ok… if you say so… (secret­ly pray X doesn’t lose his lap­top or devel­op a belat­ed grudge against the com­pa­ny)”

For­tu­nate­ly, both can be eas­i­ly con­fig­ured using the Serverless frame­work.

Leaked credentials

Don’t become an unwilling bitcoin miner.
Don’t become an unwill­ing bit­coin min­er.

The inter­net is full of hor­ror sto­ries of devel­op­ers rack­ing up a mas­sive AWS bill after their leaked cre­den­tials are used by cyber-crim­i­nals to mine bit­coins. For every such sto­ry many more would have been affect­ed but choose to be silent (for the same rea­son many secu­ri­ty breach­es are not announced pub­licly as big com­pa­nies do not want to appear incom­pe­tent).

Even with­in my small social cir­cle (*sobs) I have heard 2 such inci­dents, nei­ther were made pub­lic and both result­ed in over $100k worth of dam­ages. For­tu­nate­ly, in both cas­es AWS agreed to cov­er the cost.

I know for a fact that AWS active­ly scan pub­lic Github repos for active AWS cre­den­tials and try to alert you as soon as pos­si­ble. But as some of the above sto­ries men­tioned, even if your cre­den­tials were leaked only for a brief win­dow of time it will not escape the watch­ful gaze of attack­ers. (plus, they still exist in git com­mit his­to­ry unless you rewrite the his­to­ry too, best to deac­ti­vate the cre­den­tials if pos­si­ble).

A good approach to pre­vent AWS cre­den­tial leaks is to use git pre-com­mit hooks as out­lined by this post.

Conclusions

We looked at a num­ber of secu­ri­ty threats to our server­less appli­ca­tions in this post, many of them are the same threats that have plight­ed the soft­ware indus­try for years. All of the OWASP top 10 still apply to us, includ­ing SQL, NoSQL and oth­er forms of injec­tion attacks.

Leaked AWS cre­den­tials remain a major issue and can poten­tial­ly impact any organ­i­sa­tion that uses AWS. Whilst there are quite a few pub­licly report­ed inci­dents, I have a strong feel­ing that the actu­al no. of inci­dents are much much high­er.

We are still respon­si­ble for secur­ing our users’ data both at rest as well as in-tran­sit. API Gate­way is always pub­licly acces­si­ble, so we need to take the nec­es­sary pre­cau­tions to secure access to our inter­nal APIs, prefer­ably with IAM roles. IAM offers fine grained con­trol over who can invoke which actions on your API resources, and make it easy to man­age access when employ­ees come and go.

On a pos­i­tive note, hav­ing AWS take over the respon­si­bil­i­ty for the secu­ri­ty of the host OS gives us a no. of secu­ri­ty ben­e­fits:

  • pro­tec­tion against OS attacks because AWS can do a much bet­ter job of patch­ing known vul­ner­a­bil­i­ties in the OS
  • host OS are ephemer­al which means no long-lived com­pro­mised servers
  • it’s much hard­er to acci­den­tal­ly leave sen­si­tive data exposed by for­get­ting to turn off direc­to­ry list­ing options in a web frame­work — because you no longer need such web frame­works!

DoS attacks have tak­en a new form in the server­less world. Whilst you’re prob­a­bly able to scale your way out of an attack, it’ll still hurt you in the wal­let, hence why DoS attacks have become known in the server­less world as Denial of Wal­let attacks. Lamb­da costs incurred dur­ing a DoS attack is not cov­ered by AWS Shield Advanced at the time of writ­ing, but hope­ful­ly they will be in the near future.

Mean­while, some new attack sur­faces have emerged with AWS Lamb­da:

  • func­tions are often giv­en too much per­mis­sion because they’re not giv­en indi­vid­ual IAM poli­cies tai­lored to their needs, a com­pro­mised func­tion can there­fore do more harm than it might oth­er­wise
  • unused func­tions are often left around for a long time as there is no cost asso­ci­at­ed with them, but attack­ers might be able to exploit them espe­cial­ly if they’re not active­ly main­tained and there­fore are like­ly to con­tain known vul­ner­a­bil­i­ties

Above all, the most wor­ri­some threat for me are attacks against the pack­age authors them­selves. It has been shown that many authors do not take the secu­ri­ty of their accounts seri­ous­ly, and as such endan­gers them­selves as well as the rest of the com­mu­ni­ty that depends on them. It’s impos­si­ble to guard against such attacks and erodes one of the strongest aspect of any soft­ware ecosys­tem — the com­mu­ni­ty behind it.

Once again, peo­ple have proven to be the weak­est link in the secu­ri­ty chain.