Takeaways from “Simplifying the Future” by Adrian Cockcroft

Simplifying things in our daily lives

Life is com­pli­cat­ed… but we use sim­ple abstrac­tions to deal with it.”

- Adri­an Cock­croft

When peo­ple say “it’s too com­pli­cat­ed”, what they usu­al­ly mean is “there are too many mov­ing parts and I can’t fig­ure out what it’s going to do next, that I haven’t fig­ured out an inter­nal mod­el for how it works and what it does”.

Which bags the ques­tion: “what’s the most com­pli­cat­ed thing that you can deal with intu­itive­ly?”

Dri­ving, for instance, is one of the most com­pli­cat­ed things that we have to do on a reg­u­lar basis. It com­bines hand-eye-feet coor­di­na­tion, nav­i­ga­tion skills, and abil­i­ty to react to unfore­see­able sce­nar­ios that can be life-or-death.

 

A good exam­ple of a sim­ple abstrac­tion is the touch-based inter­face you find on smart phones and pads. Kids can dis­sim­u­late the work­ing of an iPad by exper­i­ment­ing with it, with­out need­ing any for­mal train­ing because they can inter­act with them and get instant feed­back which helps them build the men­tal mod­el of how things work.

As engi­neers, we should inspire to build things that can be giv­en to 2 year olds and they can intu­itive­ly under­stand how they oper­ate. This last point reminds me of what Brett Vic­tor has been say­ing for years, with inspi­ra­tional talks such as Invent­ing on Prin­ci­ple and Stop Draw­ing Dead Fish.

Net­flix for instance, has invest­ed much effort in intu­ition engi­neer­ing and are build­ing tools to help peo­ple get a bet­ter intu­itive under­stand­ing of how their com­plex, dis­trib­uted sys­tems are oper­at­ing at any moment in time.

Anoth­er exam­ple of how you can take com­plex things and give them sim­ple descrip­tions is XKCD’s Thing Explain­er, which uses sim­ple words to explain oth­er­wise com­plex things such as the Inter­na­tion­al Space Sta­tion, Nuclear Reac­tor and Data Cen­tre.

side­bar: wrt to com­plex­i­ties in code, here are two talks that you might also find inter­est­ing

 

Simplifying work

Adri­an men­tioned Netflix’s slide deck on their cul­ture and val­ues:

Inten­tion­al cul­ture is becom­ing an impor­tant thing, and oth­er com­pa­nies have fol­lowed suit, eg.

It con­di­tions peo­ple join­ing the com­pa­ny on what they would expect to see once they’re onboard­ed, and helps frame & stan­dard­ise the recruit­ment process so that every­one knows what a ‘good’ hire looks like.

If you’re cre­at­ing a start­up you can set the cul­ture from the start, don’t wait until you have acci­den­tal cul­ture, be inten­tion­al and ear­ly about what you want to have.

 

This cre­ates a pur­pose-dri­ven cul­ture.

Be clear and explic­it about the pur­pose and let peo­ple work out how best to imple­ment that pur­pose.

Pur­pos­es are sim­ple state­ments, where­as set­ting out all the indi­vid­ual process­es you need to ensure peo­ple build the right things are much hard­er, it’s sim­pler to have a pur­pose-dri­ven cul­ture and let peo­ple self-organ­ise around those pur­pos­es.


Net­flix also found that if you impose process­es on peo­ple then you dri­ve tal­ents away, which is a big prob­lem. Time and again, Net­flix found that peo­ple pro­duce a frac­tion of what they’re capa­ble of pro­duc­ing at oth­er places because they were held back by process­es, rules and oth­er things that slow them down.

On Reverse Conway’s Law, Adri­an said that you should start with an organ­i­sa­tion­al struc­ture that’s cel­lu­lar in nature, with clear respon­si­bil­i­ties and own­er­ship for a no. of small, co-locat­ed teams — high trust & high cohe­sion with­in the team, and low trust across the teams.

The morale here is that, if you build a com­pa­ny around a pur­pose-dri­ven, sys­tems-think­ing approach then you are build­ing organ­i­sa­tions that are flex­i­ble and can evolve as the tech­nol­o­gy moves on.

The more rules you put in, and the more com­plex and rigid it gets, then you end up with the oppo­site.

You build it, you run it”

- Wern­er Vogel, Ama­zon CTO

 

Simplifying the things we build

First, you should shift your think­ing from projects to prod­ucts, the key dif­fer­ence is that where­as a project has a start and end, a prod­uct will con­tin­ue to evolve for as long as it still serves a pur­pose. On this point, see also:

I’m sor­ry, but there real­ly isn’t any project man­agers in this new mod­el”

- Adri­an Cock­croft

As a result, the over­head & ratio of devel­op­ers to peo­ple doing man­age­ment, releas­es & ops has to change.

 

Sec­ond, the most impor­tant met­ric to opti­mise for is time to val­ue. (see also “beyond fea­tures” by Dan North)

The lead time to some­one say­ing thank you is the only rep­u­ta­tion met­ric that mat­ters”

- Dan North

Look­ing at the cus­tomer val­ues and work­ing out how to improve the time-to-val­ue is an inter­est­ing chal­lenge. (see Simon Wardley’s val­ue-chain-map­ping)

And last­ly, and this is a sub­tle point — opti­mise for the cus­tomers that you want to have rather than the cus­tomers you have now. Which is an inter­est­ing twist on how we often think about reten­tion and mon­eti­sa­tion.

For Net­flix, their opti­mi­sa­tion is almost always around con­vert­ing free tri­als to pay­ing cus­tomers, which means they’re always opti­mis­ing for peo­ple who haven’t seen the prod­uct before. Inter­est­ing­ly, this feed­back loop also has the side-effect of forc­ing the prod­uct to be sim­ple.

On the oth­er hand, if you opti­mise for pow­er users, then you’re like­ly to intro­duce more and more fea­tures that con­tribute towards the prod­uct being too com­pli­cat­ed for new users. You can poten­tial­ly build your­self into a cor­ner where you strug­gle to attract new users and become vul­ner­a­ble to a new com­ers into the mar­ket with sim­pler prod­ucts that new users can under­stand.

 

Mono­lith­ic apps only look sim­ple from the out­side (at the archi­tect dia­gram lev­el), but if you look under the cov­er to see your object depen­den­cies then the true scale of their com­plex­i­ties start to become appar­ent. And they often are com­pli­cat­ed because it requires dis­ci­pline to enforce clear sep­a­ra­tions.

If you require con­stant dili­gence, then you’re set­ting every­one up for fail­ure and hurt.”

- Bri­an Hunter

Microser­vices enforce sep­a­ra­tion that makes them less com­pli­cat­ed, and make those con­nec­tiv­i­ties between com­po­nents explic­it. They are also bet­ter for on-board­ing as new join­ers don’t have to under­stand all the inter­de­pen­den­cies (inside a mono­lith) that encom­pass your entire sys­tem to make even small changes.

Each micro-ser­vice should have a clear, well-defined set of respon­si­bil­i­ties and there’s a cap on the lev­el of com­plex­i­ties they can reach.

side­bar: the best answers I have heard for “how small should a microser­vice be?” are:

  • one that can be com­plete­ly rewrit­ten in 2 weeks”
  • what can fit inside an engineer’s head” — which Psy­chol­o­gy tells us, isn’t a lot ;-)

 

Mon­i­tor­ing used to be one of the things that made microser­vices com­pli­cat­ed, but the tool­ing has caught up in this space and nowa­days many ven­dors (such as NewRel­ic) offer tools that sup­port this style of archi­tec­ture out of the box.

 

Simplifying microservices architecture

If your sys­tem is deployed glob­al­ly, then hav­ing the same, auto­mat­ed deploy­ment for every region gives you sym­me­try. Hav­ing this com­mon­al­i­ty (same AMI, auto-scal­ing set­tings, deploy­ment pipeline, etc.) is impor­tant, as is automa­tion, because they give you known states in your sys­tem that allows you to make asser­tions.

It’s also impor­tant to have sys­tems think­ing, try to come up with feed­back loops that dri­ve peo­ple and machines to do the right thing.

Adri­an then ref­er­enced Simon Wardley’s post on ecosys­tems in which he talked about the ILC mod­el, or, a cycle of Inno­vate, Lever­age, and Com­modi­tize.

He touched on Server­less tech­nolo­gies such as AWS Lamb­da (which we’re using heav­i­ly at Yubl). At the moment it’s at the Inno­vate stage where it’s still a poor­ly defined con­cept and even those involved are still work­ing out how best to utilise it.

If AWS Lamb­da func­tions are your nano-ser­vices, then on the oth­er end of the scale both AWS and Azure are going to release VMs with ter­abytes of mem­o­ry to the gen­er­al pub­lic soon — which will have a mas­sive impact on sys­tems such as in-mem­o­ry graph data­bas­es (eg. Neo4j).

When we move to the Lever­age stage, the con­cepts have been clear­ly defined and ter­mi­nolo­gies are wide­ly under­stood. How­ev­er, the imple­men­ta­tions are not yet stan­dard­ised, and the chal­lenge at this stage is that you can end up with too many choic­es as new ven­dors and solu­tions com­pete for mar­ket share as main­stream adop­tion gath­ers pace.

This is where we’re at with con­tain­er sched­ulers — Dock­er Swarm, Kuber­netes, Nomad, Mesos, Cloud­Foundry and what­ev­er else pops up tomor­row.

As the tech­nol­o­gy matures and peo­ple work out the core set of fea­tures that mat­ter to them, it’ll start to become a Com­mod­i­ty — this is where we’re at with run­ning con­tain­ers — where there are mul­ti­ple com­pat­i­ble imple­men­ta­tions that are offered as ser­vices.

This new form of com­mod­i­ty then becomes the base for the next wave of inno­va­tions by pro­vid­ing a plat­form that you can build on top of.

Simon Ward­ley also talked about this as the cycle of War, Won­der and Peace.