NDC Oslo 15 – Takeaways from “Lean and Functional Programming”

Bryan Hunter has been respon­si­ble for organ­is­ing the FP track at NDC con­fer­ences as well as a few oth­ers and the qual­i­ty of the tracks have been con­sis­tent­ly good.

The FP track for this year’s NDC Oslo was excep­tion­al and every talk has had a full house. And this year Bryan actu­al­ly kicked off the con­fer­ence with a talk on how func­tion­al pro­gram­ming can help you be more ‘lean’.


History of Lean

Bryan start­ed by talk­ing about the his­to­ry of lean, and whilst most peo­ple (myself includ­ed) thought the lean prin­ci­ples were cre­at­ed at Toy­ota, turns out it actu­al­ly orig­i­nat­ed from the US.

At a time when the US work­force was dimin­ished because all the men were sent to war, a group called the TWI (Train­ing With­in Indus­try) was formed to find a way to bring women into the work­force for the first time, and train them.

The process of con­tin­u­ous improve­ment the TWI cre­at­ed was a stun­ning suc­cess, before the war the US was pro­duc­ing around 3,000 planes per year and by the end of the war the US was pro­duc­ing 300,000 planes a year!

Unfor­tu­nate­ly, this his­to­ry of lean was most­ly lost when the men came back from war and the fac­to­ries went back to how they worked before, and jobs were pri­or­i­tized over effi­cien­cy.

Whilst the knowl­edge from this amaz­ing peri­od of learn­ing was lost in the US, remains of the TWI was sent to Ger­many and Japan to help them rebuild and this was how the basic foun­da­tions of the lean prin­ci­ples were passed onto Toy­ota.

side­bar: one thing I find inter­est­ing is that, the process of con­tin­u­ous improve­ment that lean intro­duces:

Lean Principles

is actu­al­ly very sim­i­lar to how deep learn­ing algo­rithms work, or as glimpsed from this tweet by Eveli­na, how our brain works.


Bryan then out­lined the 4 key points of lean:

Long term phi­los­o­phy : you need to have a sense of pur­pose that super­sedes short-term goals and eco­nom­ic con­di­tions. This is the foun­da­tion of lean, and with­out it you’re nev­er sta­ble.

The right process will pro­duce the right results : if you aren’t get­ting the right results then you’ve got the wrong process and you need to con­tin­u­ous­ly improve that process.

Respect, chal­lenge, and devel­op your peo­ple : and they will become a force mul­ti­pli­er.

Con­tin­u­ous­ly solv­ing root prob­lems dri­ves orga­ni­za­tion­al learn­ing.


Com­pa­nies adopt lean because it’s a proven path to improv­ing deliv­ery times, reduc­ing costs and improv­ing qual­i­ty.

The way these improve­ments hap­pen is by elim­i­nat­ing waste first, then elim­i­nat­ing over-bur­den and incon­sis­ten­cy.


Lean Thinking

The so-called lean house is built on top of the sta­bil­i­ty of hav­ing a long-term phi­los­o­phy and the process of con­tin­u­ous improve­ment (or Kaizen, which trans­lates to changes for the bet­ter in Chi­nese and Kan­ji).

Then through the two pil­lars of Just-In-Time and Act on Abnor­mal­i­ty we arrive at our goal of improved Deliv­ery Times, Qual­i­ty and reduced Costs.



A pow­er­ful tool to help you improve is “go and see”, where you go and sit with your users and see them use your sys­tem. Not only do you see how they are actu­al­ly using your sys­tem, but you also get to know them as humans and devel­op mutu­al empa­thy and respect which leads to bet­ter com­mu­ni­ca­tion.


Anoth­er thing to keep in mind is that you can’t adopt lean by just adopt­ing a new tech­ni­cal solu­tion. Often when you adopt a new tech­ni­cal solu­tion you just change things with­out improv­ing them, and you end up with con­tin­u­ous change instead of con­tin­u­ous improve­ment.

Func­tion­al pro­gram­ming fits nice­ly here because it holds up under­neath the scruti­ny of Plan-Do-Check-Act (PDCA). Instead of hav­ing a series of changes you real­ly have to build on the idea of stan­dard work.

With­out the stan­dard you end up with a shot­gun map where things change, and improve­ments might come about (if you change enough times then it’s bound to hap­pen some time, right?) but not as the result of a for­malised process and are there­fore unpre­dictable.


Seven Wastes

Then there are the Sev­en Wastes.


Over­pro­duc­tion is the most impor­tant waste as it encom­pass­es all the wastes under­neath it. In soft­ware, if you are build­ing fea­tures that aren’t used then you have over­pro­duced.

side­bar : for reg­u­lar read­ers of this blog, you might remem­ber Melis­sa Per­ris call­ing this ten­den­cy of build­ing fea­tures with­out ver­i­fy­ing the need for them first as The Build Trap in her talk at QCon Lon­don. Dan North also talked about the issue of over­pro­duc­tion through a fix­a­tion on build­ing fea­tures in his talk at Craft­Conf – Beyond Fea­tures.


By doing things that aren’t deliv­er­ing val­ue you cause oth­er waste to occur.


Trans­porta­tion, in soft­ware terms, can be thought of as the cost to go from require­ment, through to deploy­ment and into pro­duc­tion. DevOps, Con­tin­u­ous Inte­gra­tion and Con­tin­u­ous Deploy­ment are very much in line with the spir­it of lean here as they all aim to reduce the waste (both cog­ni­tive and time) asso­ci­at­ed with trans­porta­tion.

Inven­to­ry can be thought of as all the things that are in progress and haven’t been deployed. In the water­fall mod­el where projects go on for months with­out any­thing ever being deployed, then all of it will become inven­to­r­i­al waste if the project is killed. The same can hap­pen with scrums where you build up inven­to­ry dur­ing the two-week sprint ‚which as Dan North men­tioned in his Beyond Fea­tures talk, is just long enough for you to expe­ri­ence all the pain points of water­fall.

You expe­ri­ence Unnec­es­sary Motion when­ev­er you are fire fight­ing or doing things that should be auto­mat­ed. This waste equates to the wear-and-tear on your peo­ple, and can cause your peo­ple to burn out.

Wait­ing is self-explana­to­ry, and can often result from defi­cien­cies in your organization’s com­mu­ni­ca­tion and work sched­ul­ing (require­ments tak­ing too long to arrive whilst assigned work­ers are wait­ing on their hands).

Over pro­cess­ing is equiv­a­lent to the idea of gold plat­ing in soft­ware. Although as Jett Atwood point­ed out in his post here, refac­tor­ing can be thought of as gold plat­ing in the purest sense but it’s also impor­tant in pro­duc­ing sane, main­tain­able code.


Last­ly, we have Defects. It’s expo­nen­tial­ly cheap­er to catch bugs at com­pile time than it is in pro­duc­tion. This is why the notion of Type Dri­ven Devel­op­ment is so impor­tant, and the goal is to make invalid state unrep­re­sentable (as Scott Wlaschin likes to say!).

But you can still end up with defects relat­ed to per­for­mance, which is why I was real­ly excit­ed to hear about the work Greg Young has done with Pri­va­te­Eye (which he pub­li­cal­ly announced at NDC Oslo).



Bryan used the work­ing of a (some­what dys­func­tion­al) ware­house to illus­trate the prob­lem of a large code­base where you have lots of unnec­es­sary code you have to go through to get to the prob­lems you’re try­ing to solve.image

side­bar : whilst Bryan didn’t call it out by name, this is an exam­ple of cross-cut­ting con­cerns, which Aspect-Ori­ent­ed Pro­gram­ming aims to address and some­thing that I have writ­ten about reg­u­lar­ly.


One way to visu­al­ize this prob­lem is through Val­ue Stream Map­ping.

In this dia­gram the  batch process­es are where val­ue is cre­at­ed, but we can see that there’s a large amount of lead time (14 days) com­pared to the amount of pro­cess­ing time (585 sec­onds) so there are lots of waste here.

As one process fin­ish­es its work the mate­r­i­al is pushed aside until the next process is ready, which is where you incur lead time between process­es.

This is the push mod­el in mass man­u­fac­tur­ing.

In soft­ware, you can relate this to how the water­fall mod­el works. All the require­ments are gath­ered at once (a batch process); then imple­men­ta­tions are done (anoth­er batch); then test­ing, and so on.

In between each batch process, you’re cre­at­ing waste.


The solu­tion to this is the flow mod­el, or one-piece flow. It focus­es on com­plet­ing the pro­duc­tion of one piece from start to fin­ish with as lit­tle work in process inven­to­ry between oper­a­tions as pos­si­ble.

In this mod­el you also have the oppor­tu­ni­ty of catch­ing defects ear­ly. By exer­cis­ing the process from start to end ear­ly you can iden­ti­fy prob­lems in the process ear­ly, and also use the expe­ri­ence to refine and improve the process as you go.

You also deliv­er val­ue (a com­plet­ed item) for down­stream as ear­ly as pos­si­ble. For exam­ple, if your users are impact­ed by a bug, rather than have them wait for the fix in the next release cycle along with oth­er things (the batch mod­el) you can deliv­er just the fix right away.

Again, this reminds me of some­thing that Dan North said in his Beyond Fea­tures talk:

“Lead time to some­one say­ing thank you is the only rep­u­ta­tion met­ric that mat­ters.”

- Dan North

And final­ly, you can eas­i­ly par­al­lelise this one-piece flow by hav­ing mul­ti­ple peo­ple work on dif­fer­ent things at the same time.


Bryan then talked about the idea of Sin­gle-minute Exchange of Die (SMED) – which is to say that we need an effi­cient way to con­vert a man­u­fac­tur­ing process from run­ning the cur­rent prod­uct to run­ning the new prod­uct.

This is also relat­able to soft­ware, where we need to stay away from the batch mod­el (where we have too much lead time between val­ues being deliv­ered), and do as much as nec­es­sary for the down­stream and then switch to some­thing else.

side­bar : I feel this is also relat­ed to what Greg Young talked about in his The Art of Destroy­ing Soft­ware talk where he pushed for writ­ing soft­ware com­po­nents that can be entire­ly rewrit­ten (i.e. exchanged) in less than a week. It is also the best answer to “how big should my microser­vice be?” that I have heard.


You should flow when you can, and pull when you must, and the idea of pull is:

“pro­duce what you need, only as much as you need, when you need”

- Tai­ichi Ohno

when you do this you’re forced to see the under­ly­ing prob­lems.


With a 2 week sprint, there is enough buffer there to mask any under­ly­ing issues you have in your process. For instance, even if you have to spend 20 mins to fight with some Team­C­i­ty con­fig­u­ra­tion issue then that inef­fi­cien­cy is masked by you work­ing just a bit hard­er. How­ev­er, that prob­lem is not fixed for any­one else and is a recur­ring cost that your orga­ni­za­tion has to pay.

In the pull mod­el where you have down­stream wait­ing on you then you’re forced to see these under­ly­ing prob­lems and solve them.


There’s a wide­spread mis­con­cep­tion that kan­ban equals lean, but kan­ban is just a tool to get there and there are oth­er tools avail­able. An inter­est­ing­ly, Tai­ichi Ohno actu­al­ly got the idea for kan­ban from pig­gly wig­gly, based on the way they stock sodas and soup.

Anoth­er tool that is sim­i­lar to kan­ban is 5S.



Act on Abnormality

A key com­po­nent here is to decou­ple humans from machines – that you shouldn’t require humans to watch the machines do their job and babysit them.

Anoth­er tool here is mis­take-proof­ing, or poka yoke. If there’s a mis­take that can hap­pen we try to min­i­mize the impact of those mis­takes.

A good exam­ple is the design of the man­hole cov­er:


which is round so that it can’t fall down the hole no mat­ter which way you turn it.

Anoth­er exam­ple is to have visu­al con­trols that are vary obvi­ous and obnox­ious so that you don’t let prob­lems hide.


I’m real­ly glad to hear Bryan say:

“you shouldn’t require con­stant dili­gence, if you require con­stant dili­gence you’re set­ting every­one up for fail­ure and hurt.”

- Bryan Hunter

which is some­thing that I have been preach­ing to oth­ers in my com­pa­ny for some time.

An exam­ple of this is in the man­age­ment of art assets in our MMORPG Here Be Mon­sters. There were a set of rules (nam­ing con­ven­tions, fold­er struc­tures, etc.) the artists have to fol­low for things to work, and when they make a mis­take then we get sub­tle asset-relat­ed bugs such as:

  • invis­i­ble char­ac­ters and you can only see his/her shad­ow
  • a char­ac­ter appear­ing with­out shirt/pants
  • trees mys­te­ri­ous­ly dis­ap­pear­ing when tran­si­tion into a state that’s miss­ing an asset

So we cre­at­ed a poka yoke for this in the form of some tools to decou­ple the artists from these arbi­trary rules. To give you a flavour of what the tool did here’s some screen­shots from an inter­nal pre­sen­ta­tion we did on this:



When you do have a prob­lem, there’s a process called the 5 Whys which helps you iden­ti­fy the root cause.



Functional Programming

The rest of the talk is on how func­tion­al pro­gram­ming helps you mis­take-proof your code in ways that you can’t in imper­a­tive pro­gram­ming.



A key dif­fer­ence between FP and imper­a­tive pro­gram­ming is that imper­a­tive pro­gram­ming relies on mutat­ing state, where­as FP is all about trans­for­ma­tions.

This quote from Joe Arm­strong (one of the cre­ators of Erlang) sums up the prob­lem of muta­tion very clear­ly:

“the prob­lem with OO is that you ask  OO for a banana, and instead you get a Goril­la hold­ing the banana and the whole jun­gle.”

- Joe Arm­strong

With muta­tions, you always have to be dili­gent to not cause some adverse effect that impacts oth­er parts of the code when you call a method. And remem­ber, when you require con­stant dili­gence you’re set­ting every­one up for fail­ure and hurt.

side­bar : Venkat Sub­ra­ma­ni­am gave two pret­ty good talks at NDC Oslo on the pow­er and prac­ti­cal­i­ty immutabil­i­ty and things we can learn from Haskell, both are rel­e­vant here and you might be inter­est­ed in check­ing out if you’re still not sold on FP yet!


In C#, mak­ing a class immutable requires dili­gence because you’re going up against the defaults in the lan­guage.


side­bar : as Scott Wlaschin has dis­cussed at length in this post, no mat­ter how many FP fea­tures C# gets there will always be an unbridge­able gap because of the behav­iour the lan­guage encour­ages with its default set­tings.

This is not a crit­i­cism of C# or imper­a­tive pro­gram­ming.

Pro­gram­ming is an incred­i­bly wide spec­trum and imper­a­tive pro­gram­ming has its place, espe­cial­ly in per­for­mance crit­i­cal set­tings.

What we should do how­ev­er, is to shy away from using C# for every­thing. The same is true for any oth­er lan­guage – stop look­ing for the one lan­guage that rules them all.

Explore, learn, unlearn, there are lots of inter­est­ing lan­guages and ideas wait­ing for you to dis­cov­er!


How vs What

Anoth­er impor­tant dif­fer­ence between func­tion­al and imper­a­tive pro­gram­ming is that FP is focused on what you want to do where­as imper­a­tive forces you to think about how you want to do it.

Imper­a­tive pro­gram­ming forces you to under­stand how the machine works, which brings us back to the human-machine lock-in we talked about ear­li­er. Remem­ber, you want to decou­ple humans from machines, and allow humans to focus on under­stand­ing and solv­ing the prob­lem domain (which they’re good at) and leave the machine to work out how to exe­cute their solu­tion.


Some his­tor­i­cal con­text is use­ful here, in that back in the days when com­put­ers were slow and bug­gy, the hard­ware was the bot­tle­neck. So it pays to have the devel­op­er tell the com­put­er how to do our com­pu­ta­tions for us, because we knew bet­ter.

Nowa­days, soft­ware is the thing that is slow and bug­gy, both com­pil­ers and CPUs are capa­ble of doing much more opti­miza­tion than most of us can ever dream of.

Whilst there are still legit­i­mate cas­es for choos­ing imper­a­tive pro­gram­ming in per­for­mance crit­i­cal sce­nar­ios, it doesn’t nec­es­sar­i­ly mean that devel­op­ers need to write imper­a­tive code them­selves. Libraries such as Streams is a good exam­ple of how you can allow devel­op­ers to write func­tion­al style code which is trans­lat­ed to imper­a­tive code under the hood. In act, the F# com­pil­er does this in many places – e.g. com­pil­ing your func­tion­al code into imper­a­tive while/for loops.


Anoth­er way to look at this debate is that, by mak­ing your devel­op­ers deal with the HOW as well as the WHAT (just because you’re think­ing about the how doesn’t mean that you don’t have to think about the what) you have increased the com­plex­i­ty of the task they have to per­form.

Giv­en the amount of cog­ni­tive resources they have to per­form the task is con­stant, then you have effec­tive­ly reduced the like­li­hood your devel­op­ers would deliv­er a piece work­ing soft­ware that does the right thing. And a piece of soft­ware that does the wrong faster is prob­a­bly not what you are after…


p.s. don’t take this dia­gram lit­er­al­ly, it’s mere­ly intend­ed to illus­trate the rela­tion­ship between the three things here — cog­ni­tive resources, com­plex­i­ty of the prob­lem and the chance of deliv­er­ing a cor­rect solu­tion. And before you argue that you can solve this prob­lem by just adding more peo­ple (i.e. cog­ni­tive resources) into the equa­tion, remem­ber that the effect of adding a new mem­ber into the group is not lin­ear, and is espe­cial­ly true when the devel­op­ers in ques­tion do not offer suf­fi­cient­ly dif­fer­ent per­spec­tive or view point.


null references

And any con­ver­sa­tion about imper­a­tive pro­gram­ming is about com­plete with­out talk­ing about null ref­er­ences, the inven­tion that Sir Tony Hoare con­sid­ers as his bil­lion dol­lar mis­take.

side­bar : BTW, Sir Tony Hoare is speak­ing at the CodeMesh con­fer­ence in Lon­don in Novem­ber, along with oth­er indus­try greats such as John Hugh­es, Robert Vird­ing, Joe Arm­strong and Don Syme.

In func­tion­al lan­guages such as F#, there are no nulls.

The absence of a val­ue is explic­it­ly rep­re­sent­ed by the Option type which elim­i­nates the no. 1 invalid state that you have to deal with in your appli­ca­tion. It also allows the type sys­tem to inform you exact­ly where val­ues might be absent and there­fore require spe­cial han­dling.


And to wrap things up…