CraftConf 15 experience report

2015-04-24 13.19.45

Phew, what a week, finally back in the UK after a good few days in Budapest for Craft­Conf. What a beau­ti­ful city and what a great conference.

On a per­sonal level it’s been a good trip, caught up with some old friends, and met some new ones. Big thanks to Adam Granicz and the lovely folks of Intel­li­Fac­tory for show­ing us a good time and the Budapest night life, it was a blast!

I also caught up with Mar­tin Klepp­mann whom I met at dev/winter/2015 and found out about the book he’s been work­ing on — Design­ing Data-Intensive Appli­ca­tions — which gives you an overview of just about every­thing you need to be aware of in that space.

The Venue & Food

As a whole, Budapest is a beau­ti­ful city with plenty to see, and the views at night is breath­tak­ing. The con­fer­ence is hosted at one of the prime loca­tions in Budapest — the Várk­ert Bazar. It was recently ren­o­vated (at a great cost as locals kept telling us!) and only open to pub­lic since last Sep­tem­ber.

The food at the con­fer­ence was one of the best I’ve had at any con­fer­ence. They even served seafood at lunch, seafood!

The Talks

There were some really good talks at the con­fer­ence, I had to make some tough choices. For­tu­nately all the talks were recorded and also streamed live on the con­fer­ence web­site thanks to the uStream guys. Based on what I’ve been told, the record­ings should be avail­able as early as this weekend.

I took plenty of notes dur­ing the ses­sions I attended (see below) and would write up my key take­aways from each over the next few weeks, so watch this space!

  • Scal­ing micro-services at Gilt (Adrian Trenaman)
  • Jepsen IV: Hope Springs Eter­nal (Kyle Kingsbury)
  • Archi­tec­ture with­out an end state (Michael Nygard)
  • Con­cur­rency: It’s Harder (and Eas­ier) Than You Think (Paul Butcher)
  • Beyond Fea­tures: Rethink­ing Agile Plan­ning and Track­ing (Dan North)
  • The Hid­den Dimen­sion of Refac­tor­ing (Michael Feathers)
  • Why is an API like a puppy? (Ade­wale Oshineye)
  • Microser­vices AntiPat­terns (Tam­mer Saleh)

My per­sonal favourite talk was Kyle Kingsbury’s Jepsen IV: Hope Springs Eter­nal. Not only did Kyle pit var­i­ous NoSQL prod­ucts against their out­landish claims and put them to the knife, he also pro­vided a nice frame­work for you to thank and rea­son about the dif­fer­ent prop­er­ties a con­sis­tency model can give you. If you enjoyed Kyle’s post Call me maybe : Elas­tic­search then you’ll love this talk.

The not-so-great

My over­all expe­ri­ence of Craft­Conf was great, but there are two things which I hope they’ll improve on in the future:

  • lack of func­tional pro­gram­ming talks, I saw a total of two talks that are FP-related
  • almost all the talks are very high-level and few showed any code

Whilst I appre­ci­ate that the con­fer­ence focuses on crafts­man­ship, so it’s nat­ural for the talks to stay high-level. In my opin­ion, expos­ing atten­dees to non-mainstream par­a­digms (FP, logic pro­gram­ming, AOP, stack-oriented) would also serve to improve their craft. After all, what bet­ter way to expand your hori­zon than to learn your­self a new par­a­digm?

Craft­Conf vs CodeMesh

In terms of con­tent, Craft­Conf is a great con­fer­ence to go and get inspired. But, if you’re look­ing for a con­fer­ence where you can find out about emerg­ing tech­nolo­gies and lan­guages (and have your brain hurt after two days) then you should come to Lon­don in Novem­ber for CodeMesh.

If Craft­Conf is a con­fer­ence for soft­ware engi­neer­ing and crafts­man­ship, then CodeMesh is a con­fer­ence for com­puter sci­ence.

Slides for my Craft-Conf meetup talk on DSLs and F#

Binary and Json benchmarks updated

DISCLAIMER : as always, you should bench­mark against your pay­load and use case, the bench­mark num­bers I have pro­duced here is unlikely to be rep­re­sen­ta­tive of your use cases and nei­ther is any­body else’s bench­mark numbers.

You can use the sim­ple test har­ness I cre­ated and see these exam­ple code to bench­mark against your par­tic­u­lar payload.


Only FsPick­ler and Json.Net was updated for this bench­mark and there are no sig­nif­i­cant changes in per­for­mance in either update.





Quite a few of the JSON seri­al­iz­ers had been updated since the last update :

  • fastJ­SON
  • FsPick­ler
  • Jil
  • Mon­goDB Driver
  • NetJ­SON
  • Json.Net
  • ServiceStack.Text
  • System.Text.Json

Of the updates, System.Text.Json has seen the biggest change in per­for­mance, nearly halved its dese­ri­al­iza­tion time.



*protobuf-net is in this list purely as a bench­mark to show how the tested JSON seri­al­iz­ers com­pare to one of the fastest binary seri­al­izer in .Net.

Design for Latency issues

The most com­mon issue I have encoun­tered in pro­duc­tion are latency/performance related. They can be symp­toms of a whole host of under­ly­ing causes rang­ing from AWS net­work issues (which can also man­i­fest itself in latency/error-rate spikes in any of the AWS ser­vices), over-loaded servers to sim­ple GC pauses.

Latency issues are inevitable – as much as you can improve the per­for­mance of your appli­ca­tion, things will go wrong, even­tu­ally, and often they’re out of your control.

So you must design for them, and degrade the qual­ity of your appli­ca­tion grace­fully to min­i­mize the impact on your users’ experiences.

As back­end devel­op­ers, one of the fal­lac­ies that we often fall into is to allow our dev envi­ron­ments to be too lenient. Servers and data­bases are never under load dur­ing devel­op­ment, so we lure client devel­op­ers into a false sense of com­fort and set them up to fail when the appli­ca­tion runs into a slow-responding server in pro­duc­tion for the first time.


Latency Injec­tion

To pro­gram my fel­low client devel­op­ers to always be mind­ful of latency spikes, we decided to inject ran­dom latency delays on every request:

  1. check if we should inject a ran­dom delay;
  2. if yes, then work out how much latency to inject and sleep the thread;
  3. and finally invoke the orig­i­nal method to process the request

This is an imple­men­ta­tion pat­tern that can be auto­mated. I wrote a sim­ple Post­Sharp attribute to do this, whilst pig­gy­back­ing exist­ing con­fig­u­ra­tion mech­a­nisms to con­trol its behav­iour at runtime.

Then I mul­ti­cast the attribute to all our ser­vice end­points and my work was done!

We run latency injec­tion in our dev envi­ron­ment and it helped iden­tify numer­ous bugs in the client appli­ca­tion as a result and proved to be a worth­while exercise.

But we didn’t stop there.


Error Injec­tion

We throt­tle user requests to all of our ser­vices to stop mis­chie­vous play­ers from spam­ming our servers using proxy tools such as Charles and Fid­dler, or hand­crafted bots.

But, occa­sion­ally, legit­i­mate requests can also be throt­tled as result of client bugs, over-zealous retry strat­egy or incor­rectly con­fig­ured throt­tling threshold.

Once again, we decided to make these errors much more vis­i­ble in the dev envi­ron­ment so that client devel­op­ers expect them and han­dle them gracefully.

To do that, we:

  1. set the thresh­old very low in dev
  2. used a Post­Sharp attribute to ran­domly inject throt­tling error on oper­a­tions where it makes sense

The attribute that injects throt­tling error is very sim­ple, and looks some­thing along the lines of:

The same approach can be taken to include any ser­vice spe­cific errors that the client should be able to grace­fully recover from – ses­sion expi­ra­tion, state out-of-sync, etc.


Design for Failure

Sim­u­lat­ing latency issues and other errors fall under the prac­tice of Design for Fail­ure, which Simon Ward­ley iden­ti­fies as one of the char­ac­ter­is­tics of a next-generation tech company.


p.s. you should check out Simon’s work on value chain map­ping if you haven’t already, they’re inspiring.


Chaos Engines

Netflix’s use of Chaos Mon­key and Chaos Gorilla is a shin­ing exam­ple of Design for Fail­ure at scale.

Chaos Mon­key ran­domly ter­mi­nates instances to sim­u­late hard­ware fail­ures and test your system’s abil­ity to with­stand such failures.

Chaos Gorilla takes this exer­cise to the next level and sim­u­late out­ages to entire Ama­zon avail­abil­ity zones to test their system’s abil­ity to auto­mat­i­cally re-balance to other avail­abil­ity zones with­out user-visible impact or man­ual intervention.

Net­flix has taken a lot of inspi­ra­tion from Release It! by Michael Nygard and Drift into Fail­ure by Sid­ney Dekker. Both books are awe­some and I highly rec­om­mend them.

image image


Global redun­dancy, or not

Based on reac­tions to AWS out­ages on social media, it’s clear to see that many (our­selves included) do not take full advan­tage of the cloud for global redundancy.

You might scoff at that but for many the deci­sion to not have a glob­ally redun­dant infra­struc­ture is a con­scious one because the cost of such redun­dancy is not always justifiable.

It’s pos­si­ble to raise your single-point-of-failure (SPOF) from indi­vid­ual resources/instances, to AZs, to regions, all the way to cloud providers.

But you’re incur­ring addi­tional costs at each turn:

  • your infra­struc­ture becomes more com­plex and dif­fi­cult to rea­son with;
  • you might need more engi­neers to man­age that complexity;
  • you will need to invest in bet­ter tool­ing for mon­i­tor­ing and automation;
  • you might need more engi­neers to build those tools;
  • you incur more wastage in CPU/memory/bandwidth/etc. (it is called redun­dancy for a reason);
  • you have higher net­work latency for cross-AZ/region communications;


Global redun­dancy at Uber

On the other hand, for many orga­ni­za­tions the cost of down­time out­weighs the cost of global redundancy.

For instance, for Uber’s cus­tomers the cost of switch­ing to a com­peti­tor is low, which means avail­abil­ity is of para­mount impor­tance for Uber.

Uber devised a rather sim­ple, ele­gant mech­a­nism for their client appli­ca­tions to failover seam­lessly in the event of a dat­a­cen­tre out­age. See this post for more details.


Latency Prop­a­ga­tion

Finally, as more and more com­pa­nies adopt a microser­vices approach a whole host of chal­lenges will become evi­dent (many of which have been dis­cussed in Michael Nygard’s Release it!).

One of these chal­lenges is the prop­a­ga­tion of latency through inter-service communications.

If each of your ser­vices have a 99 per­centile latency of 1s then only 1% of calls will take longer than 1s when you depend on only 1 ser­vice. But if you depend on 100 ser­vices then 63% of calls will take more than 1s!

In this regard, Google fel­low Jeff Dean’s paper Achiev­ing Rapid Response Times in Large Online Ser­vices presents an ele­gant solu­tion to this problem.


I haven’t put this into prac­tice myself, but I imag­ine this can be eas­ily imple­mented using Rx’s amb operator.



Don’t learn a syntax, learn to change the way you think

A lan­guage that doesn’t affect the way you think about pro­gram­ming, is not worth knowing.”

- Alan Perlis

In my last post, I out­lined the tech­niques to learn­ing that I picked up from Josh Kaufman’s TEDx talk. I briefly men­tioned that one should learn a new par­a­digm rather than a lan­guage with a slightly dif­fer­ent syn­tax (e.g. C# <=> Java, Action­Script <=> Haxe) but didn’t stress nearly enough the impor­tance of doing so.

As the math­e­mati­cian Richard Ham­ming once said, that it’s pos­si­ble for there to be thoughts that we can­not have:

Just as there are odours that dogs can smell and we can­not, as well as sounds that dogs can hear and we can­not, so too there are wave­lengths of light we can­not see and flavours we can­not taste. Why then, given our brains wired the way they are, does the remark “Per­haps there are thoughts we can­not think,” sur­prise you? Evo­lu­tion, so far, may pos­si­bly have blocked us from being able to think in some direc­tions; there could be unthink­able thoughts.”

- Richard Hamming

but what does this have to do with pro­gram­ming lan­guages, or any lan­guages for that matter?

What we can­not express, we can­not think

BBC went to a remote, uncon­nected tribe called the Himba, whose colour vocab­u­lary is very dif­fer­ent from us in that they only have 5 words for colour.


What they found was intriguing.

The Himba is able to eas­ily iden­tify the square with a dif­fer­ent shade of green in this picture:


because they have dif­fer­ent words for the two shades of green. For us, this feat is dif­fi­cult because to us, they’re all green.

On the other hand, because we have two words for blue and green, it is easy for us to spot the blue square from the rest:


For the Himba, who has only one word that describes both blue and green, they strug­gle to iden­tify the square with a dif­fer­ent colour.

The impli­ca­tion is pro­found, that not only do lan­guages shape our think­ing, they might pro­hibit it.

The lim­its of my lan­guage mean the lim­its of my world.”

- Lud­wig Wittgenstein

What if the same is true for pro­gram­ming languages?

Pro­gram­ming lan­guages have a devi­ous influ­ence: they shape our think­ing habits.”

- Dijk­stra

What if our cre­ative pow­ers are lim­ited by the ideas that can be expressed by the pro­gram­ming lan­guage we use?

What if some solu­tions are out of our reach because they can­not be clearly expressed in those lan­guages? How would we ever dis­cover these unknown unknowns?

With­out being aware of the out­side pos­si­bil­i­ties, would we ever become aware of our uncon­scious incom­pe­tence?

I don’t know of any research into the impact pro­gram­ming par­a­digms have on cre­ativ­ity. From per­sonal expe­ri­ence, my most cre­ative solu­tions have come from my ven­tures into non-mainstream pro­gram­ming and data­base par­a­digms. These ven­tures have opened doors in my mind and allowed me to see solu­tions I couldn’t before.

By con­trast, my other learn­ing efforts such as Dart and Haxe has taught me a slightly dif­fer­ent syn­tax for express­ing the same ideas that I’m already famil­iar with. Whilst they can prove to be use­ful tools for spe­cific tasks, they failed to have the same mind-expanding effect that is far more valu­able to my long term career.

On Diver­sity

We can diver­sify our ways of think­ing and approaches to prob­lem solv­ing by learn­ing new pro­gram­ming par­a­digms, but that’s not the only way. In fact, some of the most notable peo­ple in the field of com­puter sci­ence has come from other disciplines.

For exam­ple, Alan Kay was a biol­o­gist, and it was the study of biol­ogy that inspired him to coin the term objects.

I thought of objects being like bio­log­i­cal cells and/or indi­vid­ual com­put­ers on a net­work, only able to com­mu­ni­cate with messages.”

- Alan Kay

(the fact that we have come to teach and think about OOP in a very dif­fer­ent sense to what Alan Kay envi­sioned is another matter..)

Sim­i­larly, Joe Arm­strong started out as a physicist.

And Adam Torn­hill, a speaker whose work I thor­oughly enjoy, has learnt much about pro­gram­ming from his stud­ies in psy­chol­ogy.


We also need diver­sity in our communities.

There’s a famous Chi­nese proverb that goes like this,

three hum­ble shoe­mak­ers brain­storm­ing make a great statesman.”

How­ever, this only holds true if each of the shoe­mak­ers is able to offer a dif­fer­ent per­spec­tive and ideas.


In real­ity, given that the shoe­mak­ers have the same pro­fes­sion and gen­der they prob­a­bly have a much greater over­lap in mindscape.


The result? The shoe­mak­ers’ numer­i­cal advan­tage amounts to lit­tle when it comes to find­ing cre­ative solu­tions. Which is why, in a cre­ative indus­try like ours,  we need to guard against mono­cul­ture.


To sum­marise, learn­ing a new lan­guage within the same par­a­digm gives you imple­men­ta­tion options and trade-offs for the set of approaches that is best afforded by the paradigm.

Learn­ing a new par­a­digm gives you entire new approaches that you didn’t have access to before, and has a far longer-lasting ben­e­fit. Which is why, given the choice, you should learn a new par­a­digm rather than a new language.