CraftConf 15–Takeaways from “Architecture Without an End State”

In enter­pris­es we tend to sell archi­tec­tur­al projects by show­ing you the cur­rent messy state and pitch you an ide­al­ized end state. But the prob­lem is that, the ide­al end state we’re pitch­ing exists only as an idea, with­out hav­ing gone through all the com­pro­mis­es and organ­ic growth that made the cur­rent state messy.

Once we have gone through the process of imple­ment­ing and grow­ing the sys­tem over the course of its life­time, how do we know we won’t end up in the same messy sit­u­a­tion again?

bsg-happened-before

The 3-year Plans

Worse still, these pitch­es are always fol­lowed by a 3 year plan where you see no return on your invest­ment for 2 whole years!

3-year-plan

And since 3 years is a long time, many things can hap­pen along the way — merg­ers, acqui­si­tions, new tech­nolo­gies, new CTO/CIO, etc. — and often we don’t ever real­ize the orig­i­nal 3 year plan before these forces of change takes us to a new direc­tion.

And the vicious cycle con­tin­ues.…
3-year-plans

 

With­in a large orga­ni­za­tion there are always mul­ti­ple forces of change hap­pen­ing at the same time, and your per­spec­tive changes depend­ing on your posi­tion with­in the orga­ni­za­tion.

In a com­plex sociotech­ni­cal sys­tem like this, there’s no priv­i­leged van­tage point. One per­son can­not under­stand all ram­i­fi­ca­tions of their deci­sions, not even the CEO of a com­pa­ny.

The best you can do, is local­ly con­tex­tu­al actions.

 

8 Rules for Survival

1. Embrace plurality

Avoid ten­den­cy for the “one”, e.g.

  • sin­gle source of truth;
  • sin­gle sys­tem of record;
  • one sys­tem to rule them all;
  • etc.

The prob­lem with hav­ing a sin­gle sys­tem of records (SSOR) is that it’s ter­ri­bly frag­ile. Many real-world events (e.g. M&A, divesti­tures, enter­ing new mar­ket) can eas­i­ly vio­late your pur­suit for the “one” sys­tem.

In epis­te­mol­o­gy (i.e. the study of “how we know what we know”), there’s the idea that we can only know what we are able to record in a sys­tem. If your SSOR is bad­ly shaped for the infor­ma­tion that exist, you might be unaware of the rest of the instances of infor­ma­tion you’re look­ing for.

 

When mod­el­ling a domain, some­times it’s even tough to find “one” def­i­n­i­tion for some­thing that every­one can agree on.

For exam­ple, a “cus­tomer” can mean dif­fer­ent things to dif­fer­ent depart­ments:

  • for cus­tomer ser­vice, a cus­tomer is some­one who is pay­ing you mon­ey
  • for sales, some­one is a cus­tomer as soon as you start talk­ing to them

 

Final­ly, there’s this philo­soph­i­cal idea that, the mod­el you use to view the world shapes the thoughts you are able to think.

Your par­a­digm defines the ques­tions you can ask.

ludwig-wittgenstein

I think this also applies to pro­gram­ming lan­guages and par­a­digms, with the same detri­men­tal effects in lim­it­ing our poten­tial as cre­ative prob­lem solvers. And it’s a good rea­son why you should learn at least a few pro­gram­ming par­a­digms in order to expand your mind.


 

When deal­ing with mul­ti­ple sys­tems of records, we change our empha­sis and instead focus on:

  • who’s the author­i­ty for what span?
  • define a sys­tem to find the author­i­ty for a span giv­en some unique iden­ti­fi­er, e.g. URIs, URNs (AWS’s ARN sys­tem springs to mind)
  • agree­ing on a rep­re­sen­ta­tion so we can inter­change them across dif­fer­ent sys­tems
  • enable copies and have a mech­a­nism for syn­chro­niza­tion

Some­thing inter­est­ing hap­pens when you do these — your infor­ma­tion con­sumers no longer assume where the infor­ma­tion comes from.

If you pro­vide full URLs with your prod­uct details, then the con­sumers can go to any of the URLs for your prod­uct infor­ma­tion. They must become more flex­i­ble and assume there is an open world of infor­ma­tion out there.

Along with the infor­ma­tion you get, you also need a mech­a­nism for deter­min­ing what actions you can per­form with them. These actions can be encod­ed as URLs and may point to dif­fer­ent ser­vices.


As I lis­tened, two words sprang to mind:

microser­vices - break­ing up a big, com­plex sys­tem into small­er parts each with author­i­ty over its own span; infor­ma­tion is accessed through a sys­tem of iden­ti­fiers and an agreed-upon for­mat for inter­change (rather than shared data­base, etc.)

HATEOAS — or REST APIs in the way Roy Field­ing orig­i­nal­ly described; actions per­mit­ted on infor­ma­tion is pass back along­side the infor­ma­tion itself and encod­ed as URLs

 

2. Contextualize Downstream

We have a ten­den­cy to put too much in the data we send down­stream, which can often end up being a very large set.

But our busi­ness rules are usu­al­ly con­tex­tu­al:

  • not all infor­ma­tion about an enti­ty is required in every case
  • oper­a­tions are not avail­able across entire class­es of enti­ties

Again, HATEOAS strikes me as a par­tic­u­lar­ly good fit here for con­tex­tu­al­iz­ing the data we send to down­stream sys­tems.


We build sys­tems as a net­work of data-flows, but we also need to look at the ecosys­tem of sys­tem of sys­tems and be aware of the down­stream impacts of our changes.

Some­thing like the fol­low­ing is obvi­ous­ly not desir­able!

downstream-impact

So the rule is:

  • Aug­ment the data upstream, add fields but not rela­tion­ships if you can avoid it.
  • Con­tex­tu­al­ize down­stream, apply your busi­ness rules as close to the user as pos­si­ble. Because as it turns out, the rules that are clos­est to the users change the most often.
  • Min­i­mize the enti­ties that all sys­tems need to know about.

 

3. Beware Grandiosity

Fight the temp­ta­tion to build the mod­el to end all mod­els (again, embrace plu­ral­i­ty), aka Enter­prise Mod­el­ling Dic­tio­nary.

The prob­lem with an Enter­prise Mod­el­ling project is that, it requires:

  • a glob­al per­spec­tive, which we already said is not avail­able in a com­plex sys­tem;
  • agree­ment across all busi­ness units, which is extra­or­di­nar­i­ly hard to achieve;
  • tal­ent for abstrac­tion; but also
  • con­crete expe­ri­ence in all con­texts;
  • a small enough team to make deci­sions.

Such teams only exist in Hol­ly­wood movies..

TheATeam-77904-4

Instead, look to how stan­dard com­mit­tees work:

  • they com­pro­mis­es a lot;
  • they don’t seek agree­ments across the board, and instead agree on small sub­sets and leave room for ven­dor ven­tures;
  • they take a long time to con­verge
  • they delib­er­ate­ly lim­it their scope

All mod­els are wrong, but some are use­ful.”

- George Box

So the rule is:

  • don’t try to mod­el the whole world;
  • find com­pro­mis­es and sub­sets that every­one can agree on;
  • start small and incre­men­tal­ly expand;
  • assume you don’t know every­thing and leave room for exten­sion;
  • allow very lengthy com­ment peri­od (with plen­ty of email reminders since every­one leaves it till the last minute..)

 

4. Decentralize

Decen­tral­iza­tion allows more explo­ration of mar­ket space because it allows mul­ti­ple groups to work on dif­fer­ent things at once.

You can also explore the solu­tion space bet­ter because dif­fer­ent groups can try dif­fer­ent solu­tions for sim­i­lar prob­lems.

How­ev­er, you do need some pre­req­ui­sites for this to work suc­cess­ful­ly:

  • Trans­paren­cy — meth­ods, work and results must be vis­i­ble
  • Iso­la­tion — one group’s fail­ure can­not cause wide­spread dam­age
  • Eco­nom­ics — cre­ate a frame­work for every­one to make deci­sions that con­tribute towards the larg­er goal

The Boe­ing 777 project is an excel­lent exam­ple of dis­trib­uted eco­nom­ic deci­sion-mak­ing.

The no. 1 thing that deter­mines a plane’s life­time cost is the weight of the air­craft. So the weight of the air­craft is in direct trade-off with cost (e.g. lighter mate­ri­als = more expen­sive man­u­fac­tur­ing cost).

They worked out what the trade-off was and set out rules so that an engi­neer can buy a pound of weight for $300 — i.e. as an engi­neer, you can make a deci­sion that’ll shave a pound of weight off the air­craft at the expense of $300 of man­u­fac­tur­ing cost with­out ask­ing any­body.

Engi­neer­ing and pro­gramme man­agers and above have a big­ger man­u­fac­tur­ing cost budge per pound, and any­thing above these pre­de­fined lim­its requires you to go before the gov­er­nance board and jus­tice your deci­sions.

What they have done is set­ting the trade-off poli­cies cen­tral­ly and by defin­ing the bal­anc­ing forces glob­al­ly they have cre­at­ed a frame­work that allows deci­sions to be made local­ly with­out requir­ing reviews.

In IT, Michael has observed that cen­tral­iza­tion tends to lead to bud­get fights between depart­ments. Where­as decen­tral­iza­tion allows for sliv­ers of bud­gets work­ing towards a com­mon goal.

 

5. Isolate Failure Domains

If you reverse the direc­tion data flows through your sys­tems and you can find the direc­tion of depen­den­cies between them.

dir-dataflow-and-dependencies

And the thing about depen­den­cy is that some­times you can’t depend on them. So you def­i­nite­ly want to iso­late your­self from upstream depen­den­cies. In his book, Release It!, Michael out­lined a num­ber of use­ful pat­terns to help you do just that, e.g. cir­cuit break­er, bulk­heads, time­out,  fail fast. Many of these pat­terns have been incor­po­rat­ed in libraries such as Netflix’s Hys­trix.

 

Anoth­er kind of fail­ure is when you have missed a solu­tion space entire­ly. This is where mod­u­lar­i­ty can give you val­ue in that it gives you options that you might exer­cise in future.

Just as options in the finan­cial sense (a long call in this case), there’s a neg­a­tive pay­off at the start as you need to do some work to cre­ate the option. But if you choose to exer­cise the option in the future then there’s a long pos­i­tive pay­off.

value-of-option

Whilst Michael didn’t spell it out, I think the val­ue of mod­u­lar­i­ty in this case lies in the abil­i­ty to recov­er from hav­ing missed a solu­tion space the first time around. If the mod­u­lar­i­ty is there then at least you have to option to adopt a dif­fer­ent solu­tion at a lat­er point (i.e. exer­cis­ing the option).


Michael then gave an exam­ple of a trad­ing com­pa­ny who has build their sys­tem as many small appli­ca­tions inte­grat­ed by mes­sag­ing. The appli­ca­tions are small enough and fine grained enough that it’s cheap­er to rewrite than to main­tain, so they treat these appli­ca­tions as dis­pos­able com­po­nents.


I’d like to give a shout out to Greg Young’s recent talk, The Art of Destroy­ing Soft­ware, where he talked at length about build­ing sys­tems whilst opti­miz­ing for deletabil­i­ty. That is, you should build sys­tems so that they’re nev­er big­ger than what you can rea­son­ably rewrite in a week. And that is as good a guide­line on “how big should my microser­vice be?” as I have heard so far.

 

6 & 7. Data Outlives Applications; Applications Outlives Integrations

As things change, a typ­i­cal lay­ered archi­tec­ture breaks when there are pres­sure from either side.

layered-architecture-breaks

A Hexag­o­nal archi­tec­ture, or Ports and Adapters style archi­tec­ture on the oth­er hand, puts your domain in the cen­tre rather than the bot­tom of the stack. The domain is then con­nect­ed to the out­side world via adapters that bridges between the con­cepts of that adapter and the con­cepts of the domain.

hexagonal-architecture

Hexag­o­nal archi­tec­tures make tech­nol­o­gy tran­si­tion more of a bound­ary lay­er issue than a domain lay­er issue, and allows you to explic­it­ly rep­re­sent mul­ti­ple adapters from the same domain for dif­fer­ent pur­pos­es.

 

8. Increase Discoverability

In large orga­ni­za­tions you often find peo­ple work­ing on projects with sig­nif­i­cant amounts of over­lap due to lack of dis­cov­er­abil­i­ty about what oth­er peo­ple are work­ing on.

Just as one would search Github or Google to see if a sim­i­lar project exists already before embark­ing on his/her own open source alter­na­tive, we should be doing more to enable dis­cov­er­abil­i­ty with­in com­pa­nies.

Doing so allows us to move faster by build­ing on top of oth­ers work, but in order to do this we need to make our work vis­i­ble inter­nal­ly:

  • inter­nal blogs
  • open, eas­i­ly acces­si­ble code repos­i­to­ries
  • mod­ern search engine

Inter­nal teams should work like open source projects whilst retain­ing a prod­uct men­tal­i­ty.

One thing to watch out for is the bud­get cul­ture where:

  • con­tri­bu­tions to oth­er people’s projects are not wel­comed due to bud­get fights;
  • inquiries about each other’s work are alarm­ing and con­sid­ered pre­cur­sor to their attempt to suck your bud­get into theirs

Instead you should embrace an engi­neer­ing cul­ture which tend to be much more open. You could start to build up an engi­neer­ing cul­ture by:

  • hav­ing team blogs
  • allow users to report bugs
  • make CI servers pub­lic

 

 

Final­ly, stop chas­ing ide­al­ized end state, and embrace the wave of change.

 

I was at QCon Lon­don ear­li­er this year, and in Kevlin Henney’s Small is Beau­ti­ful talk he deliv­ered a sim­i­lar mes­sage. That we should think of soft­ware as prod­ucts, not projects.  If soft­ware are projects then they should have well-defined end state, but most often, soft­ware do not have well-defined end state, but rather evolved con­tin­u­ous­ly for as long as it remains desir­able and pur­pose­ful.

 

 

Links