Check out my new course Learn you some Lambda best practice for great good! and learn the best practices for performance, cost, security, resilience, observability and scalability.
Unlike most conferences, the OSCON keynotes are really short (average 10-15 mins each) and having watched all the published keynote sessions here are my top picks.
Simon Wardley – Situation Normal, Everything Must Change
Simon started by pointing out the lack of situational awareness on the part of enterprise IT. Enterprise IT lives in a low situational awareness environment that relies on backward causality and verbal reasoning (or story telling), and has no position or movement.
Whereas high-level situational awareness environments (e.g. military combat) are context specific, you have positions and movements and usually employ some form of visual reasoning.
Military actions are driven by your situational awareness of the where and why, but in Business we have a tyranny of actions.
and this is where Simon’s value chain mapping comes in. With maps, you can add positions to your components based on the values they provide, as well as movement as they evolve from the unchartered world (chaotic, uncertain, unpredictable, etc.) to become industrialized.
In terms of methodology, there’s no one size that fits all.
Agil, XP and Scrum are very good on the left side (the unchartered world), particularly when you want to reduce the cost of change.
On the right side, as things become industrialized, you want to reduce the cost of deviation and Six Sigma is good.
In the middle where you want to build a product, lean is particularly strong.
If you take a large scale project, rather than having a one-size-fits-all methodology, you can employ different methodologies based on where that component is in its evolution. For developers, this is no different to the arguments for polyglot programming, or polyglot persistence, because no single language or database is good for all the problems we have to solve.
Why should the way we work be any different?
By overlapping the value chain maps for different areas of the business you can start to identify overlaps within the organization, and a snippet from his most recent post shed some horrifying light on the amount of duplication that exists:
…To date, the worst example I know of duplication is one large global company that has 380 customised versions of the same ERP system doing exactly the same process…
The US air force discovered that, as people came up with new ideas they tend to add features to that idea and made it better (and more complex); they then added even more features and made the idea so complex it’s completely useless to anyone, and that’s approximately where they shipped it. (for regular readers of this blog, you would probably have read about this obsession of features many times before)
So Lt. Col. Dan Ward came up with FIST (Fast, Inexpensive, Simple and Tiny), which in his own words:
…FIST stands for Fast, Inexpensive, Simple and Tiny. It’s a term I use to describe a particular approach to acquisitions and system development. As you might guess, it involves using a small team of talented people, a tight budget, a short schedule and adhering to a particular set of principles and practices…
in other words, small is beautiful, and it’s a theme that we have seen repeatedly – be it microservices, or Amazon’s two-pizza teams, etc.
And as you impose constraints on the teams – tight budget, short schedule – you encourage creativity and innovation from the teams (something that Kevlin Henney also talked about at length during his Joy of Coding closing keynote).
However, even with small teams, you still have this problem that things need to evolve. Fortunately we have a solution to that too, in what is known as the three party system where you have:
- pioneers – who are good at exploring the unchartered world
- settlers – who are good at taking half-baked ideas and make useful products for others
- town planners – who are good at taking a product and industrialising it into commodity and utility
Once you have a map, you can also start to play games and anticipate change. Or better yet, you can manipulate the map.
You can accelerate the pace of evolution by using open practices – open source, open API, etc. Or you can slow the process down by using patents, or FUD.
The key thing is that, once you have a map, you can see where things are moving and visually reason about why you should attack one component over another. And that’s how you can turn business into situational awareness first, and actions after.
As things move from product to commodity, they enable a new generation of services to spawn up (wonder), but they also cause death to organizations stuck behind the inertia barrier (death).
Simon finished this brilliant session with three lessons:
- if you’re a start up, have no fear for large corporates because they suck at situational awareness;
- the future is awesome, and pioneers have already moved into the space of open hardware and open biology;
- open source itself is changing, we have new people coming in as new settlers
I hope you enjoyed Simon’s talk, his blog has much more information and goes into each of these topics in a greater deal of detail. If you follow him on Twitter (@swardley) he also post regular titbits of wisdom, which I have started to collect.
James Pearce – How Facebook Open Sources at Scale
…We use in production what we open source, and we open source only what we use in production…
– James Pearce
Martin Fowler – Making Architecture Matter
I don’t like the term “software architecture” as it summons up these images of some senior person in an organization who’s setting rules and standards on how software should be written but having actually written any software for maybe 10 or 20 years. These architects, Joel Spolsky use the term “architecture astronauts”, often cause a lot of problems in software projects. So the whole term “architect” and “architecture” has a kinda nasty taste to it.
– Martin Fowler
For me, the key points from this talk are:
- the notion that architects shouldn’t code is wrong (or, don’t be an ivory tower architect!)
- architecture is really the shared understanding of the system’s design amongst its expert developers
- architecture diagrams are just (often imperfect) representations of this shared understanding
- as software projects grow, what matters the most is for you to ensure a good shared understanding between people leading the project
- architecture is also the decisions that you wish you could get right early
- your concern is therefore the decisions that are hard to change, e.g. the programming language
- combining the two definitions above, and you can think of architecture as the “important things that I need to always keep in my head whilst I’m working on the system”
- when confronted with requests for more features over quality, don’t make the moral argument of craftsmanship
- when it comes to a battle between craftsmanship and economics, economics always wins
- a common fallacy is to think that software quality is something that can be traded off for cost (like you do with cars or cellphones)
- software has both external (visible to users) as well as internal (good modularity, etc.) and architecture is about internal quality
- what matters with internal quality is the long term picture
- well maintained code base gives you a platform to build upon and can make future development easier and faster
- poorly maintained code base makes it harder and harder for you to make changes to
- this is why architecture matters!
Raffi Krikorian – Hacking Conway’s Law
Conway’s law has been a trendy topic at conferences this past 12 months, and everyone is basically singing the same tune – apply Conway’s law in reverse and organize your communication structure to fit the software you want to build.
OSCON is coming to Europe!
At long last, we’ll see a version of OSCON in Europe this year, on 26th-28th October in Amsterdam. Some pretty cool tech companies will be represented there – GitHub, DataStax, Google, ThoughtWorks, PayPal, Heroku and Spotify to name a few, and of course, our very own Gamesys
I will giving a talk on the work I did with Neo4j a while back, which you can read all about in this post.
p.s. Rachel Reese (of Jet) is coming over from the US and talking about building reactive service with F#!
I specialise in rapidly transitioning teams to serverless and building production-ready services on AWS.
Are you struggling with serverless or need guidance on best practices? Do you want someone to review your architecture and help you avoid costly mistakes down the line? Whatever the case, I’m here to help.
Check out my new podcast Real-World Serverless where I talk with engineers who are building amazing things with serverless technologies and discuss the real-world use cases and challenges they face. If you’re interested in what people are actually doing with serverless and what it’s really like to be working with serverless day-to-day, then this is the podcast for you.
Check out my new course, Learn you some Lambda best practice for great good! In this course, you will learn best practices for working with AWS Lambda in terms of performance, cost, security, scalability, resilience and observability. We will also cover latest features from re:Invent 2019 such as Provisioned Concurrency and Lambda Destinations. Enrol now and start learning!
Check out my video course, Complete Guide to AWS Step Functions. In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. There is something for everyone from beginners to more advanced users looking for design patterns and best practices. Enrol now and start learning!
Are you working with Serverless and looking for expert training to level-up your skills? Or are you looking for a solid foundation to start from? Look no further, register for my Production-Ready Serverless workshop to learn how to build production-grade Serverless applications!
Here is a complete list of all my posts on serverless and AWS Lambda. In the meantime, here are a few of my most popular blog posts.
- Lambda optimization tip – enable HTTP keep-alive
- You are wrong about serverless and vendor lock-in
- You are thinking about serverless costs all wrong
- Just how expensive is the full AWS SDK?
- Many faced threats to Serverless security
- We can do better than percentile latencies
- Yubl’s road to Serverless
- AWS Lambda – should you have few monolithic functions or many single-purposed functions?
- AWS Lambda – compare coldstart time with different languages, memory and code sizes
- Guys, we’re doing pagination wrong
- Top 10 Serverless framework best practices