QCon London 2015–Takeaways from “The Bad Idea Terminator”

It’s very uncharacteristic of me, but I went to a session on the product management track at QCon London Melissa Perris’ “The Bad Idea Terminator”. Having gone in the room with the expectation of coming out not much wiser, I was pleasantly surprised to find myself in one of the best talks at the conference.


Melissa used Firestone and FourSquare as example of the “building trap” whereby ailing companies try to counteract their decline by adding more features without really changing they do things.

We often start off doing things right – we test and iterate on our ideas before we hit the market, and then we end up with something that people want to use. But then we just keep on building without going back to finding those innovative ideas that people love.


The build trap stops us building things that people love because we lose touch with our customers. We stop testing ideas with the market, and confine ourselves in our own bubble and just put our heads down and keep on building.


We can fall into the build trap in a number of ways, including:

  • pressure from stakeholders to always release new features (Peter Higgs made similar criticisms about modern academia where researchers are pressured to keep publishing papers rather than focusing on finding the next big idea)
  • arbitrary deadlines and failure to respond to change – setting deadlines that are too far out and not being flexible enough to adapt to change
  • “building is working” mentality – which doesn’t allow time for us to step back and think if we’re building the right things

Building is the easy part.

Figuring out what to build is hard.


– Melissa Perri


Why don’t we take the time to think before we go and build something? Well, the endowment effect might has something to do with it – as you invest more and more into an idea and it starts to become part of your identity and it becomes hard for you to let go.

In behavioral economics, the endowment effect (also known as divestiture aversion) is the hypothesis that people ascribe more value to things merely because they own them. This is illustrated by the observation that people will tend to pay more to retain something they own than to obtain something owned by someone else—even when there is no cause for attachment, or even if the item was only obtained minutes ago.


One of the most important responsibility of a product manager is to say NO to ideas, until we’re able to back it up with tests that prove an idea can work, and I think the same goes to developers.

So how do you become the Bad Idea Terminator, i.e. the person that goes and destroys all the bad ideas so we can focus on the good ones? We can start by identifying some common mistakes we make.


Mistake 1 : don’t recognize bias

Product ideas suffer from several types of ideas:

  • Causality – we attribute meaning and why things happen to the wrong cause

For example,

We built a mobile app before and it was successful, let’s do another mobile app.

Everyone has a mobile app, so we need one too.

we need to recognize the differences between customers and businesses, what worked under one set of circumstances is not guaranteed to work under another.

  • Curse of Knowledge – as experts we cannot put ourselves in the shoes of someone who doesn’t know as much

You should be doing user research and user testing, bring your ideas to the customers and see if that’s what they really want.

  • Anchoring – we focus on insignificant data because it’s the first data we see

Whenever someone says something like

All my customers are asking for this!

you should always ask for data, and make people prove what they’re saying is accurate.


Mistake 2 : solutions with no problems

When people suggest new ideas, most of the time they come to the table with solutions. Instead, we need to start with the WHY, and focus on the problem that we’re trying to solve.

On the topic of starting with the why, I also find Simon Sinek’s TED talk inspirational, and he also has a book on the same topic.


There are always multiple ways to solve the same problem, and only by focusing on the problem would we be able to decide which solution is the best. Unfortunately, your idea is not always the best idea, and we should be conscious of the Not Invented Here syndrome and our propensity to fall under its influence (even if only at a subconscious level).

After we figure out the problem we still need to align it with our business goals, and decide if it’s a problem we can solve and want to solve.


Mistake 3 : building without testing

When we get stuck in the build trap we don’t tend to test our assumption, as we tend to commit to one solution too early. Instead, we should solicit many solutions at first, and get people off the fixation on the one idea.

We also tend to invest too much into the one idea and then have trouble letting go (endowment effect again).

Instead, we should pick out a few ideas that are the most viable and then test them to find the ideas that:

  • have the most positive customer feedback, and
  • require the smallest investment

using small, data-driven experiments. Techniques such as A/B testing falls right into this (but remember though, A/B testing doesn’t tell the whole story, you probably also want A/A testing to act as blind test group). It could also be as simple as talking to a few customers to get their feedbacks.

There are 3 key experiments you should run:

  1. do the customers have this problem?
  2. are they interested in solution ideas?
  3. are they interested in our solution?


Mistake 4 : no success metrics

Another common mistake is to not set success metrics when we go and do experiments, and we also don’t set success metrics when building new features.

Instead, we should set goals early. Doing so early is important because if we set up goals in hindsight then we’ll just change the goals to make the feature look good…

We should be asking questions such as

How much value do we need to capture to make this feature worth building?

We also need to accept that leaning on its own is also a form of success.


The risk of continuing with a bad idea is really great, so the earlier we can kill of these bad ideas the lower our risk will be.

And the fast you kill the bad ideas, the more time you will have to devote to the good ones.

Fail fast, so you can succeed faster.


– Melissa Perri

and finally,an obligatory picture of terminator of course!



I really enjoyed Melissa’s talk, and although I’m a developer, I believe everyone inside an organization has the responsibility to ask questions and help push the organization towards building better products that actually match its customer’s needs.


Having read a couple of Dan Ariely’s books in the past year, I find they provide a very insightful backdrop on many of the human irrationalities that underlies/causes us to make the common mistakes that Melissa has identified in her talk.

image   image



Slides for the talk

Simon Sinek – Start with Why TED talk

Start with Why : how great leaders inspire everyone to take action

Predictably Irrational : The Hidden Forces That Shape Our Decisions

The Upside of Irrationality

QCon London 2015–Takeaways from “Scaling Uber’s realtime market platform”

On day three of QCon London, we were treated to some really insightful stories from the likes of Google, Atlas and Spotify. And for the first time in a while Uber is talking publically about what they’ve been up to.


The challenge for Uber’s platform is that both supply (Uber drivers) and demand (riders) are dynamic and matching them efficiently in real-time is not easy.

Uber’s services are written in a mixture of Node.js, Python, Java and Go, whilst a whole mix of databases are used – PostgreSQL, Redis, MySQL and Riak.


From a high level, they have a number of backend components:


They recently rewrote the Dispatch system despite Joel Spolsky advising against complete rewrites. To that, Matt Ranney said there are a number of built-in assumptions in the current dispatch system that is so deep-rooted that a revolutionary step is more efficient and productive:

  • assumes 1 rider per vehicle, hard to support vehicle pooling
  • the idea of moving people is baked into domain and code, making it hard to move into new markets (Matt didn’t elaborate, but I assume transportation of goods might be one such market)
  • sharding by city, which is not a sustainable approach as Uber moves into more and more cities
  • multiple points of failure that can bring everything down

The dispatch system was hard to fix incrementally, and since everything runs as a service, it was feasible to replace the existing system outright.


The new dispatch system looks like this:


where DISCO stands for DISpatCh Optimization service.


For geo-indexing, the dispatch service needs to know not only the physical distance between supply and demand, but also ETA based on historical travel data. The ETA calculation also needs to handle a number of special cases, including airports, where demands need to be queued (i.e. first come first served) to provide a fair service to everyone waiting at an airport.

The old system can only track available supplies (i.e. cars with no riders), which means there are missed optimization opportunities such as the following:


where the demand (D1) can be met by an in-flight supply (S2) earlier than an available supply (S1).

DISCO is able to consider supplies that are currently in-flight and project their route into the future and take that into the matching process, and supports vehicle pooling (if both D1 and D2 agrees to share a vehicle):



Uber breaks up the earth into tiny cells (like in Google Maps) and each is given a unique ID. Using the Google S2 library, you can identify cells that will completely cover a shape you’ve supplied:


Uber uses these cell IDs as sharding key to update supply, and when DISCO needs to match supply to demand, you can use that information to find supplies that are in the matching cells.

A limitation with this approach is that the cells have fixed size, so one would imagine the update activities are not well spread out through the key space. It’s natural for supply and demand to be concentrated around  city centres where the night life is – central London being a prime example.

Nonetheless, the goal of the routing is to:

  • reduce wait time for riders
  • reduce extra driving for drivers
  • lower overall ETAs


In order to scale their services, Uber went with an approach of building stateful services using Node.js. In addition, they also introduced a custom RPC protocol called ringpop, which is based on the SWIM paper. Ringpop also runs on its own TChannel multiplexing and framing protocol.

The goal of these projects is to provide:

  • performance across different languages
  • high performance request forwarding
  • proper pipelining
  • support for checksums and tracing
  • encapsulation


On a high-level, nodes in a cluster is able to handle any request, and if the data is not available on the node then the request is forwarded to the correct node.


This essentially deals with the need for managing consistent hashing on the client.


For Uber, availability is of paramount importance, as the cost of switching to competitor is low. So they decided to:

  • make everything retryable, which means making every operation idempotent (something which I suspect can be challenging in practice)
  • make everything killable (chaos monkey style), ringpop detects failed nodes and remove them from the cluster
  • crash only, no complicated graceful shutdowns
  • break things up into small pieces

which in turn required some cultural changes:

  • no pairs (I think he was talking about read-replica setups where there’s a potentially complicated fallover process)
  • kill everything, even databases


Since service talk to each other via load balancers, so you will need to be able to kill load balancers too, so instead load balancer logic is put in the service client (similar to Netflix Ribbon from what I gathered). I didn’t buy Matt’s rationale here since it’s possible to make load balancers highly available too, but then he also mentions the ability to do smarter routing – choosing data centre with better latency in a globally deployed infrastructure for example – which makes more sense.


Matt then went on to talk about some of the challenges with large fanout services, and in particular, the challenge with getting predictable latency when a large number of services are involved.


He also referenced Google fellow Jeff Dean’s paper Achieving Rapid Response Times in Large Online Services which is a great read, slide 39-70 describes the approach Uber has adopted.


In the example above, the following happened:

  1. service A sends req 1 to service B (1), informing it that the request will also be sent to service B (2)
  2. 5ms later, service A indeed sends the same request to service B (2), which goes into its backlog, service B (2) also finds out that service B (1) also got the same request
  3. meanwhile, service B (1) starts to process the request, sends a signal to service B (2) to cancel req 1 from its backlog
  4. service B (1) completes the request and replies to service A

If service B (1) was under load and couldn’t process the request fast enough then service B (2) would have processed the request and replied to service A, unless of course service B (2) is also under load.

In case you’re worried about the extra requests that would need to be processed with this approach, Jeff Dean paper (above) has the following results to show:


A more naive approach would be to always send the request to both service B (1) and service B (2) and just ignore the slower response. Based on a previous talk I watch this is (at least was) what Netflix does.


Finally, Matt touched on how Uber deals with datacentre outages. Their approach is quite simple and effective:


In this example, when the mobile app sends a location update, the service will respond with an encrypted state digest. When datacentre 1 fails:

  1. app will send the location updates to datacentre 2 instead
  2. since datacentre 2 doesn’t have the user’s state, so it requests the last state digest the app has received
  3. the app then sends the encrypted state digest in datacentre 2
  4. datacentre 2 decrypts the digest and initialize the user state
  5. now the app can converse with data centre 2 normally



Slides for the talk

Ringpop project page

TChannel project page

SWIM : Scalable Weakly-consistent Infection-style process group Membership protocol

Jeff Dean – Achieving rapid response times in large online services

QCon London 2015–Takeaways from “Service Architectures at Scale, Lessons from Google and eBay”

Day three of QCon London was a treat, with full day tracks on architecture and microservices, it presented some nice challenges of what to see during the day.

My favourite talk of the day was Randy Shoup’s Service Architectures at Scale, Lessons from Google and eBay.


Randy kicked off the session by identifying a common trend in the architecture evolution at some of the biggest internet companies.


An ecosystem of microservices also differ from their monolithic counterparts in that they tend to organically form many layers of dependencies rather than fall into strict tiers in a hierarchy.


At Google, there has never been a top-down design approach to building systems, but rather an evolutionary process using natural selection – services survive  by justifying their existence through usage or they are deprecated. What appears to be a clean layering by design turned out to be an emergent property of this approach.


Services are built from bottom-up but you can still end up with clean, clear separation of concerns.


At Google, there are no “architect” roles, nor is there a central approval process for technology decisions. Most technology decisions are made within the team, so they’re empowered to make the decisions that are best for them and their service.

This is in direct contrast to how eBay operated early on, where there was an architecture review board which acted as a central approval body.


Even without the presence of a centralized control body, Google proved that it’s still possible to achieved standardization across the organization.

Within Google, communication methods (e.g.. network protocol, data format, structured way of expressing interface, etc.) as well as common infrastructure (source control, monitoring, alerting, etc.) are standardized by encouragement rather than enforcement.


By the sound of it, best practices and standardization are achieved through a consensus-based approach in teams and then spread out  throughout the organization through:

  • encapsulation in shared/reusable libraries;
  • support for these standards in underlying services;
  • code reviews (word of mouth);
  • and most importantly the ability to search all of Google’s code to find existing examples

One drawback with following existing examples is the possibility of random anchoring – someone at one point made a decision to do things one way and then that becomes the anchor for everyone else who finds that example thereafter.



Whilst the surface areas of services are standardized, the internals of the services are not, leaving developers to choose:

  • programming language (C++, Go, Python or Java)
  • frameworks
  • persistence mechanisms



Rather than deciding on the split of microservices up ahead, capabilities tend to be implemented in existing services first to solve specific problems.

If it prove to be successful then it’s extracted out and generalized as a service of its own with a new team formed around it. Many popular services today all started life this way – Gmail, App Engine and BigTable to name a few.


On the other hand, a failed service (e.g. Google Wave) will be deprecated but reusable technology would be repurposed and the people in the team would be redeployed to other teams.


This is a fairly self-explanatory slide and an apt description of what a microservice should look like.



As the owner of a service, your primary focus should be the needs of your clients, and to meet their needs at minimum cost and effort. This includes leveraging common tools, infrastructures and existing service as well as automating as much as possible.

The service owner should have end-to-end ownership, and the mantra should be “You build it, you run it”.

The teams should have autonomy to choose the right technology and be held responsible for the results of those choices.


Your service should have a bounded context, its primary focus should be on the client and services that depend on the service.

You should not have to worry about the complete ecosystem or the underlying infrastructure, and this reduced cognitive load also means the teams can be extremely small (usually 3-5 people) and nimble. Having a small team also bounds the amount of complexity that can be created (i.e. use Conway’s law to your advantage).


Treat service-service relationship as a vendor-client relationship with clear ownership and division of responsibility.

To give people the right incentives, you should charge for usage of the service, this way, it aligns economic incentives for both sides to optimize for efficiency.

With a vendor-client relationship (with SLAs and all) you’re incentivized to reduce the risk that comes with making changes, hence pushing you towards making small incremental changes and employing solid development practices (code reviews, automated test, etc.).


You should never break your clients’ code, hence it’s important to keep backward/forward compatibility of interfaces.

You should provide an explicit deprecation policy and give your clients strong incentives to move off old versions.


Services at scale are highly exposed to performance variability.


Tail latencies (e.g. 95%, 99% latency) are much more important than average latencies. It’s easier for your client to program to consistent performance.


Services at scale are also highly exposed to failures.

(disruptions are 10x more likely from human errors than software/hardware failures)

You should have resilience in depth with redundancy for hardware failures, and have capability for incremental deployments:

  • Canary releases
  • Staged rollouts
  • Rapid rollbacks

eBay also use ‘feature flags’ to decouple code deployment from feature deployment.

And of course, monitoring..



Finally, here are some anti-patterns to look out for:

Mega-Service – services that does too much, ala mini-monolith

Shared persistence – breaks encapsulation, and encourages ‘backdoor’ violation, can lead to hidden coupling of services (think integration via databases…)


Gamesys Social

As I sat through Randy’s session, I was surprised and proud to find that we have employed many similar practices in my team (backend team at Gamesys Social), a seal of approval if you like:

  • not having architect roles, instead using a consensus-based approach to make technology decisions
  • standardization via encouragement
  • allow you to experiment with approaches/tech and not penalizing you when things don’t pan out (the learning is also a valuable output from the experiment)
  • organic growth of microservices (proving them in existing services first before splitting out and generalize)
  • place high value on automation
  • autonomy to the team, and DevOps philosophy of “you build it, you run it”
  • deployment practices – canary release, staged rollouts, use of feature flags and our twist on the blue-green deployment


I’m currently looking for some functional programmers to join the team, so if this sounds like the sort of environment you would like to work in, then have a look at our job spec and apply!




Slides for the talk

We’re hiring a Functional Programmer!

QCon London 2015–Takeaways from “Small is Beautiful”

From the first day of QCon London, I really enjoyed Kevlin Henney’s Small is Beautiful talk. Titled after E.F. Schumacher’s book (below) of the same name, the title itself should give you a pretty good idea of what to expect from this talk.



I’m a big fan of Kevlin, and his “Seven Ineffective Coding Habits of Many Programmers” talk at BuildStuff was the inspiration behind my parody post for the F# advent calendar last year. And as ever, Kevlin has plenty of well-applied, memorable quotes, starting with this one:

Sustainable development is development that meets the needs of the present without compromising the ability of future generations to meet their own needs.

– the report of the Brundtland Commission

when applied to software development, my interpretation of it is : “don’t take on more technical debt than you can reasonably pay back in the future in favour of short-term gains”. Which is nicely backed up by this tweet:



On the other extreme of the spectrum, you have people who are so concerned about future needs they end up completely over-engineering their solution to cope with this uncertainty and end up with projects that are delayed or worse, never delivered.



You should think of software as products, not projects.

If software are projects then they should have well-defined end state, but most often, software do not have well-defined end state, but rather evolved continuously for as long as it remains desirable and purposeful.


Despite its naivety when it comes to measuring complexity, lines-of-code is still a good metric to consider as it fits nicely with our main constraint as developers – our working memory.


There’s a learnt complacency amongst many developers that has allowed them to rest on their laurels and accept large codebases as a reality that cannot be challenged.

Kevlin showed a perfect counter-example of a computer chess program that is less than 512-bytes! Even the Unix 6 OS was less than 10k lines of code long, which is a fraction of the size of many modern day software.


(from Alan Kay’s 2011 talk, Programming and Scaling)


Creativity needs a boundary. Without any boundaries, a painter might be lost if you just ask him to “draw a picture”, and would you create anything more than a “hello, world!” application if asked to just “write a program”?

As Dennis Ritchie and Ken Thompson pointed out, size constraints played a key role in encouraging their creativity and come up with designs that are both elegant and small:

There has always been fairly severe size constraints on the Unix operating system and its software. Given the partially antagonistic desires for reasonable efficiency and expressive power, the size constraint has encouraged not only economy but a certain elegance of design.

– Dennis Ritchie and Ken Thompson

“The UNIX Time-Sharing System”, CACM

One reason for systems to become large is because we expect them to be that way without challenging that belief, putting boundaries pushes us to challenge those complacencies.

Kevlin also pointed out another good point – the more time you spend working on a project, the more the endowment effect kicks in and we become less inclined to change.


Conway’s law

organizations which design systems… are constrained to produce designs which are copies of the communication structures of these organizations.

– M. Conway

tells us that, another reason for systems to become large is because of the way they are staffed. Ironically, staffing decisions are usually taken at the point of your greatest ignorance – at the start of a project.

In other words, if you hire 100 people to work on a system, then you put in motion the events that will result in a communication structure consisting of 100 people, and the size of the system you get at the end becomes a self-fulfilling prophecy.


Software development does not have economies of scale.

Development has diseconomies of scale.

– Allan Kelly



You should be aware of code that serves no purpose.

Cargo cult programming is a style of computer programming characterized by the ritual inclusion of code or program structures that serve no real purpose.

Cargo cult programming can also refer to the results of applying a design pattern or coding style blindly without understanding the reasons behind that design principle.

– Wikipedia

What followed was a hilarious example of “enterprise” code at its finest – FizzBuzzEnterpriseEdition, seriously, check it out, I guarantee it’ll be the best thing you’ll see today!


Kevlin then retold the well-known tale of how Knight Capital Group lost $460 million in 45 minutes from a different perspective to highlight the potential danger of leaving old code that no longer serve any purpose around.

A classic case of “what’s the worst that could happen?”


And if you need any more convincing about removing redundant/unused code.

Our task is not to find the maximum amount of content in a work of art.

Our task is to cut back content so that we can see the thing at all.

– Susan Sontag

A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away.

– Antoine De Saint-Exupery



And finally, Kevlin’s offered one last bit of insight – system failures don’t usually happen because of one thing, they usually occur as the result of multiple failures.

This is, interestingly, echoed by none other than Man Utd’s legendary manager Sir Alex Ferguson who once said that goals are usually conceded as the result of multiple failures throughout the build-up play even if it ends with an individual’s error.


Slides for Small is Beautiful

QCon London 2015–Takeaways from “Code as Crime Scene”

The first day of this year’s QCon London is over, and it’s been a thoroughly enjoyable day of talks. Most of the  talks are on softer, more philosophical topics, which is a nice change of pace from last week’s LambdaDays.

One of my favourite talks from today was Adam Tornhill’s Code as Crime Scene and here are my key takeaways.


Many studies have showed that we spend most of our time making changes or fixing bugs, which always start with understanding what the code does. We should therefore we optimize for that.

A common problem we face in today’s world is that software is produced by many developers across many teams, and no one has a holistic view of how the whole looks.


When it comes to measuring complexity, both lines-of-code and  cyclomatic complexity are useful metrics to consider even though neither provide a full picture of what we’re up against. They are useful because they fit nicely with our main constraint as developers – our working memory.

Since we don’t have a metric that can provide a complete and accurate view on complexity, some has advocated for the use of good old human intuitions to measure complexity instead. However, intuitions are prone to social and cognitive bias, and doesn’t scale well because of the same cognitive constraints that necessitate the measuring of complexity in the first place.

Instead, Adam shows us how techniques from forensic psychology can be applied in software, specifically the practice of geographical offender profiling.


Most offenders behave like us most of the time, and that’s where they spot opportunities for crime. Hence there’s an overlap between the offender’s area of activity and the locations of his/her crimes.

Whilst this technique does not necessarily give you exact locations of suspect, it does help narrow down the area of search. Using tools such as CodeCity you can lay down the geography for your code which reflex their complexity.


But complexity alone is not the problem, it only becomes a problem when we have to deal with it.

If you overlay this geography with developer activities (i.e. commit history) and you will be able to identify hotspots – complex code that we need to work with often.



Defects tend to cluster, and if you overlay area of code where defects occur with hotspots then you’re likely to find a high correlation between hotspots and defects.

Adam also showed how you can track complexity of hot spots over time and use them to project into the future with Complexity Trend analysis.



Temporal Coupling – by analysing your commit history, you can find source files that are changed together in commits to identify dependencies (physical coupling), as well as ‘copy-and-paste’ code (logical coupling).


And remember, if you have dependency between software components developed by different people, then you essentially have dependency on people.


When Conway’s law

organizations which design systems… are constrained to produce designs which are copies of the communication structures of these organizations

– M. Conway

is applied in reverse, it becomes a useful organization tool, i.e. organize your communication structure to fit the software you want to build. It’s worth mentioning that this mirrors the shift in organizational structure that is happening in the DevOps movement.


If you connect people who commit to the same code by building links between them, then you build up a graph that tells you how the members of your team interact with each other through the parts of your codebase that they need to work on.

You can then compare that with your real organizational structure to see how well it supports the way you actually work. In the example below, you can see that members of the 4 teams are highly connected to everyone else, so it’s an indication that the team-level grouping does not reflect areas of responsibility as everyone’s responsibilities are overlapped.



The number of programmers behind a piece of code is the most effective indicator of the number of defects in that code – more programmers = more defects. You should pay attention to code that are changed by a lot of developers, it might be an indication that it has too many responsibilities and therefore reasons for different developers to change it.


By showing the number of commits each developer makes on a source file you can identify the knowledge owners of that part of your codebase.


You can then build up a knowledge map of your organization and even group the knowledge owners into their respective team structure to identify relation between code changes to teams.


In the perfect world, all knowledge owners for a component (source files for one project, for instance) would be concentrated within a team, which shows that the responsibility of that component is well defined and aligns with the organizational structure.

However, when you find components whose knowledge owners are scatter across your organization, then it might be an indication that:

  • maybe you’re missing a team to take ownership of that component, or
  • that component has too many responsibilities and in need of refactoring


Using the knowledge map, you can also identify key players in your organization – people who are knowledge owners in many areas of your codebase. This can help you identify risks of knowledge loss should they ever leave so you can mitigate these risks via planned knowledge share with other members of the team.

As it often happens, when key players leave, they also leave behind dead spots in your codebase – entire components which are abandoned because the people who understands them are no longer around. I have personally witnessed this happening multiple times and it’s often the reason why projects are “reinvented”.



Adam’s talk was awesome, and his book will be released at the end of the month on Amazon, you can also get the beta version eBook from The Pragmatic Bookshelf too.



Adam Tornhill’s article on Code as Crime Scene

Your Code as a Crime Scene on Amazon

YouTube – Code as Crime Scene at TEDx

Code Maat – command line tool to mine and analyse data from version control systems