Adapt - why success always begins with failure: Tim Harford

Why are you reading this book review?

Adapt came out in 2011 and it's a masterful summary of evolutionary development in economics, society, business and personal endeavour. It looks at the problems of finding solutions in complex spaces, safety measures in complex interconnected systems (like nuclear power stations, domino toppling, oil rigs and banking systems). Harford is erudite, amusing, insightful, anecdotal, informed and at times gripping. It's a great book.

Presumably you're reading this book review because you haven't read it.

WHAT?

You haven't read it?!

Stop reading this book review now! AND READ IT!

What's the quickest way to make a Gantt chart from an agile board?

Don't say it! I know. :-)

But someone actually asked this question this week, adding presumably to avoid embarrassment that the purpose was to "translate reality into management artifacts". He has a point of course. Familiar artifacts can allow people to see new things more clearly; we just also need to ensure that we've all understood what is really different in the new world.

Any way, I suggested looking at the Open Source tool xProcess, which uses its built-in forecasting engine to put start and end dates on a hierarchical collection of tasks or work items based on size estimates and team availability. I reckon if you have a table with task names you can generate such a chart in about 5 minutes.

Here are the instructions. Download xProcess from this site: http://sourceforge.net/projects/xprocess/

1. Create a project by hitting the "New" button like this:

You can choose various templates but just choose Simple Process / Simple Project. It's the.. well simplest.

2. Add the list of work items by selecting "New" again and "Task". (See next screenshot).

3. Hit "Next" and you can add all the names from the cards on your wall as a comma separated list (see screenshot on the right). Leave the "Size" as 2 - we'll come back to that.

4. Now when you go to the Task List tab (or hit  the "Tasks" button) you should see something like the screenshot below. All your tasks are listed all of size 2.0 and with start and end forecast dates of NEVER.

In other word the Scheduler is forecasting none of these task will finish. That's because we have no resources assigned to the project. The simplest quick fix for this is to add yourself to the project. So...

5. Hit the "Resources" button and then "Add/New Resource", select yourself (or add team members) and then hit "Next". This gets to the "Appointment to Project" dialog shown below. You can define availability on this screen or just hit "Finish".

We now have enough information to generate a Gantt chart, albeit with some fairly big assumptions such as a uniform size of tasks and 100% availability of the team we've assigned. Nevertheless we can hit the "Gantt Chart" button and see the result.

Here's my attempt.

There are lots of things you may wish to adjust at this point like the priority order of the tasks, the estimates for the items (either the default value or individual ones) and team availability. You can even add separate subtasks for different parts of the process into a task template and add team roles so the right people are assigned to the right types of subtask. There are lots of adjustments you can make depending how much longer than 5 minutes you want to spend on this. You can even enter (or generate) fixed "target" dates which compare the forecast dates with dates that may have been agreed with a customer ( or are simply the forecast from last month). Endless hours of fun are possible. :-)

However if you just want to give your manager a comforting diagram, the steps above could be 5 minutes well spent!

What is Kanban?

"What is Kanban?" you ask.

Well really that is three questions. Firstly what is a kanban?

1. A kanban is a visual signal.

It is a Japanese term meaning a card, brand or sign. In lean production systems it's used as the signal to an upstream part of the process that items are required by a downstream part of the process.

Which leads to the next question. What is a kanban system?

2. A kanban system is a system for managing work that uses (real or virtual) kanbans to control the flow.

Kanban systems were first used in Toyota for manufacturing but are now widely used in a wide variety of applications including health services and knowledge-based work such as software development. In principle a kanban system is a "pull-system" where work is triggered by demand from a downstream part of the process - ultimately by the demand of the consumer or commissioner of the product. The systems use kanban signals to prevent over or under production in various parts of the process. The kanbans may be "real", for example an empty hopper which is sent back on the production line to indicate that more parts of a given type are needed, or "virtual", such as the signals derived from a kanban board when a "story card" is moved out of a column. In both cases they indicate to the upstream process that another item should be produced, processed or selected.

"But hang on! I thought Kanban was a method."

You're right. That's the third question, the third part of what is Kanban. To be crystal clear, we should ask "What is the Kanban Method?"

3. The Kanban Method is an approach to defining and improving kanban systems.

The Kanban Method emerged from work being done by many practitioners in software management, agile and lean methods and new product development  during the first decade of this century. The author of the method was David Anderson, but many others contributed or were doing similar work which fed into the method, including Drago Dumitriu, Jim Benson (co-author of Personal Kanban), Don Reinertsen (author of Flow), Karl Scotland, Corey Ladas (author of Scrumban), Alan Shalloway, Arne Rooke, Mattias Skarin... there are many more than I know about. Kanban was first formulated as a method in a paper presented by Anderson at the Agile 2007 conference in Washington DC, and in 2010 Anderson published what is still the principal authoritative text on the method, "Kanban: Successful evolutionary change for your technology business".

As Alistair Cockburn has observed the word "method" may confuse here - he suggests "reflective improvement framework" is closer to the intention (if it weren't such a mouthful I might agree with him!). I would say Kanban is a method, but a method for defining and improving a process, not the process itself or even a process framework. As such people are often confused by what they think are properties of the Kanban Method, whereas in fact they are simply characteristics of a particular kanban system that they have observed. Try not to be confused or you may miss the crucial value that Kanban can bring to your organisation - continuous evolutionary improvement. A method such as Scrumban for example is an application of the Kanban method to a process that starts off as Scrum or Scrum-like. Scrumban is Kanban, but in a particular context!

Elsewhere in this blog I discussed my shortest possible guide to adopting Kanban, based on the underlying paradigm, the principles and the practices of the Kanban Method. Here it is again:

1. Change your viewpoint (lean flow paradigm):
   See work as flow
2. Change your mindset (foundational principles):
   Start from here and improve
3. Change your process continually (core practices):
   Make work and policies visible; make validated improvements

The mechanism of change in agile approaches

"Evolution" and "revolution" describe 2 mechanisms of change:

1. Revolution: sweep away and replace
2. Evolution: copy; differentiate; select; amplify; repeat

This defines the difference between evolution and revolution - not the size of the change, or even the size of the steps in that change. It's the mechanism of change that is significant, because evolution (surprisingly to most people) can sometimes produce large changes in short periods, while revolution sometimes involves quite small changes.

Perhaps a discussion for another time is how this relates to politics and management, and why it is that many politicians/managers (even conservative ones) favour revolutionary changes - that they can take credit for? - over evolutionary changes, which necessarily require competition between ideas and failure of quite good ones, so that better ideas are amplified.

The mechanism of evolutionary change can be seen again and again in agile methods. I''ve written elsewhere about its relevance to the evolution of processes in Kanban (see "Evolution and the culture of an adaptive organisation"). Here's another example from a test driven process: the "test-change-test-change-test" cycle in BDD and TDD-like approaches.

A Test-Driven cycle in agile development

The point about this process is that it must start and end at "survivable" points, where specs, tests and model (i.e. code) agree. There are many cases where tests or metrics fail the quality test (a "red-bar" or failed build results) and these cases must either be changed again or reverted. There are also many points where change can be introduced: the model, the requirements or the tests of fitness, but every change is subject to the selection criteria which eliminates the unfit and amplifies (promotes to "head" - where it is again copied and modified) the instance with improvements.

This is not like evolution. It is evolution.

Improving Waterfall Processes (a thought experiment)

Some recent discussions about whether Kanban is an agile method or not seems to me to miss the point somewhat. The Kanban method is about improving your process, so whether you end up with an agile process depends on two things:

1. whether your process was agile to start with
2. whether "more effective" maps to "more agile"

The first point started me thinking whether I could design a thought experiment by using Little's Law on that anathema of all non-agile processes - Waterfall! (Just say "no" I hear +Karl Scotland say. :-) More on that below, but let's deal with the second point first. 

Agility is the ability to change direction quickly in response to changing circumstances. A few businesses operate in very stable conditions and can optimise their processes to just these conditions. For them increasing throughput is usually the highest priority - agility may not be high on their priority list. However most of us working in knowledge-based disciplines find that the needs of our stakeholders change very rapidly. The ability to track this moving target and deliver new ideas faster is the key to being "more effective". For us agility does map to effectiveness.

So if we've decided being more agile is desirable in our context, how to go about it? Here are two possible approaches:

1. Look for a checklist of "agile practices" and adopt them (you may need a consultant who's cleverer than me to advise you on which ones must be adopted together and in which order!)
2. Look for a way to measure your "agility" then see if these measures improve as you make incremental changes to your current practices.

There is value in both approaches of course, since they are not mutually exclusive. However I know I am not alone in seeing failures in agile adoption initiatives where dependent practices are introduced in the wrong order (for example Scrum for team organisation, without automated build and test for regression testing). 

The second approach relies on having some measures of agility. Here are three from a recent David Anderson blog. They are cadences all measured in units of time:

1. how often can an organization interact with its customers to make selection and prioritization decisions about what to work on next? (queue replenishment period)
2. how quickly can the work be completed from making a commitment to do it? (lead time)
3. how often can new completed work be delivered and operational? (release period)

Now let's go back to whether your process was agile to start with. Let's take a fairly "waterfall" process, typical of many larger organisations, and let's see if focusing on just these metrics could provide significant improvements in agility. This thought experiment relies on Little's Law, which we can express as:

Lead Time =  WIP / Delivery Rate, where

Lead Time is the time a work item spends in the process (TIP or time in process is an alternative term)

Delivery Rate is the number of items delivered per unit of time (also called Throughput)

WIP is the number of work items are in the process at any point in time.

The initial waterfall(-ish) process

Our initial scenario is a product development company releasing a new version of a product roughly every 6 months, say 120 working days. It's not as extreme as some waterfall projects where no customer deliveries are made until years after the requirements are defined, but nor is it very agile. 

Let's say this project releases 24 new "Features" (independently releasable functionality) which are selected at the start of the analysis phase for each 6-monthly release. Each Feature can be further broken down to around  10 User Stories. (Or whatever the waterfall equivalent is! Development tasks say.) There are 3 stages in our (very simplified) process: Analysis, Development, and Release. Testing is included in the Development phase (something they must have learned from a passing agile team!).

• Analysis of an Feature takes a Business Analyst an average of 20 working days. The team has 4 Analysts so 24 Features will take 120 working days to specify.
• Development of a User Story takes 2 developers (possibly a Programmer and a Tester) 5 elapsed days to complete. With a team of 20 developers, this phase will take around 120 working days to complete the 240 stories.
• Release typically takes 10 days elapsed regardless of these size of the release. We assume however that development and analysis work may continue in parallel with this phase.

A View of the Features in Process

How do the 3 agility metrics look with this process?

• queue replenishment period: this is around 120 working days. When a release comes out and the development team start on the next release, the analysts start on the release after that, so its 120 days between each of these requirement selection meetings.
• lead time: from selection of the 24 Features, through their analysis (120), development (120) and release (10), the wait is 250 working days
• release cadence: this is the 120 working days between releases

The average Delivery Rate for this process is 0.2 Features per working day.

Let's compare this with the ideal continuous process assuming the same productivity as above but releasing as frequently as possible. Now such a change cannot be made immediately - it's a major mistake to increase the frequency of releases without improving automated build and test for example. It's more important to select smaller incremental steps where the risk of breaking the system is lower but effectiveness (including agility) can be shown to improve. But for the purpose of our thought experiment let's jump forward, assuming the basic numbers are the same, but we've reached a stage where the process can run continuously. Let's also assume the 4 analysts working together could deliver one Feature spec in 5 days rather than the 20 taken by one lone analyst.

WIP Limits for the New Process

A new Feature will be started on average every 5 working days. Dividing our 20 developers into 4 teams will allow each team to deliver separate Features, averaging 20 development days per Feature. The average arrival rate of completed Features will be 1 every 5 days. Unless we can reduce the 10 days required for each release, we would have to release 2 Features at a time, but even so our agility metrics will show massive improvement.

• queue replenishment cadence: as a new Feature is started every 5 days, we can select Features one at a time every 5 days (down from 120). If the business selected 2 at a time the period would be 10 days.
• lead time: from selection of each Features, through its analysis (5), development (20) and release (10) the wait is 35 working days (down from 250!)
• release cadence: this is now down to 10 working days between releases (down from 120). This is the minimum without addressing the release process itself - an obvious place to look for further improvement.

The average Delivery Rate is still 0.2 Features per working day (this is inherent in our assumptions above). 

The conclusion from this experiment? Agility can be measured and it can be improved radically by changing batch processes into continuous ones (or making batches as small as possible). Agile practices are needed to enable effectiveness and improved productivity at low batch sizes, but the pay-off is seen in the reduction in the times between queue replenishments and releases... and most especially in the reduction in lead time.

Footnote on Little's Law calculations:

The improvements in the 3 agile cadences in this thought experiment are based simply on adding up the times suggested by our scenario. We can derive these results using Little's Law instead, which is easier if the numbers don't drop out quite so easily as our simple scenario. First let's re-run the scenarios above.

Here's the initial waterfall process:

Average Delivery Rate = 0.2 Features per day
Average WIP = 24 Features in Analysis + 24 Features in Development + (24 in Release) * 10 days / 120 days
Thus Average WIP = 50 Features

This results in an Average Lead Time = 50 / 0.2 = 250 days

Here's the more continuous process:

Average Delivery Rate = 0.2 Features per day (still)
Average WIP = 1 Feature in Analysis + 4 Features in Development + 2 in Release
Thus Average WIP = 7 Features

This results in an Average Lead Time = 7 / 0.2 = 35 days

What would be the impact if we could reduce the 10 days of the release process down to 1 day (maybe by introducing techniques from Continuous Delivery). What would the new Lead Time be? Is it just 9 days less? Let's see.

With a 1 day release process, we can deliver Features one by one with a release on average every 2.5 days. So the figures become:

Average Delivery Rate = 0.2 Features per day (still our assumption)
Average WIP = 1 Feature in Analysis + 4 Features in Development + (1 in Release) * 1 day / 2.5 days
Thus Average WIP = 5.4 Features

This results in an Average Lead Time = 5.4 / 0.2 = 27 days

The reduction of 8 rather than 9 days is not easy to derive intuitively, proving the worth of Little's Law to continue the thought experiment and gauge the impact of improvements to other aspects of the process.

Why not try a few scenarios yourself?

Evolution and the culture of an adaptive organisation

One of the books that has most influenced my thinking about agile methods recently is The Origin of Wealth by Eric Beinhocker (Random House 2007, McKinsey & Company Inc. 2005). It's a book about economics rather than software products or development processes, so it's perhaps unsurprising it's less well known in agile circles than it ought to be. However it contains some key ideas vital to understanding the context of agile methods, specifically how products and processes evolve. Significantly it points the way for agile methods to focus on the evolution of process rather than simply the evolution of products - an insight Kanban has brought sharply into focus in recent times (not without controversy it has to be said!).

The mechanisms of evolution are the same whether the subject of evolution is a life form or the elements of an economy. These mechanisms - copy; differentiate; select; and amplify; repeated over many iterations - occur in all instances of evolution. Beinhocker demonstrates that the process by which "Physical Technology" and "Social Technology" have developed in the history of human economic activity is evolution. It's not just like evolution, it actually is evolution, with similar characteristics advantages and drawbacks that arise from it.

Furthermore its success is absolutely remarkable in both the scope of economic growth and its accelerating speed. Physical Technology refers to the artifacts of human activity from pottery and stone axes to razor blades, mobile phones, insurance policies and car washes - "product" is perhaps a simpler term. Social Technology refers to the means by which humans organise themselves and the mechanisms they use to produce the products of the economy - again I prefer a simpler term, "process". Products and processes have been evolving in an accelerating manner over millennia. At this point in history it is essential to economic survival, not only to understand these mechanisms, but also to actively participate in them. As others have said: "change - like survival - is optional!"

One of the most relevant sections of what is after all a fairly dense 600+ pages, occurs at the end of the book. Beinhocker turns his attention to how businesses need to behave within the very rapid cycle of innovation that now characterises the economy. Evolution is taking place outside organisations whatever organisations do. Whether it happens within an organisation however, and thereby enables the products and processes of the organisation to evolve sufficiently to survive and thrive in the new context, depends on the culture of the organisation - whether it is an "adaptive organisation". Rigid hierarchical organisations can survive if their context (market) is either controlled by the organisation itself or changing only very slowly. When enough change occurs to allow new organisations to compete, such rigid organisations disappear. In fast moving markets only "fit" organisations can survive in the long term.

Here is Beinhocker's summary of the characteristics of adaptive organisations that can survive in today's markets:

• Performance norms
• Performance orientation – people go the extra mile, continuous improvement expected
• Honesty – people are honest, transparent and face reality
• Meritocracy – rewards are set on the basis of merit
• Cooperating norms
• Mutual trust – trusting and trustworthy
• Reciprocity – the golden rule
• Shared purpose – common goals above personal
• Innovating norms
• Non-hierarchical – quality of the idea over status of proposer
• Openness – curious, outside thinking, experiment, seek the best
• Fact-based – facts rather than opinions ultimately count
• Challenge – competitive urgency, race with no finish line

These are the characteristics that allow evolution to occur within an organisation - not just by its death and replacement. Are you struck, as I was, how closely this mirrors the goals and norms of agile teams?

It is no coincidence that agile values match those of adaptive organisations. Agile methods recognise evolution as the key mechanism at work in improving products and improving processes. Kanban as defined in +David Anderson's Blue Book explicitly calls out evolution as the mechanism of change in its principles. Furthermore it has made the key leap beyond earlier agile methods that proposed a static Social Technology to implement evolving Physical Technologies (i.e. the use of a pre-defined process). Kanban applies evolution to the Social Technology itself. Kanban in that sense is a meta-process, a means to keep organisations in Beinhocker's "fit" state by evolving the processes it uses. In my view this makes it important for all agile teams to understand, whatever label they apply to their own current process.

The way you yourself change is at least as important as the way you change the things you are producing.