Communications of the ACM
Home/Magazine Archive/January 2013 (Vol. 56, No. 1)/How We Build Things: . . . and Why Things Are 90%.../Full Text
The business of software
How We Build Things: . . . and Why Things Are 90% Complete
Credit: Teknorezin.ro
It seems to be a law of software development that things always take longer than we expect. So we expect them to take longer and make appropriate allowances and they still seem to take longer. Even if we carefully account for unrealistic expectations as Tom DeMarco so testily observed in Why Does Software Cost So Much?2 we are regularly and unpleasantly surprised when we overrun still more. And often the person doing the work is astonished that: it takes longer than it was thought, planned, and committed to; even in the middle of the work, while acknowledging it has taken longer than expected to this point, it will still take longer to finish than will be predicted right now; and we continue to repeat this behavior.
This scares project managers. Due to the rather opaque work done in the business of software it is difficult for a project manager to know what progress has been made without asking those doing the work how they are doing. The code is written, but is it the right code? Is there enough of it? The test was run, but was it the right test? Was the result what we expected? If it was not what are the consequences in terms of progress? If we unearthed a defect does that put us further ahead (one less defect to deal with) or further behind (yet another thing gone wrong)?
Zeno's Paradox of Progress
When a project manager talks to a designer, programmer, or tester and tries to get a sense of how "complete" the assigned task is, the normal reply is "about 90%." Asking the same person the same question the following week often elicits the same reply. Sometimes for weeks, with equal amounts of rising embarrassment and certainty that this time the estimate is good, the programmer or tester will affirm that he or she is really, truly, 90% complete. If the embarrassment gets too high, the engineer might employ the "Zeno's Paradox"a approach of halving the distance to the goal each time. So 90% becomes 95%, then 97.5% then 98.75%. We may laugh at this, but perhaps it contains a germ of truth.
A programmer describing progress as "90% complete" is not usually intentionally lying, but clearly the implied 10% remaining time to complete is not correct. So why do people say this and why do project managers continue to operate on this assumption? There are several reasons.
Second Order Ignorance (2OI)
Systems development is primarily a knowledge discovery activity. Even in coding, the thing that takes the most time is figuring out what we don't know. If systems development were a matter of transferring what we already knew into code, we would build systems as fast as we can type. And for some aspects of development, such as testing, discovering what we don't know represents almost all the effort.
We may be quite aware of some of the things we don't knowthis is lack of knowledge or First Order Ignorance (1OI).1 But sometimes there are things about the system we don't know we don't know, which means we have Second Order Ignorance (2OI). Building what is already known is easy, resolving clearly defined questions is a bit more challenging, but discovering what we are not aware of is by far the most difficult and time consuming task. So asking the question "...how complete are you?" or its corollary "...how much more work is there left to do?" is the equivalent of asking "...how much do you not yet know?" There is no easy way for someone to answer that question. Most people extrapolate from their past experience and what they have learned on the task so far to figure how much they might have left to learn. But they don't actually know. Then a certain amount of optimism, a modicum of professional pride, and perhaps a dash of hubris and the calculation yields the ubiquitous "90%." I have observed that years of experience have a way of tempering one's optimism and experienced engineers do tend to be more realistic about what they don't know yet. But there is another reason why we overestimate progress.
Linear Equation. Project managers get confused because they associate percentage complete with percentage of resources used. They assume a linear relationship between these two variables (see Figure 1). So when the quantity produced = 90%, the resources used must be = 90%. But in development this is not so.
Rayleigh Curve. Research has determined that the rate at which we build things in systems development can be described by a Rayleigh Curve3,4 (see Figure 2). This curve shows the amount of work that can be done varies with time. It rises from zero to a peak, perhaps a third of the way into the activity, and then drops off back to zero. The cumulative version of this curve is shown in Figure 3.
Non-Linear. If we compare Figure 3 with Figure 1 we notice the cumulative curve relationship of percentage of product complete and percentage of resource used (time in this case) is not a straight line. Some percentage on one axis does not mean the same percentage on the other. In fact, in Figure 3, when the product is 90% complete, the activity is only about halfway through its total time. This is partly where the miscalculation of progress comes from. If developers or project managers, consciously or unconsciously, apply a mental model that is linear to something that behaves non-linearly it is not surprising that they incorrectly assess progress.
Three Stages
This model also makes intuitive sense. In Figure 3, the curve has three distinct stages and they apply as much to painting a room as they might to building software.
Preparation. The environment must be prepared before work can commence. For software this means setting up servers and version control systems, acquiring tools, planning and estimating, sourcing staff and other essential activities that take time but might not actually create much in the way of software product. When painting a room, we have to choose the color scheme, measure, tape up the woodwork, and buy the paint, all before we can start putting it on the wall.
Production is the steepest slope where the maximum rate of measurable work occursthe most code is written, the most paint is applied.
Proving is the final long tail of the process. In software it is completing the exception code (the mainline code was done in production), testing it to prove it works and fixing it when it does not. This always seems to take longer than it should partly because we invariably find out things we were not expectingwhich is where the Second Order Ignorance comes in. In painting, this is the detail work, the tricky corners and, of course, the cleanupwhich also always seems to take longer than it should.
Mathematical Solution
There are many reasons why people overreport progress: optimism, pride, pressure to show results, inexperience, poor resource allocation, even how they are compensated, and they all play a part in overstating development and underestimating the remaining work.
But realizing that we cannot calculate exactly how far we have left to go because we do not know exactly where we are going and simply using the correct mathematical model of progress would certainly help.
References
1. Armour, P.G. The Five Orders of Ignorance. Commun. ACM 43, 10 (Oct. 2000), 17.
2. DeMarco, T. Why Does Software Cost So Much? Dorset House, 1995, 4.
3. Norden, P.V. On the anatomy of development projects. IRE Trans. Eng. Management 7, 1 (1960), 40.
4. Putnam, L.H., and Myers, W. Measures for Excellence. Yourdon Press, 1992, 46.
Footnotes
a. The Greek philosopher Zeno of Elea was credited with identifying an apparent paradox in movement where, in order to travel any distance, we must first move half the distance, and then half the remaining distance, and then half that remaining distance. The sum of 1/2 + 1/4 + 1/8 +... never reaches 1 so therefore we should never be able to move anywhere.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2013 ACM, Inc.
Comments
John Switlik
This article motivated me to pick up a subject that had seemed to lose importance, for several reasons. No joke: I used Zeno throughout my career doing development work. It was a nice way to think about earned value.
Modern approaches push the uncertainty on the user. Who now even thinks of trying to specify how some system will look or work. Any attempt at such is full of problems.
The chaotic collective that we call markets (and their servers) might be an example. Just recently, there was news that an exchange had processed incorrectly.
http://fed-aerated.blogspot.com/2013/01/zeno-in-modern-context.html
Phillip Armour
The classic Zeno's paradox is, of course, not a paradox at all. The distance shortening of 1/2 + 1/4 + 1/8 + 1/16 +... does indeed never reach zero and if we were to spend the same amount of time traveling each increment we would never get anywhere. But of course we take 1/2 the time to move the 1/4 distance increment that we did to move the 1/2 distance increment, so the time increments are getting shorter at the same rate that the distance increments are getting shorter.
However, the surface paradox does somewhat resemble the eternal 90% complete issue in that people's halving the distance to the goal line each week has a basis in fact--specifically that by most measures of "production" in software the overt production does "slow down" as we get toward the end. This is the long tail of the cumulative Rayleigh function that I showed in the article.
As an aside, Diogenes the Cynic refuted Zeno's point by simply walking across the Athenian senate floor.
Thomas Johnston
With forty years of IT development and support work behind me, the only projects I have ever seen completed on time and within budget were those that delivered much less than what was explicitly or implicitly promised when the project began. Having been a contractor for over half of that time, I have seen a large number of projects, in nearly two-dozen different companies. So I don't think that my experience is atypical.
But I do think that there is a way to do better, although I was never given the go-ahead to try it. All too briefly, it is this: treat the deliverables of any project as release 2 stuff. Define a subset of those deliverables as a project-internal release 1, and then carry out a full project life cycle on that subset with a delivery date that leaves at least a third of the project calendar. Then use that last third to expand the release 1 into the final project deliverables. One of the most important things this does is work out all the communication and support snags with outside personnel and departments who cooperation is required to successfully start out, continue, and complete the project.
If this is just that tired old horse "iterative development", then I'll just have to say that in the course of a long career, and projects with many different companies, I've never seen it actually used.
Tom Johnston
Phillip Armour
Forty years Tom? Me too. That's a long time.
And a long time ago, Fred Brooks had a whole chapter of "The Mythical Man-Month" called "Plan to Throw One Away" suggesting this. But we don't learn it seems.
The two cycle release makes sense from the "Orders of Ignorance" perspective. We start off a project with things we know (Zero Order Ignorance or 0OI), things we know we don't know (1OI) and things we don't know we don't know (2OI). It takes one pass to convert the 2OI into 1OI and the second pass to convert 1OI into 0OI which is the provably correct knowledge now resident in executable form.
This is also what happens in testing which generally consists of two stages:
1. find a bug (2OI->1OI = become aware of what you don't know) and
2. fix a bug (1OI->0OI = resolve the answer to the now-clear questions).
How did you come up with the 2/3:1/3 division between release 1 and 2?
Displaying all 4 comments