Communications of the ACM

The business of software

Return at Risk

By Phillip G. Armour
Communications of the ACM, September 2010, Vol. 53 No. 9, Pages 23-25
10.1145/1810891.1810902
Comments

In a previous column,^a I noted that many organizations do not seem to explicitly calculate the "cost of risk" on their projects. Companies may acknowledge risk, identify risk items, implement risk management programs, track risk indicators, and adjust project management actions to mitigate risk. But they often don't actually compute how much of it there is and what it will likely cost them.

Even businesses such as insurance companies for whom risk quantification is a core competency often fail to assess the cost of the risk they are taking on when they run software projects. This apparently unconscious failure to numerically deal with a critical item of business knowledge seems to extend to other disciplines too. A while ago I came across a financial services company that was routinely performing a quite incorrect calculation of ROI on its software projects.

Back to Top

Straight-line ROI

Imagine a project with an expected cost of (say) $1 million and an expected return of $1.1 million. Ignoring issues of inflation, cost of capital, and alternative investment profits, the ROI for this project appears to be 10%. This project should produce a $100,000 return on a $1 million investment. In my experience, this is the most common calculation performed by companies on most internal development projects. More correctly, it is the calculation done by those companies that actually do estimate their ROI; there are many companies that don't do it at all or do a very perfunctory job of it when they do. But that is a topic for another day. This simple arithmetic could be called a "straightline ROI." It is simply the expected returned value divided by the expected cost. It is a simple calculation, easy to compute and to understand. But, in most cases, it is also wrong.

Back to Top

The Role of Risk

The reason why straight-line ROI is wrong for most projects is simply that it does not account for risk. The return computed using the above straight-line ROI calculation will be true only if there is:

• 100% guarantee of cost containment at $1 million—that is, the project has no cost risk, the project cannot/will not run over or under in budget.
• 100% guarantee of value returned at $1.1 million—the project has no return risk, the return is fixed and invariable no matter what happens to the project.

These are the necessary conditions for the calculation to be valid. There are conditions where the cost risk and value risk cancel out and the calculated return ends up at 10%. For instance, if a project overruns on cost but is able to recoup more value than expected, it may cancel out the budget overrun. Note, this does not mean the calculation is correct, simply that the project was "lucky"^b in that the inaccuracies happened to be equal and opposite.

The moment we introduce risk, the straight-line ROI calculation does not work. If we only have a 20% probability of cost containment at $1 million, given a typical set of project conditions, the ROI is not a positive 10%, it is more like a negative 18% (!)

Back to Top

Stocks and Bonds

This is true in other disciplines. Equities typically carry more risk than government, municipal, or corporate bonds so we expect higher return to compensate us for the risk. Bonds are safer, so we are content with less income because the downside is more controlled. A less sophisticated financial consultant may show a customer what savings might be achieved over time based on the average returns of the stock market. The U.S. stock market has realized approximately 9% average annual gains over the last 100 years or so (depending on the index used and whether returns are compounded). Showing a potential investor a nice straight line of ever-increasing wealth is more a sales gimmick than a realistic prediction of return since it does not account for risk. Similarly, calculating the likely return for a software project without accounting for risk is bogus.

The financial services company I mentioned would not dream of using a straight-line return calculation for its investments but, like the insurance company that did not calculate cost of risk, it blithely performed the wrong calculation on its internal projects. And it wondered why it got blindsided by failure to achieve returns on most of its projects.

Back to Top

Risk-Weighting

To more correctly calculate the true likely return we must incorporate the cost of risk and its counterpart—what we could call the value at risk. Recently, financial markets have shown what failure to properly account for risk does to one's investment, either through not including risk in the calculation or having the risk hidden inside complex derivatives. It is very important that we learn from this in the business of software by performing the right calculation.

There are six elements to computing return at risk:

• The expected cost of the project
• The likelihood of achieving that expected cost
• The risk profile or "shape" of the cost risk distribution
• The expected returned value of the project
• The likelihood of achieving that value
• The value profile or "shape" of the value distribution

---

The reason why straight-line ROI is wrong for most projects is simply that it does not account for risk.

---

Back to Top

Expected Cost/Value

These are typically calculated using traditional estimation approaches. The cost components may be assisted by estimation tools. The value side is usually calculated using some internal assessment of operational cost containment, market expansion, revenue return, and the like. Neither of these processes is particularly easy, but they are quite well-defined.

---

Risk is always paid for somewhere: in the stock market, in insurance underwriting, and in software projects.

---

Back to Top

Likelihood of Cost/Value

These can be computed using techniques such as Monte Carlo analysis operating on the ranges of key variables that contribute to the cost (or value). There are tools available that can perform these calculations easily.^c More sophisticated financial planners will typically use this approach when laying out projections for their customers.

Back to Top

Shape of Risk/Value

This is a complex subject, the detail of which is beyond the scope of this column. The "shape" of risk/value is driven by the expected likelihood of costs and value over- or underrunning. The mistake the straight-line ROI makes it is assumes the risk profile looks like Figure 1. This probability distribution shows there is one and only one likelihood of a result. The chart shows the cost (or delivered value) of the project is 100% guaranteed at the expected value. This means the project is carrying no risk. It also means the project has no unknown factors or variables that affect cost or value. Such projects do not exist in the real world.

The simplest and most common probability distribution is the Gaussian (see Figure 2). With this distribution the likelihood of over- or under-running budget (or value) from the midpoint "most likely" is equal. While cost and value distributions are rarely symmetrical in real life, this can be a useful distribution provided the "expected" cost or value is set off-center.

Cost profiles usually look something like the curve shown in Figure 3. The distribution shows there is much more likelihood of the project overrunning its budget than there is in under-running. Unless the likelihood of cost containment is very high (the project's expected cost is set to the right of the midpoint on the x-axis) this project is carrying a high cost of risk.

Value profiles (see Figure 4) are often the reverse of the cost profiles. In general, experience shows we are more likely to underachieve our value goals than we are to overachieve them. Therefore, to better guarantee returns we would need to have our expected value moved to the left (we expect lower value delivered) of the midpoint.

Using these models, we can calculate a risk-weighted return for our projects and either choose not to carry such a high risk or to more realistically resource our projects based on the challenges they are likely to experience.

Back to Top

Real Return

The difference between straight-line return and risk-weighted return is simply the aggregate cost of risk as expressed in the likelihood of a project both running over budget and under-achieving in the value it delivers. Risk is always paid for somewhere: in the stock market, in insurance underwriting, and in software projects. It seems that few companies perform this kind of calculation, even when it is one of their core competencies—which is odd to say the least. In the business of software, we can't complain about our performance if we resource our projects but don't quantify and resource the risk on the projects. When that risk comes home—and it will—our projects will fail. And they do.

Back to Top

Back to Top

Back to Top

Back to top

The Digital Library is published by the Association for Computing Machinery. Copyright © 2010 ACM, Inc.

---

No entries found