December 10, 2005

How to Select a Software Project Estimation Tool

Here’s a follow up to my earlier entry on Stan Rifkin’s wisdom. In Part 1, Stan basically said “Don’t plan beyond what you’re capable of delivering.” This might seem to be pure common sense, and that it is so obvious it’s almost insulting.

But if you consider this fact – projects that meet their deadlines and cost commitments do so by cutting out almost half of what they promise, then it follows to reason that most teams over-commit by a factor of two! Moreover, these overruns are frequently in the millions or tens of millions of dollars. One team that called me in for help in recent times was suffering an overrun of $11 million. When this happens between companies, they sue each other. My friend Ed Yourdon, an intellectual giant in the technology field, is now – almost exclusively – an expert witness for software-related lawsuits.

These things tell me that somewhere along the way, companies drastically lose sight of their estimates. (It’s my belief, as you’ve probably read on this blog, that a major cause is the blinding bias caused by the deadline.) Estimating is very very hard. Often you need a computer to help do it right. If you were to shop for a software project estimation tool, here is what Stan says to look for, in his words:

1) The underlying algorithm is published in the public domain. No surprises. If the duration=effort/resources formula is in the tool, I would reject it. The relationship among duration, effort and resources is not linear.

2) It accurately estimates completed projects. We use a few typical completed projects to see if the tool estimates them accurately in retrospect. If the tool cannot accurately estimate our type of work, we do not want to use it on real, future projects.

3) I don’t want to subjectively “guess” at the values of variables. I only want to work with items that I can objectively measure during the conduct of a project.

4) The assumptions of the tool mirror my realities. Estimates are application-area specific so we need to be sure that the candidate tool has been particularized for my application type.

5) The tool will not generate an impossible schedule. Clearly, there is a region or range of project values (duration, effort, rate of adding people, quality, and number of features) such that it is impossible to accomplish a project with that combination. I want to know that set of values so that I am sure to steer clear.

6) The tool takes into account the effects of schedule compression. In software projects we are commonly asked to produce the minimum duration estimate; i.e. what is the effort required for the stated requirements such that the project will finish in the shortest time? One of the reasons I like tools that model compression is that we are commonly given the delivery date during the commitment process, so I need to be able to ask and answer whether I can stand (manage) the compression in the dictated schedule.

7) I want a range, not a point, estimate and the probability of achieving it. We need to know the probability or risk for each feasible band of estimates. This way, we can adjust our input parameters in order to compute not only an answer about duration, effort, quality, and features, but also a risk level that the organization can tolerate.