Software Estimation Roulette
6 February 2003
by Michael Mah
In a recent survey by Cutter Consortium of more than 100 software development organizations of varied sizes, the most common method of software estimation was — drum roll please — “gut feel.” People would pick a number for cost and schedule estimates based on rough judgment of experienced developers nearly 50% of the time.
That’s a lot of projects relying on what some people also describe as “wet finger in the wind”! It’s very likely that more than US $250 billion is spent on software projects in any given year. Imagine $125 billion of those projects being “estimated” by gut feel.
These projects probably also include scenarios that my colleague, Cutter Business Technology Council Fellow Tim Lister, once described. Tim gave an amusing demonstration at a recent conference on what he thinks is a typical interaction between a boss and subordinate on software estimation. It goes something like this:
Boss: “Tell me, Sally, when do you think our project, the one with the October deadline, will be done?”
Sally: “Honestly boss, I think it won’t be ready until November.”
Boss: “Hmmm. That’s a bad estimate, a really bad estimate. I’m afraid that won’t do. Try again.”
Sally: “Uh, well let me see. October?”
Boss: “Sally, that’s a GOOD estimate!”
The crowd roared. Why? Because it’s true! That’s not “rough judgment of experienced developers.” That’s emotional and psychological blackmail. And it happens all the time.
It’s logical that knowledge of experienced developers is important for software estimating. But political pressures and retrofitting answers to others’ expectations can corrupt the results in spite of the best intentions. It’s my belief that these dynamics contribute to poor estimates that result in project overruns, slippages, and high defects. Indeed, our track record as an industry may be why so many CIOs have sought refuge in outsourcing projects to places like India.
A couple of weeks ago I was given the latest scoop on a huge project at a major retail conglomerate. The company awarded a $20-million contract to a company to build them a distribution system we’ll call the “Jetson Project.” Here are the highlights. The contract was awarded as fixed price. It had to be done in 10 months after signing — so there was a fixed schedule. But there was no spec. After the contract was awarded, one month was allocated to define the requirements. (You read that right — the requirements weren’t fully known, but there was a deadline and a fixed price estimate, so they signed a $20-million contract.)
After contract signing, more than 125 people were ramped onto the team (supposedly to finalize the requirements and to start coding like mad). The effort estimate at that point was that the software part of the project would take about 46,525 person-days. It was determined by “the judgment of experienced developers.” Each task was estimated for the number of people per task and rolled up in a giant spreadsheet. But after a few weeks following the contract award, the cost of the project was “reestimated” — at $30 million. Ten million dollars of projected overrun, and the project had barely begun.
Here’s what was interesting. Both the customer and the supplier said that the original estimate of 46,525 person-days, or 2,160 person-months, was irrespective of the schedule. That is, if the schedule was 10 months, the effort would be spread over the 10 months, and they’d ramp up to 216 people. If the schedule were compressed to 8 months, then the effort would be divided by 8 months to determine the staff size, and they’d ramp up to 270 people.
You get the math. Nine labor months could also yield a baby in one month, if you ramp up to nine women by “compressing” the schedule. That’s the danger with bottom-up effort estimating. It may work for one- or two-person projects, but when you get a large team of knowledge workers, the effort estimate, produced with mostly good intentions, often gets jammed into whatever deadline is on the table. Kooky math. Project tradeoff laws proven by researchers like Fred Brooks, Larry Putnam, and others get violated all over the place. Do you really think you can cut a schedule in half by doubling the people? Subjective estimates (and sadly, many simple project management tools) use this logic all the time. Like I said, kooky math.
I compared their implied productivity to industry statistics contained in a commercially available software estimation model. In short time I had the skinny on how this project stacked up. Although the overall implied productivity was in the norm, the deadline and the staffing plans were just plain nuts. The model predicted that the defects would soar through the roof if they tried this (like a flight simulator might predict a crash). A big sign flashed, and it read: “Danger, Will Robinson!”
And what about the Jetson Project? In the initial months of complete chaos, desperate workarounds ensued, mostly around attempts to purchase off-the-shelf packages where possible. In the end, after spending millions, the project crashed. It was abruptly cancelled, and the 200 or so developers suffering from IT burnout went home.
Software development is hard. Reliable commitments and reasonable expectations are critical to establish workable and sustainable relationships between development teams and customers. So for the $125 billion or so in development projects out there using “gut feel,” my heartfelt wish for them is that they don’t try to get a baby in one month with nine women. If just a few savvy managers out there get the message and as a result make meaningful shifts in their strategy and change how they estimate and negotiate important projects, then I will have had a good day.
— Michael Mah, Senior Consultant, Cutter Consortium
Back to Top | Advisory Service
© 2003 Cutter Consortium. All rights reserved. Comments/suggestions to firstname.lastname@example.org.