August 21, 2008

I really liked this article by Jim Highsmith who is the Director of Cutter’s Agile Product & Project Management Practice, so much so that I thought to post it here.

Much of Jim’s writing on topics like these are available via Cutter Consortium. Ask for Jack Wainwright who can set you up with readership passwords and trial evaluations of Cutter’s content.

Cheers 🙂 Michael

—————————–

Intrinsic Quality: Why Testing Takes Time

This is a continuation of previous Advisors on quality, specifically ones on intrinsic quality (see ” Investigating Agile: Inside and Out,” 19 June 2008 and ” Intrinsic Quality?” 3 July 2008). I want to address a very basic question: why is technical (intrinsic) quality so important? In previous Advisors and in the agile literature, the reduction of technical debt has been widely discussed. Agile pundits pose that continuous, comprehensive (every day, every iteration), automated testing is required to be truly agile. In this article, I’d like to address three very important aspects of why the focus on technical debt and testing are so critical: the impact of code quality on testing time, error location dynamics, and error feedback ratio.

Many people estimate testing time completely incorrectly — mostly because they don’t understand testing. The rough guideline they may use is something like: “Well, it took five days to code it, I guess it will take about three days to test it.” While this rough estimate may work at times, testing time in general is not related directly to coding time. Testing time is related to defect density. For example, take a coding effort that takes four developers 10 days and they produce four KLOC (thousand lines of code). Assuming a half day to find and fix a defect, the testing time for a team that produces a module with one defect per KLOC (an achievable level) is two days. Code produced that had 15 defects per KLOC (very possible with a team that does minimal unit testing nor any automated testing) would require 30 days of testing time!

Many development teams, and many managers, wonder why testing takes so long — and blame the testing team (the coding team made their schedule!). The greatest impact on testing time is not the testing team, however, but the coding team — and their defect densities. A number of quantitative studies done in recent years by Cutter Senior Consultant Michael Mah attest to the higher quality of many agile projects and the positive impact on scheduling.

A second testing issue is error-location dynamics. A number of years ago, a large computer manufacturer did some studies of the time it took to find errors in software. The curve goes up from one or two hours to find easier defects to more than 50 hours for a small percentage of hard-to-find defects. Years ago, in one major airline reservation system, it took more than six months to find a bug that brought both primary and secondary systems down. One question this raises for testing is: how much money can you spend looking for unfound defects? The answer for a computer game and the space shuttle’s avionics software would be much different. The agile practice of refactoring (both code and tests) can significantly decrease the percentage of hard-to-find bugs by improving code design, thereby reducing testing time. There will always be a curve of harder-to-find bugs, but the shape of the curve can be greatly altered by producing quality code.

The final testing factor to explore is error-feedback ratio, which is the number of new defects injected when fixing existing defects (20 new defects generated in fixing 100 defects would be an error-feedback ratio of 20%). Several years ago, Jerry Weinberg conducted studies on error-feedback ratio and found that a 20% difference in feedback ratio leads to an 88% difference in completion time (bad enough), but the next 10% increase leads to a 112% increase.

Have you ever worked on a project in which the code never seemed to stabilize, no matter how much testing was done? If the code has a high defect density to begin with, then it will probably have a high error-feedback ratio as well. Low-quality code causes worse error-location dynamics. Have you ever worked on a project where the testing seemed to take forever — multiples of the budgeted time? All three of these factors we’ve discussed show why poor code (high defect density, lengthy error location curves, and high error-feedback rations) can lead to an inordinate amount of testing — testing that can never (and I mean never, no matter how much testing is done), result in a high-quality code base.

In every maintenance release, the problem gets worse.

Agile developers and testers know that reducing technical debt (increasing intrinsic quality) is important. These three intrinsic quality factors — the impact of code quality on testing time, error location dynamics, and error feedback ratio — can help explain technical debt.

I welcome your comments on this Advisor and encourage you to send your insights on agile techniques and practices in general to me at jhighsmith@cutter.com.