21 August 2003

by Michael Mah, Senior Consultant, Cutter Consortium

The phone rang, and on the other end was a longtime client of mine. “Hi Michael, it’s Rob. I’ve got a project that I need help with — we’re thinking of offshoring a major rewrite of one of our applications. The existing app is about half a million lines of code written in Smalltalk, and we want to reengineer it in Java and IBM WebSphere. We’re issuing the RFP and want to evaluate bids from two suppliers this month. We think they can do it cheaper than we can inhouse, since the labor rates in India are so low. Plus, we need it fast. Can you help us evaluate the bids?”

“Sure,” I replied. “When are you looking to make a decision?”

“In about two weeks,” said Rob.

Egads! Two weeks was a short fuse. “Rob, if that’s the case, we’ll have to move quickly and furnish each supplier with a core metrics data capture form as part of the RFP. We’ll ask them to provide size, time, effort, and defect metrics on three or four recently completed projects, similar to the one on which they’re bidding. Then I’ll need their proposals with the same information. I’ll use our estimation model to recreate their bids and will compare them to both their historical projects and against our industry database benchmarks, which also contain reference data from offshore India projects. You’ll see their estimates plotted against these comparable projects. The data on their past projects will tell us their track record and whether their proposal is consistent with what they’ve been able to achieve in the past. Do you think they can provide data by next Friday?”

“I think they can,” said Rob. “After all, the basic data you’re asking for is mandatory even for an SEI Level 2 organization, isn’t it? These guys both claim to be Level 5. If they can’t give us this data, then they’re not Level 5, are they? I’ll set up a con call.”

I e-mailed Rob the short data collection forms based on the Carnegie Mellon industry standard metrics. Conference calls were scheduled with each supplier, and I answered their questions on the information. There were several folks from Mumbai, India, on the calls. The line was noisy and it was hard to understand each other, but eventually we got the bases covered. Nonetheless, it was a lot harder than calling, say, New Jersey. I got the sense that if this were a requirements review for software project, it wouldn’t be easy.

In the end, I was impressed with how complete the information was when it was returned to me. Not only was there complete time, effort, and milestone data, but there were also stats on the code and the number of defects found during testing and post-production, by month. One supplier provided counts of objects, modules, code per module, and function points. It was all very neat. I recreated each supplier’s bid with a software estimation model called SLIM-Estimate, and the findings were very interesting indeed.

A pattern emerged — both suppliers tended to use fairly large teams when compared against the reference database of several thousand projects built into SLIM. Their history showed this consistently, and the proposed project plans indicated the same. I surmise that, given the abundant labor pool in India, these particular companies tended toward applying large teams on projects to get them experienced and put them to work (billable hours, you know). In both cases, double the norm — they could get away with that since labor rates were about half of what you’d find in the US. Now, if you’re paying a lower labor rate for an offshore staff person, that’s a cost savings all right. But if you double the staff, the net savings is zero when all is said and done, isn’t it?

Granted, the project would be done faster. But not by as much as you’d expect. We know from Fred Brooks’ research at IBM that doubling staff on software projects doesn’t cut the schedule in half. In these cases, they aimed to finish about 20% faster. So on a 10- month project, the schedule compression was roughly two months — or about eight weeks.

The problem was that this strategy results in higher defects. So while you get the project eight weeks early, there are more bugs in the code than if you used a smaller team and took a little longer. Would you take that tradeoff? This happens on any project where you compress the schedule — it doesn’t matter if the team is based in India or Indiana; haste makes waste. It’s a software law of nature even in a mature organization. Interestingly, the suppliers were bidding fixed price on the development phase. There was no warranty after delivery, and defect fixes were “extra.” I could see that the analysis and build phase wasn’t where the suppliers would make their money — they were in a competitive bid to get the work. Nope, revenue would come from “repairs,” kind of like my car dealership. And the higher defect rates because of the fast delivery would ensure that. Lastly, to fix those bugs would take time — using up part of the eight weeks’ worth of time savings.

My work for Rob wasn’t quite finished. The contract fee for the development was only one piece. The total cost of ownership had to factor in the business analysts for his team in Houston to work with the offshore teams on requirements and testing. If he couldn’t negotiate with the suppliers to not charge extra for bug fixes, he also had to figure on the expense for that as well. One last scenario was explored — one where he chose to insource. He had a crackerjack team in Houston ready to take on this project, and they had data on their last two projects. They tracked their metrics, kept good records, and had domain knowledge the offshore folks didn’t have, so there wasn’t a learning curve associated with that.

After all was said and done, in this particular case the project ultimately went to the inhouse team. It might not always turn out this way. The offshore folks presented tempting bids, but when Rob looked closer, it wasn’t just about labor rates. Contrary to what he initially expected, the total cost of ownership brought the costs of offshore outsourcing and keeping it inhouse pretty close. Plus, since this project was strategic to distinguishing Rob’s company from his competitors, he decided that it was better to keep the technology close to home and not give it away in the name of labor arbitrage.

I think the moral of the story is that companies can fall prey to a reductionistic approach that boils complex decisions down to an overly simplistic view. Oftentimes, this view doesn’t tell the whole story. It’s not only about how cheap you can get a programmer in, say, India or China. It’s not just about cost per thousand lines of code. It’s more than that, as this case shows. If you look at things from a larger perspective, you might be surprised. Sometimes an obvious decision isn’t so obvious when you look at more than just a single dimension.

— Michael Mah, Senior Consultant, Cutter Consortium

Back to Top | Advisory Service
© 2003 Cutter Consortium. All rights reserved.