Complexity Doesn’t Scale
February 10, 2006
Last week, an excellent piece by my fellow Cutter Consortium colleague Ken Orr crossed my inbox. I decided to excerpt it here since it directly speaks to a subject near and dear to my heart: software complexity.
After reading this, you may want more access to insights from various Cutter authors like Tom DeMarco, Ed Yourdon, Tim Lister, Rob Austin and scores of other experts. We publish research on Agile methods, Outsourcing, Business Technology Trends, Benchmarking, and the like. Check out the Cutter Consortium website at www.cutter.com. Sign up for trial subscriptions by contacting us there. You’ll be plugged in to some real interesting stuff!
Complexity Doesn’t Scale
by Ken Orr, Fellow, Cutter Business Technology Council
Cutter Senior Consultant Tom Welsh asked in a recent Business Technology Trends Executive Update whether software development had gotten too complex. He asked for feedback, so here it is: the answer to Tom’s question is unquestionably “Yes,” software development has clearly become too complex! While it is true that the software that people use today is more sophisticated, at least at the user interface level, the complexity of software development has clearly spun out of control.
There are plenty of villains in this piece. There are the hardware and software vendors who have pushed new generations of user interface, operating systems, and programming languages while largely ignoring business analysis, requirements, and design. And there are the software developers who, until many of their jobs were swept away by outsourcing, were so enamored with the latest bells and whistles that they lost track of delivering high-quality, easy-to-maintain software.
There are also the software tool vendors who stopped working on Computer Aided Software Engineering (CASE) tools in favor of more and more complex development environments. As I work with clients around the world, I am amazed how complex their development environments are and how difficult it is to do the simplest things. In previous Advisors, I have commented on how difficult something as simple as deployment (of anything) is today.
This is a travesty! Deployment should be as easy as pushing a button. To use a tried and true object paradigm: software tool vendors ought to “hide” the complexity of deployment from the developers.
The same is true of security. I have watched development teams all over the world struggle to get their programs past operational security barriers long before they should have been worrying about fitting into role-based security or biometrics or anything like that.
Complexity is killing us and it doesn’t have to. We need to reduce the high-order complexity of our systems and our programs. But we can’t program our way out of complexity, we have to architect/design our way out! Recently, I read a long article on ways around the “buffer overflow problem,” which continues to allow hackers to break into our operating systems. We know how to solve the problem — make it a systems error with no return — but there are some presumed performance problems, so we add complex, tricky code to try to get rid of a design problem and instead confound our difficulties.
Complexity doesn’t scale! It’s much like Michael Jackson used to say of optimization, which is the reason most often given for complex design, “Don’t do it, and if you have to do it, don’t do it yet!” Every generation of new hardware often erases the need for the previous generation’s complex optimization schemes. But complexity, once loosed on the software world, is nearly impossible to take back, because, naturally enough, people take advantage of it.
In the end, nothing scales like elegant design. Despite nearly a generation of “denormalization” tricks, Codd’s rules of normalization still yield databases that are maximally flexible and consistent to update, where denormalized databases are enormously difficult to update or extend.
Every once in a while, I end up talking about the design of things like operating systems or teleprocessing or workflow management systems design, and people go off on how hard these tools are to use and to manage. Having worked in all of these areas, I can tell you that they don’t have to be so difficult or so hard to manage.
All of that is bad design — the result of having too many coders and too few designers and architects involved. The fact that more and more people have the term “architect” in their title doesn’t make them real product architects. The real architects fight complexity every day. They know that giving in to bad design makes every thing else really difficult.
— Ken Orr, Fellow, Cutter Business Technology Council