Market Driven Unit Testing

I've been in the software industry for over 15 years, focusing on unit testing for the past two, and I can honestly say that the unit testing market is a tough market to crack.

There is no doubt that unit testing is important. Most testing vendors (us included) and many developers see the value of a unit testing program, but my experience indicates that formal unit testing solutions are frequently not adopted by upper management. This leads me to believe unit testing only makes sense depending on the market you are in. Most startups (not us, of course!), for example, really don't care about unit testing. Startups are looking for early adopters, and early adopters are interested in you and your technology, not your cyclomatic complexity or code coverage metrics. VC's who fund startups actually want the risk -- the greater the risk the greater the reward.

Andy Glover, prolific blogger, published author and president of Stelligent, recently wrote a blog about the cost of software quality. In that blog, he states that writing tests for software increases the overall development cost, but that the investment is well worth it given the greater ROI.

I completely agree with Andy. Many studies, like the one NIST reported in 2004 show the liability and cost of software defects in the market far outweigh the initial cost of setting up a quality development effort. Bill McComas, attorney for Shapiro, Sher, Guinot and Sandler gave a presentation to this effect, which is available here.

But I want to take this discussion a step further. I recently had a conversation with the CTO of another startup company, who will remain unmentioned because of a potential distracting flame war -- let's call him Mike. Mike is someone I respect for a number of reasons. First, Mike helped create a company that was sold for almost $200 million in 2001. Second, he started another software company and raised VC funding, which is very difficult to do.

Our conversation focused on software quality, specifically the practice of unit testing. Mike agrees with me, on an intellectual level, about all the benefits of unit testing. Practically, however, he will never implement a unit testing program in any of his companies because of the upfront cost and potential delay in getting his solution out the door. Mike made a number of excellent points:

Consumer demand for software provides leeway for defects.

VC’s invest money to take risk, meaning they don’t care if software has some bugs, so long as they're managed well.

Companies that acquire other companies do not look at the code or outstanding defects, but instead focus on the market penetration, number of paying customers, etc.

Testing, whether it is unit testing, functional testing, or any other type of testing, will always get cut in the effort to maintain competitive advantage.

My conclusion? The upfront cost of implementing a unit testing solution may be worth it in the long run, but only in a market where that makes sense (including life- or mission-critical markets). Companies trying to brand themselves and break into the majority markets are likely candidates for a unit testing solution as well. After all, perception is reality.