JUnit Best Practices, Part 3 -- Constant Expectations

You've just written your new doComplicatedCalculation() method and you're ready to test it, so you create the following unit test:


It seems straightforward enough. You invoke the method under test and compare it to an expected result to see if it matches. The problem is that the use of the computeExpectedTestResult() method is double trouble.

First, it appears that computeExpectedTestResult() is intended to perform the same computation as the method under test. But if we haven't tested doComplicatedCalculation() then how can we rely on a copy of that method to verify that it works? You can't use a logical proposition to prove itself, and you can't use a method to test itself.

Second, even if you were absolutely sure that doComplicatedCalculation() is correct, any change made to this method also has to be made to computeExpectedTestResult(). You've effectively doubled your coding effort, or, at the very least, set yourself up for a fumble-fingers cut-and-paste problem.

A better solution would be to replace the computeExpectedTestResult() invocation with a constant value that you've computed yourself:


Now, if the assertion fails, you know where to look for the problem. If the method under test changes, it's also clear where to change the expected value to match it.

Using constant values in your unit test assertions reduces the guesswork in your testing effort, and increases future comprehension and maintenance efforts.

Market Driven Unit Testing

I've been in the software industry for over 15 years, focusing on unit testing for the past two, and I can honestly say that the unit testing market is a tough market to crack.

There is no doubt that unit testing is important. Most testing vendors (us included) and many developers see the value of a unit testing program, but my experience indicates that formal unit testing solutions are frequently not adopted by upper management. This leads me to believe unit testing only makes sense depending on the market you are in. Most startups (not us, of course!), for example, really don't care about unit testing. Startups are looking for early adopters, and early adopters are interested in you and your technology, not your cyclomatic complexity or code coverage metrics. VC's who fund startups actually want the risk -- the greater the risk the greater the reward.

Andy Glover, prolific blogger, published author and president of Stelligent, recently wrote a blog about the cost of software quality. In that blog, he states that writing tests for software increases the overall development cost, but that the investment is well worth it given the greater ROI.

I completely agree with Andy. Many studies, like the one NIST reported in 2004 show the liability and cost of software defects in the market far outweigh the initial cost of setting up a quality development effort. Bill McComas, attorney for Shapiro, Sher, Guinot and Sandler gave a presentation to this effect, which is available here.

But I want to take this discussion a step further. I recently had a conversation with the CTO of another startup company, who will remain unmentioned because of a potential distracting flame war -- let's call him Mike. Mike is someone I respect for a number of reasons. First, Mike helped create a company that was sold for almost $200 million in 2001. Second, he started another software company and raised VC funding, which is very difficult to do.

Our conversation focused on software quality, specifically the practice of unit testing. Mike agrees with me, on an intellectual level, about all the benefits of unit testing. Practically, however, he will never implement a unit testing program in any of his companies because of the upfront cost and potential delay in getting his solution out the door. Mike made a number of excellent points:

Consumer demand for software provides leeway for defects.

VC’s invest money to take risk, meaning they don’t care if software has some bugs, so long as they're managed well.

Companies that acquire other companies do not look at the code or outstanding defects, but instead focus on the market penetration, number of paying customers, etc.

Testing, whether it is unit testing, functional testing, or any other type of testing, will always get cut in the effort to maintain competitive advantage.

My conclusion? The upfront cost of implementing a unit testing solution may be worth it in the long run, but only in a market where that makes sense (including life- or mission-critical markets). Companies trying to brand themselves and break into the majority markets are likely candidates for a unit testing solution as well. After all, perception is reality.