Test Often.

7 Metrics to Improve Your Unit Testing

2007-07-23T20:32:00.000-04:00

Unit testing is tough, regardless of manual or automated, test-first or test-after. With any of the approaches, there are certain metrics to pay attention to while you code. Staying within acceptable levels means your code will be much more testable, whether you unit test or not!

Parameters - Each parameter added to a method's signature increases the number of possible ways that method can be invoked, eventually creating an unmanageable combinatorial problem. By limiting the number of parameters to three or fewer, you can easily increase the testability of your code.

Exceptions - Each thrown exception represents a distinct error condition that must be tested, and error conditions are very hard to test. Keeping the number of thrown exceptions to three or less greatly simplifies your testing effort.

Cyclomatic Complexity - Cyclomatic Complexity (CC) is a measure of the decision logic in a method. The more decisions your method has, the more ways it will behave. Keeping the number of CC paths to 10 or less greatly reduces the risk of defects as well as increases the overall testability of your code.

Lines of Code - One of the oldest and simplest metrics used to measure a method's testability. A shorter method is easier to understand and test.

Paths - This metric looks at distinct data paths. Data paths are paths that link a data element's definition to its use. In other words, if you define a data element, you should use it, and if you use it, you should test it. Data paths often overlap with Cyclomatic Complexity paths.

Static Invocations - Methods under test invoke methods from external classes, and these external classes can usually be mocked or dummied up to behave exactly as required for an individual unit test. Static invocations of external methods often have an unchangeable behavior that thwarts even the best testing effort. Avoid this unnecessary coupling to external classes, and keep your external static invocations to a minimum.

Anonymous Classes - Anonymous classes make unit testing difficult because they exist only within the method under test, and cannot themselves be unit tested. Although anonymous classes are convenient (especially for GUI programming), their lack of testability and mockability can hinder your testing effort.

Define-Use (DU) Paths and Unit Testing

2007-05-02T21:48:00.000-04:00

Yesterday, I blogged about the use of cyclomatic complexity (CC) and unit testing. Cyclomatic complexity is one way of looking at the paths. Specifically, CC measures the control flow within a method.

Another way of looking at paths is to analyze a methods' data flow. A well researched metric for this analysis is called 'Define Use', or DU. A simplistic definition of DU is that if a variable is defined, it should be used, and if it is used, it should be tested.

Specifically, a DU path for a variable is a path from the defining node to the usage node, thus the "flow of data". DU paths are very good at identifying problems because the error usually occurs between the definition and usage.

CC, on the other hand, only looks at control flow. A DU path has a better chance of being executed compared to a CC path, but there are also other factors that play into that as well. Which one is better for unit testing? My personal opinion is a hybrid approach, one that looks at the linear combination of DU and CC paths.

Take a look at the same example used in the last blog:

We know that total paths = 4 and CC = 3, but what about DU? And better yet, what about trying to combine DU and CC? Would that combination benefit you? Let's take a look:

Path 1: TT (DU & CC)
Path 2: FT (DU & CC)
Path 3: TF (DU & CC)
Path 4: FF (DU)

In this example, there are also 4 DU paths. If you only test the 3 CC paths, you would end up with at most 66% coverage (path 1 is unrealizable). If you test the DU & CC combination, you would have 75% coverage.

What do you think?

Cyclomatic Complexity and Unit Testing

2007-04-29T14:53:00.000-04:00

I follow both Andrew Binstock and Andy Glover from Stelligent and have much respect for them both and would like to add to a blog that was posted.

I like their approach to using metrics to figure out the number of unit tests needed for a method. Actually, metrics should be used to determine the minimal number of unit tests. Cyclomatic complexity (CC) can be used for this, but so can other metrics like Define-Use. For those who don't know what CC is, it is defined as the minimal number of linearly independent paths within a unit of code (like a method).

The blog above provides a correlation between the number of CC paths and the number of unit tests. I think any unit test is better than no unit test, but I believe their correlation only applies if you don't know what the CC paths are.

For full disclosure, I am a co-founder of a product company (www.codign.com). The products are Eclipse plug-ins that design unit tests based on Cyclomatic Complexity AND Define-Use. We have a new release coming out very soon, which takes a very interesting spin on commodity metrics, but I'll blog about that later :)

Having worked for Tom McCabe for many years (he came up with Cyclomatic Complexity), I understand the benefits and pitfalls to this metric. The calculation is very simple: CC equals the number of decisions + 1. The proof is a bit more complicated :) , but that's the gist of it.

So, for example, a method with 2 decisions has 4 total paths and 3 CC paths. Let's take this example:

The questions I hear are, what are the three CC paths? Shouldn't you try to test them all? Can you test them all?

To figure out what the three CC paths are, you have to define your basis path (or path #1). This is very much a judgment call - it can be the longest path, the shortest path, the happy path, and so on. For us at Codign, our Path 1 (aka our Basis Path) is the path that executes all the true branches in a method. From there, or from any basis path, you simply work your way down by flipping decisions.

With the example above, there are four TOTAL paths:

Path 1: TT (both if statements execute 'TRUE')
Path 2: TF (first if statement is true, second is false)
Path 3: FT (first if statement is false, second is true)
Path 4: FF (first if statement is false, second is false)

There are 2 sets of CC paths:
Set 1
Path 1: TT
Path 2: FT
Path 3: TF

Set 2
Path 1: FF
Path 2: TF
Path 3: FT

Look at the code again and you'll notice that Path 1 from Set 1 (TT) cannot be tested (it is unrealizable). But why not test the other two paths? Given this information, is the effort required to test the other remaining basis paths that difficult?

As an aside, ever measure branch coverage? The above example can be tested with two tests (TF, FT) which would produce 100% branch coverage.

My point is this ... don't take CC at face value. I don't agree that, for example, if you have 20 CC paths you should only write 10 unit tests. I believe that, if a CC path is testable, you should test it.

But like I said earlier, not all CC paths are realizable, so is there a better way to identify the minimal number of unit tests? Ones that are realizable and measurable? There is, but I will blog about that later. :)

Meet us at EclipseCON March 6th and 7th

2007-02-21T10:41:00.000-05:00

I have the privilege of speaking and exhibiting at EclipseCON this year. Our short talk will be based on our article on The Differences and Benefits of Path, Branch and Code Coverage.

Codign will also have a booth where we will be exhibiting our latest products, techniques, and integration into Lighthouse Pro.

Stop by and say hi - we have some great giveaways, including stylin' baseball caps, great t-shirts and tons of pens. We are at booth #114.

JUnit Best Practices, Part 4 -- Don't Be Too Assertive

2007-02-14T20:53:00.000-05:00

In a previous article we examined the pitfalls of invoking the method under test many times from the same JUnit test. As a counterpart to that article, let's look at the case where a JUnit test contains multiple assertions.

Assertions are the part of the JUnit test that actually determines whether or not your method under test worked as expected. With a simple method under test, it's relatively easy to make that decision with just one assertion, but when the method under test performs more complex tasks, it's tempting to add multiple assertions to your JUnit tests to verify several different behaviors at once. Despite appearances, however, this approach can actually increase your overall testing time and effort.

Let's look at an example. Suppose we have this method under test:

separator() scans its input String and fills the three StringBuffers with the vowels, consonants, and whitespace found therein. A JUnit test for this method might look like this:

This seems like a natural test to write, but the problem here is that JUnit tests are designed to fail as soon as the first assertion fails. In the worst case, when all three assertions would fail, you'll wind up with this scenario:

First test run
The first assertion fails.
Debug/fix the code.

Second test run
The second assertion fails.
Debug/fix the code.

Third test run
The third assertion fails.
Debug/fix the code.

Fourth test run
All assertions pass.

You've had to make three separate iterations of finding and fixing problems before your test passes. Since your brain is required for each debug/fix effort, this process cannot be automated. Instead it becomes time-consuming, frustrating, and inefficient. Worse, the likelihood of introducing a new bug increases every time the code is modified.

A better approach would be to split this one test method into three tests, each with a single assertion:

Now you can run your test suite once and identify all the problems before you dive into the code to fix them. True, your test suite has increased in size, from one test to three, but you've always had three tests all along anyway, and this change merely reorganizes your test suite to reflect that. More importantly, striving for one assertion per unit test streamlines the test-debug-retest cycle and improves the overall testing effort.

New Article on Path vs Branch vs Code Coverage

2007-01-30T21:02:00.000-05:00

Every so often we come across people who talk about code coverage. It's been around for a long time, but not many understand the primary types of coverage and the differences between them.

We posted a new article on the differences and benefits of path, branch and code coverage, available here.

You can also look at Bullseye - they have a nice list of coverage techniques as well.

JUnit 4 Resources

2007-01-21T12:14:00.000-05:00

Came across a blog that contains some JUnit 4 references, including those from Codign Software :)

Something About J: Some Useful JUnit 4 Links

Eclipse DTP JUnit and Coverage Assessment

2007-01-14T14:00:00.000-05:00

With any startup, one of the most important factors in mental well-being is the feedback we receive from our users. Good, bad or ugly, we need it – its more important than getting a purchase order. Really, it’s the silence that kills us.

That said, we are pleased to get some great feedback from the folks working on the Eclipse DTP project. DTP is a top level project so it gets lots of attention. The project is run by John Graham, chair of the Project Management Committee for the Dataools Platform, and I have had the opportunity of meeting John at a couple of Eclipse events. I also met a number of other DTP committers and am impressed with not only their knowledge but with their passion. These guys like unit testing.

We not only donated CoView to DTP, but also our time to help them out. Being given a free product is great but unless you have time to use it, it becomes shelfware. Our goal, worked out with DTP, was to help them analyze their existing path coverage and create additional JUnit tests to increase that coverage. It took one of our engineers 6 hours to complete the process. He analyzed their current path coverage and added 40 additional JUnit tests using CoView. The overall cost was $288. Overall increase in coverage: 46% increase in path coverage, 22% increase in branch coverage. A detailed case study is also available. DTP also gave us some great feedback during the whole process, and we certainly appreciate it.

Thanks to DTP for letting us gather this information, and great job on releasing your latest version.

JUnit Best Practices, Part 3 -- Constant Expectations

2006-11-09T11:28:00.000-05:00

You've just written your new doComplicatedCalculation() method and you're ready to test it, so you create the following unit test:

It seems straightforward enough. You invoke the method under test and compare it to an expected result to see if it matches. The problem is that the use of the computeExpectedTestResult() method is double trouble.

First, it appears that computeExpectedTestResult() is intended to perform the same computation as the method under test. But if we haven't tested doComplicatedCalculation() then how can we rely on a copy of that method to verify that it works? You can't use a logical proposition to prove itself, and you can't use a method to test itself.

Second, even if you were absolutely sure that doComplicatedCalculation() is correct, any change made to this method also has to be made to computeExpectedTestResult(). You've effectively doubled your coding effort, or, at the very least, set yourself up for a fumble-fingers cut-and-paste problem.

A better solution would be to replace the computeExpectedTestResult() invocation with a constant value that you've computed yourself:

Now, if the assertion fails, you know where to look for the problem. If the method under test changes, it's also clear where to change the expected value to match it.

Using constant values in your unit test assertions reduces the guesswork in your testing effort, and increases future comprehension and maintenance efforts.

Market Driven Unit Testing

2006-11-08T10:36:00.000-05:00

I've been in the software industry for over 15 years, focusing on unit testing for the past two, and I can honestly say that the unit testing market is a tough market to crack.

There is no doubt that unit testing is important. Most testing vendors (us included) and many developers see the value of a unit testing program, but my experience indicates that formal unit testing solutions are frequently not adopted by upper management. This leads me to believe unit testing only makes sense depending on the market you are in. Most startups (not us, of course!), for example, really don't care about unit testing. Startups are looking for early adopters, and early adopters are interested in you and your technology, not your cyclomatic complexity or code coverage metrics. VC's who fund startups actually want the risk -- the greater the risk the greater the reward.

Andy Glover, prolific blogger, published author and president of Stelligent, recently wrote a blog about the cost of software quality. In that blog, he states that writing tests for software increases the overall development cost, but that the investment is well worth it given the greater ROI.

I completely agree with Andy. Many studies, like the one NIST reported in 2004 show the liability and cost of software defects in the market far outweigh the initial cost of setting up a quality development effort. Bill McComas, attorney for Shapiro, Sher, Guinot and Sandler gave a presentation to this effect, which is available here.

But I want to take this discussion a step further. I recently had a conversation with the CTO of another startup company, who will remain unmentioned because of a potential distracting flame war -- let's call him Mike. Mike is someone I respect for a number of reasons. First, Mike helped create a company that was sold for almost $200 million in 2001. Second, he started another software company and raised VC funding, which is very difficult to do.

Our conversation focused on software quality, specifically the practice of unit testing. Mike agrees with me, on an intellectual level, about all the benefits of unit testing. Practically, however, he will never implement a unit testing program in any of his companies because of the upfront cost and potential delay in getting his solution out the door. Mike made a number of excellent points:

Consumer demand for software provides leeway for defects.

VC’s invest money to take risk, meaning they don’t care if software has some bugs, so long as they're managed well.

Companies that acquire other companies do not look at the code or outstanding defects, but instead focus on the market penetration, number of paying customers, etc.

Testing, whether it is unit testing, functional testing, or any other type of testing, will always get cut in the effort to maintain competitive advantage.

My conclusion? The upfront cost of implementing a unit testing solution may be worth it in the long run, but only in a market where that makes sense (including life- or mission-critical markets). Companies trying to brand themselves and break into the majority markets are likely candidates for a unit testing solution as well. After all, perception is reality.

JUnit Best Practices, Part 2 -- Keep 'em Separated

2006-09-26T17:40:00.000-04:00

A common practice for those new to JUnit is to try and cover more ground with fewer tests. As long as you're writing your testAdd() unit test, you may as well invoke add() as many different ways as possible and see what breaks, right?

Wrong. Not only should a unit test focus on one specific unit, it should focus on one specific behavior of that unit. Testing multiple behaviors at once only muddies the water. Consider our hypothetical testAdd() method:

Now suppose the assertion fails. Which of the two add() invocations returned the wrong value? You won't know until you start digging. The purpose of unit testing is to take the guesswork out of error location, and this technique defeats that purpose.

One exception to the rule here is that sometimes you may need to invoke your method under test multiple times in order put things in a particular state before performing the actual test of your method. In such cases, consider using a mock object to prepare the test state in a controlled fashion. The net result is the same -- the only variable in your test is the one behavior that you're testing.

Testing many different behaviors in a single test method is a common practice because it's so easy to do, but this is a clear case of being penny-wise and pound-foolish. Take the time to separate each behavior into its own unit test.

JUnit Best Practices, Part 1 -- Don't Decide Anything

2006-09-26T14:46:00.000-04:00

Creating JUnit tests can be something of a craft when you first start writing them. In our JUnit Best Practices articles, we hope to shed some light on what makes a good JUnit test, and why.

The purpose of a JUnit test is to set up a controlled environment in which the unit under test can be run. To that end, each JUnit test should be as logically simple as possible, preferably with no decisions at all.

Consider this JUnit test for a sample method called countDuplicates():

If the getTestInput() returns null, then countDuplicates() is never invoked. If you're looking at your testing status summary, this test will be marked as a success despite the fact that it didn't test anything! Of course, every rule has an exception, so if you absolutely must have decision logic in your test case, be extra diligent to make sure that your test fails if it doesn't execute as expected.

Adding decisions to your test methods only adds another variable to the test, which is exactly what you should avoid when unit testing. Keeping your unit tests decision-free will increase their effectiveness, clarity, and maintainability.

Unit Testing + Code Coverage = Bad For Business?

2006-09-26T09:54:00.000-04:00

Measuring code coverage is a natural complement to unit testing. It provides a clear picture of what parts of your code remain untested, and helps you focus additional tests on the uncovered code. Even with just a few unit tests you can quickly cover the basic functionality of several methods in your class under test. Unfortunately, a common mistake is to measure coverage without actually verifying that the executed code is correct, and this is exactly what happens with many code coverage tools.

Consider this example:

This is a simple class that converts an array of integers into the corresponding array of Strings. Note that there is an intentional bug in getIntegerName() -- case 1 of the switch is missing a break statement and causes the method to return the wrong value.

We've created a couple of JUnit tests to start with.

We'll use Coverlipse, a popular open-source coverage tool for Eclipse, to help us measure statement coverage for these unit test. Here are the Coverlipse results.

Both JUnit tests ran successfully and Coverlipse reports 100% statement coverage for the Sample class. Looking at the detailed coverage markers in the Eclipse editor, Coverlipse reports that all statements have been covered. It looks like the perfect testing summary to show to the boss, but a deeper analysis reveals some serious problems with these tests.

Most serious, obviously, is that we did not find the bug in getIntegerName(). In this case, our test assertions missed checking for the appropriate "one" value, but it could just as easily have been the case that the method under test masked the incorrect value returned by getIntegerName(). The real reason that the bug was not found is that no assertions were made about the subordinate method.

The createIntegerNameList() method is invoked by our tests. That's the unit under test, and the only method for which we should be making assertions. The getIntegerName() method was obviously executed, but because we didn't verify its behavior, we missed an obvious problem. Until we've verified the subordinate method directly, we should consider it untested.

Even when this coverage technique is successful in uncovering bugs, it's an inefficient way to conduct testing. Imagine trying to thread a needle from ten feet away. Eventually you'll be able to do it, but how many attempts will fail first? Similarly, testing a subordinate method via its invoker is coarse work at best and impossible at worst. In our example it's easy to see how we can drive down into the subordinate method, but as a method's logic increases, this task becomes increasingly difficult. If you're examining the logic of the invoking method to determine how to increase the coverage of the subordinate method, why not just examine (and test) the logic of the subordinate method directly? Don't waste time with test that might give you a minimal boost in coverage. Test the underlying method directly.

Now let's look at the same test suite using a technique that reports path coverage for unit tests in isolation.

Notice that no coverage has been reported for getIntegerName(), which is what we expect since we haven't created any unit tests for it. Until we examine and test each of the paths identified in this method, our testing effort remains incomplete.

Unit tests are so named for a reason -- concentrate on coverage only for the unit under test and ignore all others.