Code Coverage Fallacies

Tim Barcz wrote a great blog yesterday about the fallacies of measuring coverage. His point is that 100% coverage does not mean 100% tested. I agree with his observation.

I have had the privilege to work with many great developers and teams, but even great developers can get caught up in Tim's point. Focusing on coverage by itself is a mistake, even if you get 100% coverage. Focusing on unit testing by itself is also a mistake, even if all your tests pass.

The point is that you have to focus on coverage AND unit tests. If your all your unit tests pass but you only get 20% path (not branch or code) coverage, then you may have a problem that requires more attention. If you get 100% coverage with 1 unit test, then you also have a problem. I have an example here as well.

That said, here are a few points to take away:

1. Cyclomatic complexity is a metric that identifies how reliable a method is. High cyclomatic complexity means a lot of paths, which means a lot of decisions, which means a lot of results. So unless you can test each and every path, the results may be unreliable.

2. Path coverage (which is not the same as branch or code coverage) is a method used to determine how many cyclomatic paths were tested. High path coverage is admirable, BUT IT IS NOT GOOD ENOUGH BY ITSELF. It just means you tested the paths, it does not mean you verified the results and/or behavior.

3. Path Coverage and Unit Testing is the best solution. Here you can determine which paths were tested and, through proper unit testing techniques, verify the path's behavior.

Easy breezy.

New Versions of CoView and Comet

Yes, it's been a while since my last post :) Between family, consulting gigs and life, well, things gets busy.

Anyway, we've made some good changes to CoView and Comet, including a huge price reduction. You can now get a 1 year license for $9.99 or a 1 year community edition at no charge. Why? I'll post more about that shortly.

The differences can be found here.

CoView 3.0 is available!

So, after much development, some side consulting and a crazy spring and summer, we are very pleased to release CoView 3.0. This version now includes support for mock object creation using JMockit (thanks Rogerio!), support for Ganymede (Eclipse 3.4), support for Ant (more about that coming soon), and the ability to now highlight both executed and non-executed code.

Anyway, we will certainly be blogging more now that things are calming down a bit!

Keeping an Eye on Static Invocations

Method invocations are what drive every piece of Java software. One method calls another and together they provide the functionality that makes up your particular program. Unfortunately, it's those subordinate methods that can cause you headaches when creating tests for your method. You need them behave in a particular way during testing so that you can force your method under test to exhibit a specific, testable behavior.

In the case where the invoked methods are instance methods of some class, it's usually possible to create mock or dummy classes that provide the test behavior that you require for your method under test. But what about the case when the invoked method is static? Consider the following sample code:

This computePrice() method simply determines the price of a given RetailItem, and doubles that price if the current day is a Saturday.

There are two subordinate methods called by computePrice(). The getBasicPrice() method is owned by the RetailItem parameter, and we can most likely pass in a MockRetailItem if we want that method to return a particular value for our tests.

The getToday() method is another story. Since it's invoked statically, there's no way to inject a MockTimeUtil into computePrice(). Unless the TimeUtil class supplies other methods that will allow us to change the current day before we invoke computePrice(), we can only test the TRUE outcome of the IF statement if we run this test on a Saturday. (And who wants to come in to work Saturday just to test their code?)

It's clear that the static invocation of getToday() will be an obstacle during testing, so we're probably better off if we can eliminate it altogether. For example, we could pass the current day in as a parameter, which would easily let us force a particular outcome during testing:

This refactoring maintains the integrity of the code and simplifies our unit testing effort. Before we get carried away with eliminating every static invocation, we should consider a couple of things.

The first thing to consider is that in eliminating our static invocation we've added a new parameter to our method. As we've discussed in a previous post, each additional parameter increases your testing effort, so in some sense we've only traded one unit testing effort for another. In this case, however, the trade-in of a static invocation for a new parameter is well worth it due to the ease with which we can mock a parameter.

The second factor to consider is that every rule has an exception, and sometimes static invocations are desirable, or even unavoidable. Consider how often static methods like Integer.parseInt(String) are used. Rewriting a method under test to avoid these common static methods might actually increase your overall testing effort.

Despite these considerations, you should keep an eye on the number of static invocations you use in your code. By maintaining as few static invocations as possible, you'll give yourself the flexibility you need to complete your testing effort.

EclipseWorld

One of our partner companies, Artifact Software, will be showcasing their latest version of Lighthouse tomorrow at EclipseWorld.

Lighthouse is a single application which delivers complete traceability for software projects. Trace requirements status in real time, view history of project changes instantly, get complete project health reports in seconds and much more. Who knows, maybe they will mention us too :)

We use Lighthouse ourselves to manage our requirements, bugs, test cases, project plans, and issues. As a matter of fact, we are considering opening up access to our users so they can share their opinions of our upcoming development efforts. Interested in sharing your opinion? Let us know.

See You at the Verify Conference in DC

We will be hanging around at the Verify Conference October 29th and 30th. Let us know if you are attending! Come by and check out our latest product, hear our latest announcements, and walk away with some great swag!

Taking Exception with Exceptions

Exceptions in Java are thrown when an abnormal condition arises. Exceptions not handled by a method are passed (via the throws keyword) to its parent. More often that not, test suites focus on the normal behavior of a method, and skip over the cases where the unhandled exceptions are thrown. A comprehensive unit testing effort should test each error condition separately. Thus, fewer thrown exceptions means fewer individual tests will be needed.

Consider a method with this signature:

public void foo() throws Exception1, Exception2, Exception3

What if there's a bug in this method that causes Exception1 to be thrown instead of Exception2? In order to fully test this method, our test suite needs to include at least three JUnit tests to confirm that the proper exception is thrown when the corresponding abnormal condition arises.

Even if the correct type of exception is thrown, it may be the case that all three exception types are handled the same way, or it may be that some of the exceptions can be handled by the method itself. If we can remove even one of the thrown exception types, we'll reduce our testing effort by one-third. There is no shortcut for reducing the number of thrown exceptions, of course -- indeed, each distinct exception may be necessary -- but by keeping in mind the number of thrown exceptions as you code, you may be able to streamline your testing effort down the road.