Code Coverage Fallacies
Tim Barcz wrote a great blog yesterday about the fallacies of measuring coverage. His point is that 100% coverage does not mean 100% tested. I agree with his observation.
I have had the privilege to work with many great developers and teams, but even great developers can get caught up in Tim's point. Focusing on coverage by itself is a mistake, even if you get 100% coverage. Focusing on unit testing by itself is also a mistake, even if all your tests pass.
The point is that you have to focus on coverage AND unit tests. If your all your unit tests pass but you only get 20% path (not branch or code) coverage, then you may have a problem that requires more attention. If you get 100% coverage with 1 unit test, then you also have a problem. I have an example here as well.
That said, here are a few points to take away:
1. Cyclomatic complexity is a metric that identifies how reliable a method is. High cyclomatic complexity means a lot of paths, which means a lot of decisions, which means a lot of results. So unless you can test each and every path, the results may be unreliable.
2. Path coverage (which is not the same as branch or code coverage) is a method used to determine how many cyclomatic paths were tested. High path coverage is admirable, BUT IT IS NOT GOOD ENOUGH BY ITSELF. It just means you tested the paths, it does not mean you verified the results and/or behavior.
3. Path Coverage and Unit Testing is the best solution. Here you can determine which paths were tested and, through proper unit testing techniques, verify the path's behavior.
Easy breezy.
I have had the privilege to work with many great developers and teams, but even great developers can get caught up in Tim's point. Focusing on coverage by itself is a mistake, even if you get 100% coverage. Focusing on unit testing by itself is also a mistake, even if all your tests pass.
The point is that you have to focus on coverage AND unit tests. If your all your unit tests pass but you only get 20% path (not branch or code) coverage, then you may have a problem that requires more attention. If you get 100% coverage with 1 unit test, then you also have a problem. I have an example here as well.
That said, here are a few points to take away:
1. Cyclomatic complexity is a metric that identifies how reliable a method is. High cyclomatic complexity means a lot of paths, which means a lot of decisions, which means a lot of results. So unless you can test each and every path, the results may be unreliable.
2. Path coverage (which is not the same as branch or code coverage) is a method used to determine how many cyclomatic paths were tested. High path coverage is admirable, BUT IT IS NOT GOOD ENOUGH BY ITSELF. It just means you tested the paths, it does not mean you verified the results and/or behavior.
3. Path Coverage and Unit Testing is the best solution. Here you can determine which paths were tested and, through proper unit testing techniques, verify the path's behavior.
Easy breezy.
posted by Joe Ponczak at 8:59 AM
0 Comments:
Post a Comment
<< Home