 | Level: Introductory Alex Garrett (ljagged@thinkpig.org), Senior Consultant, The Isthmus Group Inc.
19 Jul 2005 The advent of JUnit has been a boon to developers. Unfortunately, many think it's enough to learn the JUnit API and write a few tests in order to have a well-tested application. This idea is worse than not testing at all because it leads to a false sense of code health. Learning JUnit is the easiest part of testing. Writing good tests is the hard part. This article presents some common JUnit antipatterns and shows how to resolve them.
A couple months ago, my wife and I decided to put up wood paneling in the kitchen. It was my first house project, and I did it with a hammer, nails, and a double handful of blind optimism. It was a disaster mostly because I'm not qualified to use a hammer. My wife had to fix all the divots and cracks I made with my well-intentioned flailings.
When it came time to turn my den into a nursery, I figured I had learned my lesson and borrowed my father-in-law's pneumatic nail gun. It took one-tenth the time to do the nursery as it did the kitchen, but the nail gun couldn't compensate for my other mistakes -- like forgetting to level the tops of the boards, miscutting, forgetting to inspect the boards and nailing up flawed ones, and a host of other problems that my wife was kind enough to notice. I learned something: A pneumatic nail gun does not a carpenter make.
|
What's an antipattern?
An antipattern is a recurring negative solution to a problem:
- Because it's a solution to a problem, it tends to stick around after it's implemented. If it didn't solve a problem, it would be thrown out.
- Because it recurs, it gets implemented -- and reimplemented -- by developers in a wide variety of settings.
- Because it's negative, it's not the optimal solution
The combination of these three elements is what makes antipatterns so troublesome.
|
|
JUnit: The pneumatic testing tool
To my mind, JUnit is a lot like my pop's pneumatic nail gun. Testing before JUnit came along wasn't impossible, but it was difficult. In fact, it was so difficult that it often didn't get done. And if it did get done, it was usually only applied to the parts that seemed so complex or fragile that they warranted the extra effort of tests.
JUnit turned that pattern on its ear. The dirty secret is that it turns out lots of programmers actually enjoy writing tests. And as they write tests, the customers come to expect tests. Although there are still some holdouts, most places welcome our new test-infected overlords (for more about love of testing, see Resources.
The problem is, JUnit isn't a panacea; it's a tool, no more and no less. Like any good tool (and it's one of the best), JUnit does one thing, and it does it well: It provides a framework for executing tests. More specifically:
- It provides a template for writing tests with setup, execution, and teardown.
- It allows you to organize tests in a hierarchy.
- It allows you to automatically and easily execute tests.
- It decouples the test reporting from the execution, allowing you to use different TestRunners with the same TestSuite.
Although JUnit is powerful in its simplicity, other tools have plenty of opportunities to fill the gaps. Here's what JUnit can't do:
- Automatically generate tests for a unit under test.
- Provide coverage metrics.
- Indicate when you've written a poor test.
The Land of point
Robert Binder has a marvelous book called Testing Object-Oriented Systems: Models, Patterns, and Tools. Binder is a man of rare wisdom -- a testing sage. As a testing resource, his book is invaluable. Binder opens the book by revisiting a testing problem from Scott Meyers. The problem is to write unit tests for a Triangle object.
The Java™ technology implementation has a constructor that takes three lengths; it also has getters and setters for each side. It has three methods: isIsosceles(), isScalene(), and isEquilateral(), which return true or false depending on the configuration of the triangle. The triangle is also a subclass of the Polygon type, which is derived from a Figure type. Figures are abstract types that represent objects that can be drawn on a raster display. The challenge is to write tests for this class.
Binder lists 33 tests from Meyers' original procedural solution and provides 32 more tests that are germane to the object-oriented nature of the problem. Sixty-five tests in all. Unless you work on loss-of-life-critical software, you probably have never tested code to that extent or seen it so tested. The reason isn't because you're dumb or lazy. It's because you haven't been trained in testing and because you spend your professional development time sharpening your programming skills, not your testing abilities. Why should you? JUnit makes it so easy.
The antipatterns
This section presents a few of the antipatterns I've run across or been guilty of.
Happy path tests
|
What's the Single Responsibility Principle?
A class or method should have a single responsibility. It should do one thing and do it well. The single responsibilities of the methods should support the single responsibility of their class. You can extend this principle to various levels of the system. For example, a component (an aggregation of tightly cohesive classes) should also have a single responsibility.
As you shift layers, the individual responsibilities become increasingly abstract. A method may be responsible for splitting an e-mail address into a user name and a host name. A component may be responsible for authenticating a user.
|
|
Happy path tests validate the expected behavior of the system under test. They follow the well-trodden execution path of everything going right. In a functional test, the happy path is the same -- or close to -- the use case. In a unit test, it's the same but smaller -- because your unit obeys the Single Responsibility Principle, you're testing its single responsibility.
Actually, happy path tests aren't an antipattern. The antipattern is when the developer stops at happy path tests. Happy path tests don't test the parts of the system most likely to fail (the sad paths). Usually, when you write code, you write it with the happy path in mind. You may even mentally test it with some happy path data. The boundary conditions wait, untested, for out-of-domain data that will permit them to bring your application to its metaphorical knees.
Let's say you're writing a Factorial class that has one method, eval, which takes an int and returns the factorial of that int. One happy path test verifies that Factorial.eval(3) returns 6. The chances that the implementation of this code is incorrect but still returns the correct result (a false positive) are slim:
public class Factorial {
public int eval(int _num) {
if (_num == 1) { return 1; }
return _num * eval(_num - 1);
}
}
|
Some would be satisfied with this test and move on. However, consider this implementation:
public class Factorial {
public int eval(int _num) {
return 6;
}
}
|
How's that for a false positive? If you've never encountered test-driven development (see Resources), you might object that no one would write such a simple-minded implementation. One of the practices of test-driven development (TDD) is to write the tests first, then "do the simplest thing that could possibly work" -- in this case, return 6.
Even if you don't do things the TDD way and take a crack at a correct implementation, you can still get false positives. Consider the following implementation:
public class Factorial {
public int eval(int _num) {
if (_num == 1) { return 1; }
return _num + eval(_num - 1);
}
}
|
This algorithm is exactly the same as the first, with the exception that the sequence of numbers is added instead of multiplied. The return value is the same for the value 3 (and for 1, as it happens), but it fails for all other values. The point is, it isn't difficult to get a test to pass coincidentally.
That's why it's important to have at least two happy path tests. Two tests dramatically reduce the chances for a coincidental pass, especially if the test values are orthogonal (independent of each other, or unrelated). Writing a test for values of 3 and 5, for example, would quickly show that the previous two implementations are wrong.
Validation and boundary tests
Two other types of tests need to be considered: validity (or domain) and boundary. The former asserts the correct behavior for data that isn't valid (or is out of the domain); the latter is a variant of a happy path test, but it asserts that the implementation works correctly at the boundaries of the domain.
In the example, consider what happens when you invoke Factorial.eval(-3). Most likely, you'll run out of stack space, and the program will crash. Of course, -3 isn't a valid input, so it wouldn't make sense to use it. However, there's a middle road between truth and destruction. It's called IllegalArgumentException, and it's demonstrated here:
public class Factorial {
public int eval(int _num) {
if (_num < 1) {
throw new IllegalArgumentException(
"Parameter must be greater than 0: " + _num);
}
if (_num == 1) { return 1; }
return _num * eval(_num - 1);
}
}
|
|
Effective exceptions
It's always a good practice to include the data that's causing the failure in the failure message. It makes debugging much easier and shows that you care.
Also, become familiar with and use the exceptions that are part of the Java API. Don't use a com.foo.bar.ParameterMustBeGreaterThanZeroException when an IllegalArgumentException will suit. |
|
If you've written factorial code, you've probably noticed that the code is still wrong. So, let's talk about boundary tests. One boundary exists where the input parameter is 0. This is a valid input; mathematically speaking, the factorial of 0 is 1. Executing the previous implementation results in a failing test because you expect the return value 1, but instead you get an IllegalArgumentException. You should also check the other side of the boundary, -1, to verify that you get the expected IllegalArgumentException, rather than an integer.
The appropriate tests for the other boundary are left as an exercise for you. Hint: what happens if you execute Factorial.eval(100)?
Easy tests
Like the happy path antipattern, the easy tests antipattern isn't about what is there but rather what isn't. The symptoms usually occur when the developer is inexperienced and the code is difficult to test. As a result, you see lots of tests for things that are easy to test (equals and toString tend to pop out; see Listing 1), and the real logic of the unit under test is ignored. The result is lots of passing tests that don't exercise the system, which leads to a misrepresentation of the code's health.
Listing 1. Some easy test signatures
testEqualsReflexive()
testEqualsSymmetric()
testEqualsTransitive()
testEqualsOnNullParameter()
testEqualsWorksMoreThanOnce()
testEqualsFailsOnSubclass()
testEqualsIsStillReflexive()
|
Often, the system seems difficult to test because you're trying to test a method and not a fixture. Imagine you're trying to test an implementation of a stack. Your test signatures might look something like those in Listing 2.
Listing 2. Potential test signatures for a stack unit test
testPopHappyPath();
testPopEmptyStack();
testPushHappyPath();
testPushFullStack();
testPeek();
|
Some of these are easy, as shown in Listing 3.
Listing 3. Unit test for empty stack
public void testPopEmptyStack() {
Stack stackUT = new Stack();
assertEquals(0, stackUT.getSize());
try {
stackUT.pop();
fail("Expected StackUnderflowException");
} catch (StackUnderflowException _expected) {}
}
|
But how do you test the happy path case for push?
Listing 4. Unit test for stack.push()
public void testPushHappyPath() {
Stack stackUT = new Stack();
Object item = new Object();
stackUT.push(item);
// now what?
}
|
It's a common mistake to test the implementation of the unit, rather than the contract the unit has with its clients. Let's say that the push method is implemented like so:
public class Stack {
private List elements;
...
public void push(Object _element) {
elements.add(_element);
}
}
|
You want the test to verify that the elements List now holds the Object added by push. So, you write a test like this:
public void testPushHappyPath() {
Stack stackUT = new Stack();
Object expectedElement = new Object();
stackUT.push(expectedElement);
List elements = stackUT.getElementsList();
assertEquals(1, elements.size());
assertEquals(expectedElement, elements.get(0));
}
|
The problem is you're breaking encapsulation by exposing the internals of the unit under test. Instead of testing that push puts an object into a list, you should be testing the contract that a stack has with its clients. J.B. Rainsberger calls this testing the fixture.
Now you have tests like those in Listing 5.
Listing 5. Unit tests for a stack fixture
public void testPushPop() {
Stack stackUT = new Stack();
Object expectedElement = new Object();
assertEquals(expectedElement, stackUT.push(expectedElement).pop();
assertTrue(stackUT.isEmpty());
}
public void testFILO() {
Stack stackUT = new Stack();
Object expectedOne = new Object();
Object expectedTwo = new Object();
stackUT.push(expectedOne);
stackUT.push(expectedTwo);
assertEquals(expectedTwo, stackUT.pop());
assertEquals(expectedOne, stackUT.pop());
assertTrue(stackUT.isEmpty());
}
|
You're no longer breaking encapsulation because you aren't making assertions about how the unit works under the hood. Instead, you're taking advantage of the tight cohesion the fixture displays. It doesn't make sense to have a stack that pushes but doesn't pop, so you test the methods together as part of the contract the stack exposes to its client.
When you're writing code, think about the contract -- the particular bit you're writing will expose to its client, whether it's a method, a class, or a group of interacting classes. The contract is the thing you're going to test, not the implementation details. Testing in this fashion will also help you formalize the contract so it's explicit and well defined by the tests, rather than being amorphous and ad hoc.
Overly complex tests
Tests are most successful when they're obviously correct. If a test is so complicated that you can't immediately tell it's correct, you don't know whether it's failing because the test is bad or, even worse, whether it's passing erroneously. This situation usually occurs when the system under test requires a complicated setup or exposes complicated data structures that need to be picked apart.
Consider an example where you have a code that takes some customer data and writes it out to a file of fixed records for use in a legacy system. You're not interested in testing that the records are in the correct format -- you already have lots of tests for that. You want to test that the correct data is in the records. In this situation, it's not uncommon to see tests that look like Listing 6.
Listing 6. An overly complex test
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import junit.framework.TestCase;
public class RecordTest extends TestCase {
public void testRecordContainsCorrectCustomerData() {
// setup
String expectedName = "Estragon";
int expectedId = 1001;
String [] expectedItemNames = {"A man", "A plan", "A canal", "Suez"};
Customer customer = new Customer(expectedId, expectedName,
expectedItemNames);
// execute
BillingCenter.processCustomer(customer);
// assert results
File file = new File("customer.rec");
assertTrue(file.exists());
FileInputStream fis = new FileInputStream(file);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte [] buffer = new byte[16];
int numRead;
while ((numRead = fis.read(buffer)) >= 0) {
baos.write(buffer, 0, numRead);
}
byte [] record = baos.toByteArray();
assertEquals(128, record.length); // exactly one record
String actualName = new String(record, 0, 15).trim();
assertEquals(expectedName, actualName);
int [] temp = new int[4];
temp[0] = record[15];
temp[1] = record[16];
temp[2] = record[17];
temp[3] = record[18];
int actualId = (temp[0] << 24) & (temp[1] << 16) &
(temp[2] << 8) & temp[3];
assertEquals(expectedId, actualId);
int itemFieldLength = 16;
int itemFieldOffset = 19;
for(int i = 0; i < 4; ++i) {
String actualItemName = new String(record,
itemFieldOffset + itemFieldLength * i, itemFieldLength);
assertEquals(expectedItemNames[i], actualItemName.trim());
}
}
}
|
Whew! What's going on there? In addition to verifying the correctness of the system under test, tests also serve as documentation. They should act as a roadmap for the correct behavior of the system. The intent of this test is to show that calling the static processCustomer method on the BillingCenter object with an appropriately populated Customer object results in an appropriately formatted record being written to the customer.rec file. However, this intent is lost in all the file I/O, byte-shifting, field-offsetting shenanigans needed to execute the test.
The test is probably more complicated than the code it's testing. I have no confidence that this test is correct, and I wrote it. Let's try something else. Let's abstract away all the bit-twiddling and make the test look like a test should look (see Listing 7).
Listing 7. A simple test
public class RecordTestImproved extends TestCase {
public void testRecordContainsCorrectCustomerData() {
// setup
String expectedName = "Estragon";
int expectedId = 1001;
String [] expectedItemNames = {"A man", "A plan", "A canal", "Suez"};
Customer customer = new Customer(expectedId, expectedName);
// execute
BillingCenter.processCustomer(customer);
// assert results
RecordFileFacade records = new RecordFileFacade("customer.rec");
assertEquals(1, records.getTotalRecords());
RecordFacade record = records.get(0);
assertEquals(expectedName, record.getName());
assertEquals(expectedId, record.getId());
for(int i = 0; i < 4; ++i) {
assertEquals(expectedItemNames[i], record.getItemName(i));
}
}
}
|
Now the test code clearly represents the intent of the test. There's no doubt that this test is correct because all it does is set up expectations, invoke the system under test, call getters, and make assertions. The logic has been pushed into the RecordFileFacade and RecordFacade classes. The RecordFileFacade is responsible for reading the data from the file and chunking it into records. The RecordFacade parses each record and exposes the data in a Java language test-friendly way. The other benefit is that the RecordFileFacade and the RecordFacade can now be tested, too. When the logic for picking apart the records was kept with the test, it was untestable.
The idea is to push the logic into the infrastructure. A good test should look like this:
- Set up
- Declare the expected results
- Exercise the unit under test
- Get the actual results
- Assert that the actual results match the expected results
A well-tested application doesn't consist merely of application code and tests. A substantial amount of infrastructure code serves as an adapter between the tests and the system under test. The purpose is twofold: It allows the tests to clearly express their intent, and, by abstracting the complex code into a separate layer, you can write tests for that layer, as well.
Conclusion
In many ways, testing has become much easier with the advent of JUnit. The ease in test writing extends to bad tests, as well as good tests. However, 1,000 bad passing tests are worse than no tests at all because bad tests give you a false sense of confidence.
When you're writing tests, be mindful of the quality of the tests you write:
- Don't just test the happy path; test boundary conditions and out-of-domain values.
- Don't test the implementation; test the fixture.
- Don't make your tests more complicated than the code under test.
And, above all, make a sincere effort to expand your testing skills as part of your professional development. Don't concentrate on programming techniques at the expense of testing.
Resources Learn
-
Read Aim, Fire, an introduction to test-driven development by Kent Beck.
-
See what the software mavens have to say about the Single Responsibility Principle (also known as the One Responsibility Rule).
-
Read more about the love of testing in JUnit Test Infected: Programmers Love Writing Tests.
-
Joe Schmetzer (whom I believe coined the phrase JUnit antipatterns) has more on this topic in his JUnit Antipatterns article.
-
The C2 Wiki also has plenty of information about antipatterns.
-
You can find more information about JUnit in the articles section of the JUnit Web site.
-
J.B. Rainsberger stands out in my mind as the person who best groks testing with JUnit. His book JUnit Recipes has been invaluable to me. He also participates in the Yahoo! JUnit group and moderates the Yahoo! JUnit Cookbook group. Search the archives for his old posts. They're well worth reading.
-
Joshua Bloch's Effective Java Programming Language Guide should be required reading for any professional Java technology programmer. Chapter 8 deals with exceptions and how to use them well.
-
Robert Binder's Practical Guide to Testing Object-Oriented Software is worth having a look at if you really want to hone your OO testing skills.
-
"Testing, fun? Really?" (developerWorks, March 2001) explores the differences between unit testing and functional testing. It also outlines a process for using them in your daily development.
-
Eric Allen's Diagnosing Java Code developerWorks series focuses on Java technology solutions to keep you on track with your daily programming efforts.
-
Visit the developerWorks Open source zone for extensive how-to information, tools, and project updates to help you develop with open source technologies and use them with IBM's products.
-
Visit the developerWorks Open source zone for extensive how-to information, tools, and project updates to help you develop with open source technologies and use them with IBM's products.
Get products and technologies
-
Innovate your next open source development project with
IBM trial software, available for download or on DVD.
Discuss
About the author | | | After pursuing an undergraduate degree in classics, linguistics, computer science, psychology, and literature, Alex Garrett finally settled on a bachelor's degree in philosophy from the University of Wisconsin. His professional career is as checkered as his academic one. He has been the e-commerce architect for a GE Capital Company, a university lecturer, a systems administrator, an acquisitions editor for a small technical publishing company, and a code toad, among other things. Currently, he's a senior consultant with a Madison-based company and a proud new father. |
Rate this page
| |
