When Unit Tests Do More Harm than Good
Today I've been up to my neck (again) in the code I inherited that I seem to be bound to with rope and Crazy Glue. This time, I was trying to add to the Flex client a little data that was in at least one of the XML data feeds. Turns out, it wasn't in all three of the necessary XML data feeds, and therein lies my sorry tale of woe today.
The original author believed that, when it came to JUnit tests, if a few are good, a lot are really super, and dozens are really far out. While that might be true, the problem today turned out that the data driving the tests wasn't contained 100% within the JUnit tests, but also took data from the database, etc. and when things changed, as they have in the last few weeks, you run into a lot of broken tests, with no real way to easily fix them.
There are a few things that unit tests - even automated unit tests, are really good for. But they are all basically small, utility class functionality that isn't dependent on date, or time of day, or anything not controlled by the test. When you have things like database configurations, or expiring instruments, then you need to be very careful, because when you come back to the tests after a prolonged period of inactivity, you can get into a really bad place... and that's where I found myself today.
There were errors all over the place, and the one line of code change could not have accounted for the failures. There was something else, but what? The problem with JUnit tests is that you typically only get an error and an exception stack. It's up to you to dig in there to see what the problem is. Sure, the unit tests I've made have a lot of sub-tests in them so that it's pretty darn easy to see what caused the problem, but that's not required. Unfortunately, these tests I was working with were very complex, and no real intermediate results, so it took me about an hour to figure out the problem: database changes.
This is where the unit test concept really fails. It took a simple one-line change and drug it out to more than an hour of work. This is not what the proponents of unit testing put forth. In theory, since I just added one thing, they would say I'd only see a different where that caused an output difference. But that's ideal, and this is the real world.
So I got the changes in, and the tests corrected for the new data in the database, and things are OK for now. But in a few months, it might happen again. A 10 min change drags out into an hour-long test-fixing enterprise. Gack!