On the False Security of Unit Testing Large Systems
In the last few weeks I've written several times about my dislike of extensive unit testing when it comes to large, complex systems. Unit tests are perfect for building blocks - simple model objects that have well-defined inputs and outputs that can be extensively tested. It's my days in the IC fab industry that makes me wary of testing. The guys at HP had amazingly hard test vectors to test every component as quickly as possible for all possible defects. It wasn't easy.
But I've been working on a system where the unit tests are large, complex, and difficult to maintain -- and don't test enough of the system. But it's assumed that these tests are complete coverage of the system. Well... today it bit me in the behind, and I have solid reasons why I now hate this philosophy.
This morning, I found that the update from last evening wasn't working exactly as it should. In fact, it wasn't working properly for a lot of cases. I was trying to see if is was the data - hard to tell, that's for sure. Maybe it was the install... nope. I was getting pretty desperate. Then someone mentioned checking the Tomcat localhost log. Bingo!
There were a bunch of exceptions that told me exactly what the problem was. Why is this not in the catalina.out log? I'm guessing it's the way this webapp is set up, but all my exceptions should go to catalina.out. Here nor there at this point, I had found the problem.
The problem was that one component was sending a record of six elements, and the other system was expecting seven. Clearly, this was the problem, and when I fixed the sender to add the missing data, it all worked fine. But it was 3 hours of a nightmare because of the impact to production.
All the unit tests would never have picked this up because it was between two components in the same system. More over, there was no testing of this specific feature in my tests, the other developer's tests, even Q/A didn't test this. While I'm not mad at anyone (other than myself), I think this highlights the folly of thinking that unit testing of large, complex systems, is anything more than a waste of time. Small, easily identifiable blocks - sure. But webapps? Please.
So now I've learned something valuable, and when the next person tries to spout the Gospel According to Unit Testing, I'll have a good story to share with them.