The Importance of Effective Testing Tools
While I've been dealing with a lot of jUnit testing issues this past week, for the last two days I've been dealing with the realities of trying to match the actual data from the system to the application I've got and that had turned out to be a lot more difficult than I had hoped.
The problem is classic: they have no comparison tools for the final stage in testing. They rely on eyes to look at one and the other over time to see if things "look right". This, as expected, can be very hit-n-miss, and I was getting a lot more of the latter than the former.
It's pretty easy to see if you're not even close, but if you're off by 10%, is that a mistake, or a sampling interval issue? Hard to say. If there's a lot of volatility in the sampled data, then maybe it's as close as you can expect to get. So I find myself wishing for fewer jUnit tests and better test frames for the end result. The fist are nice, but the second are critical.
In the end, the only way I'm going to get better test frame data is to be a part of it's creation. Unfortunately, I may have to do that.