Sometimes You Have to Debug Another System’s Code
Today, actually yesterday evening, a trader had a problem with some of the data my app was showing. Now I don't generate this data - I just display it, but as is often the case, if it's connected with your app, you get at least partial blame for it not being right. I understand this and it's something that, to a large degree, I agree with. The problem comes in where the guys feeding me the data won't (or can't) fix the data coming to me in a timely manner. Then I have to get into two modes: Defensive Systems Development and someone else's debugger.
Today was one of those days.
After coming up with a plan to avoid the problem in the short-term, I set out on the task of making my system capable of filtering out the bad data from the upstream source. It wasn't something I was planning on doing today, but it wasn't bad and allowed me to add another level of flexibility to my flagship middle-tier. Not bad really. But in the middle of trying to get al these changes in for stopping and starting the flow of updates from my main collector/distributor, there were a flurry of emails about the upstream system - a database, and trying to find the problem there so that we stop it at the source.
I'm all for fixing things at the source, but I soon realized that they weren't seeing the problem. So I had to just into the fray and try and figure it out myself. I'll admit it was a tricky little problem in a stored procedure. Basically, a table which originally presumed to have only one day's worth of data in it was having, for a few hours, two days worth of data. This wasn't bad because when they changed to making it have multiple days of data, they changed the stored procedures to look at just one day's data in that table. Or so we thought.
In actuality, it was not looking at just one day's data because the where clause on the defining cursor was not limiting the primary table's data to just one day. This meant that for a time, two day's data was being used in the cursor. This would have been fine, if it not for another bug in the storied procedure that didn't limit an update statement to a single day. Now the bug is evident - two days were being looked at in the cursor, both days were being updated at the same time in the update statement, and one of those days was not really complete.
That meant that the incomplete day was getting used and updating the complete day and that was the bug. Once I figured that out, I told the manager of the project, and when I had convinced him, told him what he had to do to fix it. He then had one of his guys fix the stored procedure and we should be fine.
I still finished my enhancements to the filtering on the data sources because I think it's a good thing to have. But it's probably no longer necessary. We have it fixed at the source and that's going to be all we need.