Chasing Down Wild Bugs
Last night I got a call from Hong Kong about slow ticks in the server. I talked to the support guy there for about 30 mins to get this out of him, but in the end, I figured out that it was a ticking issue, and further discussions with this guy were going to be pointless and frustrating. So I got online and checked things out.
Turns out there were a few things that weren't right, but nothing pointing to the real problem - which was that the ticks were seemingly working, but not as fast as they should and not as regularly as the ought to. I looked at a few of the logs and one of the instruments had no position on it, but positions on it's options, and so should have been in the 'positioned' poller queue - as opposed to the 'non-positioned' poller queue. I put in a fake position on the instrument and it moved to the right queue, but didn't start ticking like it should have.
I told the support guy over chat to simply hand mark these guys for the night, and I'd get to the issue in the morning and send an email about what I found. I needed to get some sleep or I wouldn't be able to really find the solution to the problem in the morning.
What I found was most interesting. Three problems in two systems caused this issue. The first one was a fix I had put in for the price feeder yesterday for augmenting the trade date/time with the current date/time when no trade date/time was sent by the exchange. The problem with the implementation was that it was also allowing ticks with no trade information to update the trade date/time. This meant that quotes were updating the trade date/time and that's no good. Not at night in Chicago when the ticks are for the next business day in Hong Kong. I was 'stomping' on the trade date/time and move it back a day. Bad move.
This meant that the server was looking at these prices and saying "Hey, this is old data... I need data for 'tomorrow' now." and throwing the ticks away. This explained the hit-n-miss ticks - some came in on the right date, and then the date was overwritten with the quote and no more ticks. Unfortunate, but that meant that I really needed to fix two things - the price feeder and the interpretation of 'today' for ticks.
Fixing the price feeder wasn't bad - I simply made sure that there was some evidence of a trade before defaulting to the current date/time for the trade time. This will make things much more reasonable. At the same time, I checked to see if the current date/time was before the trade date/time existing on the price. If so, then I didn't overwrite it as that would be moving it back in time.
The problem in the server was that I needed to have some knowledge of the region of the instrument. Specifically, I needed to know if it was an Asian instrument. For if it were, the checks on the date of the tick needed to be relaxed by the time difference. So I got a list of all the Asian currencies, made that a configurable option on the server, and then checked each instrument to see if it's currency was an Asian currency. If so, then I relaxed the time check to be after the start of the server so that we would not be in a position to throw away ticks when I knew they were really good - just in the wrong time zone.
The final problem was a simple fix to make sure that when I looked at an instrument to see if it needed to be in the fast tick pool or the slow tick pool, I used the method on the base instrument that checked to see if there were any positions on the instrument or on any of it's children. This fixes up the mistake of improper classification in the first place.
One nasty problem with three important changes. Not a bad morning.