Finally Coming Around to the Simple Solution
This week marks the time when we have to make final preparations for going off Daylight Savings Time (DST) which is important because Hong Kong does not observe DST, and so we're going to loose an hour at the end of our day and the start of theirs. This is important for trading and risk apps like mine. The original solution was to make my app run 24x5 without any interruptions. This was a significant technical challenge and after about a day or two of this, I decided that it'd be better to make use of extra hardware to have two servers - one that covered the US and Europe, and another that covered the Far East.
Problem was, this meant that there was going to be a lot more maintenance of the system to keep two systems up and in sync. After hearing me say this, most said "Yeah, yeah" nodding their heads, but thought I really didn't understand the situation. When I came back with a list of things that would have to be done each evening to make this work, all of a sudden the easier solution to the problem appeared.
The problem really comes down to the fact that we typically didn't start the nightly processing until about 5:30 pm, Chicago time. This was 6:30 am Hong Kong time, and after the DST change, it'd be 7:30 am Hong Kong time. Given that the start is then, it's likely that we are not up and going by 8:00 am local time in Hong Kong, and that's the problem. So I had suggested that we move up the end-of-day processing, but that got nowhere. Then they saw what they'd have to do to support the two servers capable of giving Hong Kong the data they needed when they needed it.
All of a sudden, the end-of-day processing was moved up to 4:15 pm Chicago time. Plenty of time before Hong Kong would need it.
While I'm a little frustrated by the fact that I wasn't listened to earlier, I have to take solace in the fact that eventually my ideas found ears, and things started to move in the right direction. Now the second hardware is for disaster/recovery and there's only one server covering the globe. It's simpler, and while there are bound to be bumps in the road, things are going to be better by moving up the end-of-day processing.
That first bump was last night when I got a call on the train. It was from someone asking if there was a way to edit the data that's sitting in the middle-tier of the application. I said "Nope, it flows from the server, and that's already thinking it's tomorrow." I knew what they wanted to hear, but it wasn't going to happen. The data in the middle-tier is too complex to simply edit with something like a text editor. The values are interrelated and computationally intensive calculations need to be done in order to derive some values from others. It's not easy. Which is why there's no easy editor of that data. But because of the early restart, the guys didn't check on the state of the system early enough in the afternoon to make sure that it would be good to go for the earlier restart.
Lessons learned. Like I said... a few bumps in the road, but it's a better place we're going to.