A Hallmark of a Good System – Clean Mid-Day Restarts
Well... when I left last evening, I left my system in a less-than-perfect state: if you restarted the web app in the middle of the day, it's possible that you might not get correct firm totals for P/L, etc. Why? Because it was based on a table trigger, and if you didn't get new rows, they weren't being counted. That's not going to work when some data sources send data only during fixed times of the day. Crud.
And while it's true that I don't restart production mid-day very often, it's still not the sign of a really good app to restart poorly in the middle of the day. The problem was, properly restarting mid-day was a real problem.
First, the firm totals needed to be stored on each alert - thankfully, they already were. Why? Because the alert could be an 11-point moving average filter or a 31-point moving average. In these cases, the firm totals will be different because of the different smoothing employed on the raw data. So it's got to be "local" to the alert.
Second, the firm totals needed to be updated for all incoming data. Initially, I had first checked to see if the portfolio was one of the ones I was interested in, but that was a mistake. I needed to update the firm totals and then filter on the appropriateness of the portfolio. That was a simple fix, so no big deal.
Finally, it's how the data needed to be fed into the alerts in order to get them primed for action on the restart. If there are n alerts, then we need to push n copies of each portfolio. Doing this at the alert level means there's a lot of hits to the database, and while that's logically reasonable, it's not a good plan as the number of alerts grows. So we need a different plan.
This morning I came up with a plan that seems to be working quite well.
In the alert controller code, just after creating all the alerts, I'm going to look at all the portfolios that have sent in data so far today. I'll then look at all the alerts and ask them how many data points they need, add one to that, and know that this is the number of "recent rows" in the data table I need for each portfolio.
I'll then run through all the portfolios, get the maximum number of rows I'll need from the table, and then feed those into each of the alerts, one at a time. Of course, I'll limit the data I feed any one alert to be that which is needs, but there will be at least one that will need all the data I've obtained, and that's not bad. Then, I'll be able to use this to calculate the firm totals for each alert.
What's interesting is that this process is really quite fast. One database connection, and to the H2 in-memory database at that. Most of the pulls there are less than 150 msec. in duration (yeah, I timed them in the code), and then pushing the data to the alerts is really fast as it's all just a bunch of data structures.
In the end, I have firm totals that survive a mid-day restart quite nicely. I'm more than a little pleased that all this work took no more than three hours. I was expecting quite a bit more. Nice surprise.