What's it all about, Alfie?

One of the things the traders have complained about in their testing this week has been that the values of the calculations change when we restart the servers. Well… yes, they do. We load up the instrument data, flush in the ticks, and then calc the results. If the prices have changed, and they will in after-hours and pre-hours trading, then the values you see at 8:00 am the ext morning are not the same as the ones you saw at 3:00 pm the night before. So it goes. The prices change, and I was thinking they'd want to see that.

Nope. Count not possibly have been more wrong.

OK, so what's the solution? Well… the easiest thing is just to not calculate the Option values outside of the trading day. The problem with that is: how do we get the last known good data back in after a reload or a restart? Ah… the plot thickens.

We don't want to save all the input values - that's just too much data. So let's just save the results. Thankfully, these are already confined in a simple object in the code, so that's all we have to deal with. Let's tackle the reload first. (In truth, I forgot about the restart at this point, and had to go back and put it in. Ick.)

Every morning, we drop all the instruments we have in memory and load them up from the local replica copy postgres database. It's a wonderfully efficient system we have built. So we load these guys up. If we're careful, and save off these option calculation results before we drop the instruments, and then re-apply them after the new instruments are loaded, then it'll appear that the values are persistent for all instruments that are in the reload.

Clearly, new instruments won't have any values, but they would not have had any the night before, either. Seems to be a reasonable trade-off. Not the ideal I'd like, but it's workable. I personally think it'd be better to look at those new instruments and see if they happen to have a last price in the database, and if so, then use that. But if it's really a new instrument, then we have to wait for ticks - there's just no two ways about it.

This took a good chunk of the afternoon for me to write up, but it tests out fine and should work OK too.

This morning I realized that my auto-flipping of the Ticker Plants had a big hole in it. When I did the flip based on the size of the non-preferred versus preferred side queues, I wasn't properly draining the queue with messages in it. This would be a problem because the next message to arrive would appear later than all the messages in the queue, and I'd effectively drop all them. Not good.

The solution was pretty easy - make a drainPendingMessages() method that I can call before flipping the sides, and use that to drain all the pending messages, and then do the flip. This way, we don't have the gap in the message stream even though we're clearly in trouble on one side of the feed.

Much better solution.

Recently we had a serious problem with one of our exchange feeds. Basically, at 2:00 pm, we just stopped getting the 'A' side of several feeds. Because of the way I'd changed the Ticker Plants, this amounted to a complete halt on ticks. Very bad. But not a bug.

Interestingly enough, the system we have in the Ticker Plants is a consequence of trying to be as accurate as possible. If we have two sides of the same feed - A and B, and they are both supposed to be coming in at about the same time and rate, how do you arbitrate between the two to get the most complete feed and not send down any duplicates? Well… the first idea I had was to look at each side and take the most up-to-date message. Well… that works OK, until one of the feeds gets ahead of the other, or has a skip, and then one feed is showing message 100, and the other is showing 200.

I'd take the 200, and then look for 201 - totally skipping the fact that 100 through 200 is on the other side, just waiting to be used.

The solution was then to look at one side as the "preferred side" and use the other only to "fill in the gaps". This is great as it doesn't skip over message blocks, but the problem is that if your preferred side goes down, you're dead. Even if you have the message stream on the other side, it's going to be ignored as it's only there for filling in the gaps.

As an aside, this brought to a head something that has been a great sore spot for me in the recent reforges at The Shop. That is the need to "check with the manager" to fix something. I estimated that this "auto-flippimg" would take a few days. Not bad, and then it's smart enough to always pick the right side. A far superior product. But it wasn't seen that way. It was seen as something that needed to be scheduled, and planned, and blah, blah, blah. I wanted to scream!

This was my project, and I wanted to take a few days. That should be the end of it. Period. End of story. If I'm too late with too many things, then fire me. But after 35 yrs of doing this, I'm experienced enough to know what's really important, and what is something that's optional and can be scheduled.

Exceptionally frustrating.

So I ended up getting the "OK" and today was the coding.

The solution I came up with was to look at the number of queued messages on the non-preferred side when the preferred-side was empty. If that number was large, say 100,000, then it's pretty clear that the preferred side is in trouble, so let's switch the sides. I needed to do a little more than this to make it clean and as atomic as possible, given that multiple threads are involved in this code, but it wasn't too hard.

In the end, it was a nice change, and I really liked knowing it was going to protect itself from outages in the future.

Now if fixing the management issues were only this easy...

Today has been a lot of little things that came out of the continuing testing cycle with the traders. The greek engine is at the core of quite a few products that are switching over to it as the source of their information, and it's creating quite a mess about it. Lots of people sending in the same issues, not realizing that someone else is reporting the same thing, etc. Then we're purposefully not fixing things in the code they are testing because the management of this testing cycle didn't want to shift the testing underneath them. Truth be told, I'm not a fan of this scheme, but I can't say it's wrong either. The thought is that this is what they are testing, and if the rack up 100 bugs that are all the same thing, then when we fix that, it'll be easy to check those 100 things again and prove they are all working.

I'm more of the "fast waterfall" testing - find a bug, fix it, repeat as quickly as possible. This has the problem that you're always "starting" the testing process. The upside is that you aren't going to find 100 bugs with the same cause - only one.

So I'm trying to make sense of this and keep my head above water. It's hard, because work is becoming oppressive again, and my ability to post is getting beaten out of me. I have to fight the urge to just give up on posting, as it's about the only thing I can always look at and be happy about.

I've had quite a few troubles with Twitterrific and the official Twitter client with the GPU switching from integrated to discrete and back again. In both cases, the apps appear to be working on the GPU that they started on, but if you 'hide' them (both have this feature) and then switch the GPU - for example, go from battery to power adapter, and try to un-hide the apps, they won't function. This has been ongoing for a long time with both apps. I like both, but this one feature means that I have had to stick with one GPU - regardless of the battery/power adapter setting, and leave it there if I want to make sure things are going to work.

Not horrible, but I've been hoping with each update of each of these apps that the bug will get fixed. I've sent bug report after bug report to both teams, and they either can't reproduce something that's superlatively easy for me to reproduce, or they aren't working on it. Sad.

So when Daniel J. tweeted that he was trying Hibari as a new Twitter client, I gave it a look. The big difference here is that Hibari is being actively developed, and that's something that does not seem to be the case for either of the other two I really liked. First, I was glad to see the style. It's clean, nice, solid, and shows thumbnails of the inline images. Very nice so far.

Then I realized it can hide/unhide nicely. Even better. Then I did the GPU-switch test and it passed with flying colors. OK, time to pay my $10, it's passed everything I wanted in a Twitter client. And it's still being developed!

I then saw that Gus M. of JSTalk fame was using JSTalk to tweet in Hibari. Even better. Auto-updates, I'm hoping are with Sparkle, but we'll have to see, and I'm smiling. Not bad at all.

I'm going to give it a while. See how it goes. I'm glad I can now let my GPUs switch and take advantage of the better GPU when I'm plugged into the wall, but fall back to the internal GPU when it's on battery, and I don't have to worry about my Twitter client.

Oh… and it's got great Growl integration. Can't beat that!

Today, in addition to dealing with QA questions from a lot of the groups trying to test their apps against my new engine, I had to add in the last trade on each exchange to the system so that the clients could see that - in addition to the per exchange quotes and volume. I don't know why the guy who used to work on this didn't tell me this was needed - but he didn't, and that's just the way things go. Nothing to do today, but fit it in.

The problem is that the change really makes the serialized output format different, and that means that we need to have the Java and C# clients updated as well. I did the C++ client when I put in the change, and tested it out as a part of my codebase. But the other clients are in my codebase, and so I just have to send out the change, and hope they see the note and pick it up as soon as possible.

I'm sure there will be prodding on the part of the users, and it'll make them a little grumpy, but since I didn't know about it until today, I can't really have done anything about it until today. So it goes.

It was a bit of a hassle because of the way they wanted me to implement it - first in a different branch, then fold it into mine so I could really test it. Not horrible, but annoying. So it goes.

I'm saying a lot of that recently… Not a really great sign in my book...

Today I got word that I had new boxes at out second datacenter - call it disaster/recovery, but it runs 100% all the time, so it's really just a secondary site that is fully used. There's a lot to get going on these boxes - postgres, The Broker, the user accounts, the right SSH keys, a lot of little things before I even really get to worry about the software and making it run on the second site.

Thankfully, there is no rush. I'll get it done when the sys admins get their stuff done, and then I'll fire up postgres and The Broker, and make the changes to the code so it deploys and runs as it should.

Throw in a few crontab entries and we'll be good to go. But it'll probably be tomorrow before that's all done. Just takes time.

I've been reading a bit about Apple's first quarter earnings call today, and I am just stunned. The numbers and the graphs are all impressive, but it's the real scale that's blowing me away. Here's this (supposedly) niche company that had profits in the first quarter greater than Google's revenue. Amazing. Really.

And in my book, it could not happen to a better company. iBooks Author. The iBooks Textbooks. The hardware, and the tools. Simply amazing. Catering to the creative person in all of us has certainly made them incredibly wealthy. And the stories of what they did for their people, and the citizens of Japan after the earthquake - that's a company that's not just about making a buck.

It's good to see good karma generating good outcomes.

This morning I again have been battling the latest in the list of "not quite there - yet" issues with my greek engine. This morning's list of issues are really only one - the exchanges, in this case via the CTS feed, are sending start of day Summary messages with a 'close' value that I'm taking as the close for the day.

Wrong.

What I'd really like to do is to put all these "not really used" Summary messages from the exchanges into a group via flags, so that I can still send them, still see them, but the pricing part of the system would not use them. As it is, the legacy system I'm replacing just ignores them, and that's a little too dangerous for my taste. What if someone wanted to really see them? Too bad. They just aren't there.

That's one of the many dangers of legacy systems - you end up crippling your new code just to get it to match the old. Yet if the old was all that fantastic, why replace it? Seems like you'd want to accept some of these changes as they are really making the product better.

But that's not what I did. I removed the decoding of the message. I'm going to go back in someday and use some flags and then filter in the pricing component. But for now, it's safer to just go along than to try and change people's minds. They're just too stuck in their ways.

This morning, the Google Chrome team released 18.0.1017.2 and the release notes indicate it's got several nice things like the latest V8 javascript engine (3.8.7.1) as well as several nice-to-haves like popup window fixes and flashing window fixes on the Mac. Nice to see the improvement in the V8 engine, as that's where so much of the work is going these days - heavy javascript on the client for thin client deployments.

Hope they keep up the good work.