What's it all about, Alfie?

One of the things that has come up in recent days is the quality of the end-of-day marks that the greek engine is using. The code tries pretty hard to get the right number, and it does a good job on getting the 'close' right, but it's another thing when it comes to calculating the last value of the day. After all, the greeks are meant to calculate on the 'close' - just the 'last', and that is the real problem.

The option pricer in the engine looks for the 'spot' - the price of the underlying that is used in the calculations, and it has a rather complex logical path based on the instrument type of the underlying, and what data is defined for it. Not hard to see, but sufficiently complex. In order to make it so that we include the 'close' in this logic might make it a lot worse before it gets better.

For example, we have the logic that an option will only be updated if it's a valid trading day (no need for weekends, holidays), and if it's during market hours for that instrument. The wrinkle here is that "during market hours" is updated only every couple of seconds at best, and what we really need is to look at the exchange timestamp on the message and see if it's something to include in the calculation set.

But we don't have that data in the right place in the code. It wasn't really designed for this, and there's where I blame myself. I allowed a junior guy who had been working with the legacy pricing server to lead the charge on this part of the code. As a consequence, it's a mess. It really is. We have objects with really questionable APIs, very bad separation of duties, etc. It's a mess.

But I have to live with it at this point, because so much is done, and the time required to fix it would not be allowed. That's a real shame. I'd like to fix it, and I'm not sure it's all that much time, but it's more than will be allowed, I'm sure. Even if it too no time, they'd oppose it on risk and rewards grounds.

So I have to figure out how to make this work with the tools at hand. Not as fun a job, but it's the job for today.

[12:35] UPDATE: well… I've got two little sections of code to work with. The first I've just put in is the check to make sure that - during market hours - the print must be a "qualified trade" from the exchange to update the 'last' price on the Instrument. The old code:

  if (!isnan(aPrint.getPriceAsInt()) &&
      ((mPrint.last.price != aPrint.getPriceAsInt()) ||
       (mPrint.last.size != aPrint.getSize()) ||
       (mPrint.last.exchange != aPrint.getExchange()))) {
    // update all the components of the print
    mPrint.last.price = aPrint.getPriceAsInt();
    mPrint.last.size = aPrint.getSize();
    mPrint.last.exchange = aPrint.getExchange();
    // ...and don't forget the timestamp
    mPrintTimestamp = now;
  }

became:

  bool  isMH = isMarketHours(aPrint.getCreationTimestamp());
  bool  isLPC = aPrint.isLastPriceChanged();
  if (!isnan(aPrint.getPriceAsInt()) && ((isMH && isLPC) || !isMH) &&
      ((mPrint.last.price != aPrint.getPriceAsInt()) ||
       (mPrint.last.size != aPrint.getSize()) ||
       (mPrint.last.exchange != aPrint.getExchange()))) {
    // update all the components of the print
    mPrint.last.price = aPrint.getPriceAsInt();
    mPrint.last.size = aPrint.getSize();
    mPrint.last.exchange = aPrint.getExchange();
    // ...and don't forget the timestamp
    mPrintTimestamp = now;
  }

The addition should work just as needed - if it's during the market hours, it needs to have the last price changed flag set. If not, take it anyway. Nice to have it be so surgical.

This morning the Google Chrome Team is at it again, and released 19.0.1041.0 which is primarily a bug fixes release. It's impressive to see these updates coming so quickly, as it's only been a few days since the last significant release. They are certainly back at it, and moving things forward. Nice to see.

Most of today has been spent working on little updates to the greek engine based on the results of user testing. Some of it has just been on explaining what the engine does and how to get it to do what you want it to do. This last part was really focused around one of the clients of the engine that's a neat little web page that's communicating to the engine directly through The Broker. This is really kind of neat in that the web clients now don't need a back end server to maintain their state - they maintain their own state, and make calls for data as they need it. Very much the spirit of AJAX, but not at all using XML or anything like that.

Anyway, the issue was how to get the right implied vols out of the model in the engine based on what needed to be passed in. It's not all that hard, but it's detail work, and it really helps to know the business domain reasonably well. The guy doing the web coding isn't really strong in that last category, but he'd trying. We had several meetings about what to do, and he wasn't really the best candidate to drive these meetings, but because it was his code the traders see, it was assumed that he was the guy.

So we had to clear all that up in several meetings.

Other than that, I've still been wrestling with the serialization of the data from the exchange feed recorders to the archive server. For some unknown reason, we are seemingly getting scrambled data, and it's amazing that it's just a problematic issue. Most of the serializations I've done are rock solid. It's just this one.

The size isn't horrendous. The contents are bad. It's a simple std::string containing binary data. Should be a slam dunk. But it's not so simple 100% of the time. And that's the issue. It's such a difficult problem to reproduce.

Late today I thought I'd try shifting from serializing a std::string to serializing a msg::ng::bytes_t array. This is a simple byte array that is implemented as a std::vector<uint8_t> - I just typedef it to make it a little easier to use.

What I'm hoping is that this makes the binary data a little more stable in the serialization and deserialization. We'll have to see tomorrow - I don't want to restart the feed recorders now to put in this new data.

Today I have been working on a nasty little problem that I believe is one of my own making. Maybe that's a little harsh… but certainly it's one I have allowed to exist. The problem is that the calculations in the greek engine aren't exactly self-consistent. For instance, if we have a simple AAPL option:

so that if we run the following prices through the engine, we should get some reasonable matches:

or reverse the engine to give you prices for different implied vols:

OK… these are the results after I fixed the code. Before, the initial calculations were looking OK, but when I tried to calculate the implodes for the prices, we'd match on the bid-side, but not the ask-side. And when we tried to get prices for implied vols, the same would happen - we'd match on one side, and miss on the other. Very odd, and not good. After all, this is one data set. One engine. One FE Model.

So I started digging into the code to see what it was that might possibly be causing these differences. The first thing I noticed was that the logic and code flow of the engine wasn't as clean as it could have been. There were entirely different code paths for the different types of calculations. This wasn't necessary because the calculation type was a bit-flag that was passed into the one main calculator object's calculate() method.

This screamed out for a clean-up.

To be fair, I hadn't put it this way, but I had known this was the case, and hadn't cleaned anything up to this point. After all, it all seemed to go back to the same calculate() method - how different could it be? Well… it turns out that it could be very different - depending on the conditions.

So I started collapsing the code. One at a time, I'd take out a distinct code path, replace the hard-coded values with variables, and use the generic code path. I had to add a few variables, sure, but it wasn't too bad. After the first couple, the rest just fell into place. In the end, it was about 100+ less lines of code, and it only took about 30 mins to do.

The results were dramatic. As seen above (boy! I wish I'd kept the bad data!), the numbers match on the different forms of the calculation. This was a big win for me as it had been bothering the testing for a few days. I'm glad to be done with this one.

This morning I wasn't at all surprised to see that Google Chrome dev 19.0.1036.7 was released, as I'd noted a few days ago that the beta channel had a more recent version than dev, and that's just not how they typically release things. So they've upped the major version number and included the latest V8 javascript engine (3.9.4.0) and a few UI issues. It's interesting to see that there are more than a few Windows-only fixes, but that's nice to see too - Mac OS X was a little more stable - or is still buggy, hard to say.

Great to see the improvements keep coming!

Today was a lot of little fixes to the Greek Engine - better formatting of the timestamps on some log messages, more logging on reconnection attempts, stuff like that. Things that don't really effect the overall performance, but it makes debugging a lot better, and I'm in the midst of trying to finish up this new big feature, and it's proving to be a little tougher than I'd have hoped. Still, I know I'll get it - eventually, but in the meantime, these little things are going to make figuring out what's wrong a lot easier.

Every little bit helps.

This morning I saw that Google Chrome dev 18.0.125.7 was out, and the release notes say there are a few bug fixes with regards to the settings pane. Good enough. I like reading in the comments the praise for the speed of fixing these issues. It shows me that the folks finding the issues are really happy with the team fixing them in a reasonable time frame. Says a lot, because if I had a problem, I'd like to know it was being addressed. Just nice to see.

[2/11] UPDATE: Interestingly, this morning I saw that the beat channel was updated to 18.0.1025.25 - a jump over the dev channel. This makes no sense as the dev channel is supposed to the be more often updated and less stable of the channels. So something much be going on… like maybe the dev channel is going to 19.x.x.x, or something. In any case, it's an odd move that certainly means something… I'm just not sure what it is.

I've been trying to deal with a few dodgey disk array for a few weeks. This was a consequence of the recent floods in Thailand and we were unable to get the high-capacity drives to make the 2TB array for the server, so they pressed an old email server's drive array into use, and it's been a bit dodgey to say the least.

To be fair, I'm glad we had the old array to press into service. If I had been forced to wait for the estimated 3 months, that would certainly have been worse. But I still have to say that bad device drivers are a pain, and I would really like them fixed.

So here's what's been happening… I come in in the morning and I see the mount point for this drive array is there in the filesystem, but all the jobs referencing it are failing. Wonderful. So I try to take a look at it:

  $ cd /plogs/Engine/dumps
  $ ls
  ls: cannot open directory .: Input/output error

No amount of un-mounting and re-mounting will work as the OS simply cannot see the drive array. We have to reboot the box and then it comes back online.

The problem with this approach is that I've got a ton of exchange feed recorders running on this box, and it's the only backup we have to production. If we miss recording one of these feeds, then it's gone as the exchanges aren't in the business of replaying their entire day just because we had a hardware problem.

So I'm trying to get a few things done - the first is get a real backup to the recorders in a second datacenter. The second is getting this drive array working properly on Ubuntu 10, hopefully with a kernel update that's in the offing. It is a decent array. I like it. But it's got to work first, and then I'll be happy.

This afternoon I've put the final touches on the sync start to the greek engine. Basically, when we restart the greek engine, it's possible that we are going to miss messages from the exchange because we're down/restarting. This option allow the app to recognize that it might have missed messages, and hit the archive server and ask it for any possible messages for that time frame. If there are some, we'll work them into the message stream.

It's a very nice feature to have as it means that a mid-day crash or reboot is not going to loose anything. But it's a huge load on the archive server, and it's really not been hit all that hard, so the testing is going to be a really big part of this. Fair enough, it's going to take some time to work out the kinks, but at least now we have what we think we need, and it's up to testing to either confirm or deny those beliefs.

It'll be nice to get this tested and into the main codebase. It's been a ton of work to get all the pieces working and working well.

I spent the morning today exposing another Broker service for my greeks engine - this one for stats on the running process. In the last few days, the operations folks, who have had months to decide what support tools they need, have put a halt on the deployment to production of my greek engine because they now need to have these stats for monitoring. Currently, they are running a script on the output of an IRC bot that's hitting the engine, but that parser bot depends on getting data in a specific format, and that's brittle, and doesn't allow us to expand the logging on IRC. So I built the better solution this morning.

It's all based on maps of maps, and I just put the data in what I felt made sense. It's organized by feeds and then the general engine, and within feeds, there are the stock feeds and the option feeds, and so on until you get all the data as values of leaf nodes in the maps. It's pretty simple, the only real issue was that there were several metrics that they wanted to see that I hadn't put in the code, and the person that had failed to make proper getters for the data, which meant that I had to make those before I could get at the data.

Not bad, but it took time.

The testing went really well, and they should be able to gather the stats they want at their convenience. Not bad.

As a personal aside, it really makes me wonder why it is that this is coming up right now, and why it's a show-stopper? I mean if it's a show-stopper, why wasn't it stated months ago at the beginning of testing? I think the reality is that it's not that critical, but the folks are starting to panic a bit, and are looking for the usual suspects to slow things down, or try to make this new system fit the same mold as the previous one.

It's kinda disappointing.