What's it all about, Alfie?

This morning a user wanted to be able to see if a Broker was alive and listening on the machine and port it was supposed to be on. This came up because Unix Admin decided to do a kernel update on all the staging machines last night, and we didn't have everything set to auto-restart. Therefore, we had no processes for people to look to. Not good. Thankfully, we have backups, but how does a user know when to hit the backup? After a failed call, sure, but with retries built into the code, that can take upwards of 10 sec. What about something a little faster?

Seems like a reasonable request, and to this morning I added a little isAlive() method to the main Broker client. It's very simple - just tries to connect to the specific host and port that it's supposed to use, and if something is there listening, it returns 'true', otherwise, it returns 'false'. Really easy.

Boost ASIO makes it a little un-easy, but still, it's not too bad:

  bool MMDClient::isAlive()
  {
    bool       success = false;
 
    // only try if they have set the URL to something useful...
    if (!mHostname.empty() && (mPort > 0)) {
      using namespace boost::system;
      using namespace boost::asio;
      using namespace boost::asio::ip;
 
      // getting the connection in boost is a painful process…
      tcp::resolver   resolver(mIOService);
      std::ostringstream  port;
      port << mPort;
      error_code              err = error::host_not_found;
      tcp::resolver::query    query(mHostname, port.str());
      tcp::resolver::iterator it = resolver.resolve(query, err);
      tcp::socket             sock(mIOService);
      if (err != error::host_not_found) {
        err = error::host_not_found;
        tcp::resolver::iterator   end;
        while (err && (it != end)) {
          sock.close();
          sock.connect(*it++, err);
        }
      }
      // if we got a connection, then something is there…
      success = !err;
      // …and close the socket regardless of anything else
      sock.close();
    }
 
    return success;
  }

While using boost isn't trivial, I've found that the pros outweigh the cons, and it's better to have something like this that can handle multi-DNS entries and find what you're looking for, than to have to implement it all yourself.

This guy tested out and works great. Another happy customer!

One of the things that has come up in recent days is the quality of the end-of-day marks that the greek engine is using. The code tries pretty hard to get the right number, and it does a good job on getting the 'close' right, but it's another thing when it comes to calculating the last value of the day. After all, the greeks are meant to calculate on the 'close' - just the 'last', and that is the real problem.

The option pricer in the engine looks for the 'spot' - the price of the underlying that is used in the calculations, and it has a rather complex logical path based on the instrument type of the underlying, and what data is defined for it. Not hard to see, but sufficiently complex. In order to make it so that we include the 'close' in this logic might make it a lot worse before it gets better.

For example, we have the logic that an option will only be updated if it's a valid trading day (no need for weekends, holidays), and if it's during market hours for that instrument. The wrinkle here is that "during market hours" is updated only every couple of seconds at best, and what we really need is to look at the exchange timestamp on the message and see if it's something to include in the calculation set.

But we don't have that data in the right place in the code. It wasn't really designed for this, and there's where I blame myself. I allowed a junior guy who had been working with the legacy pricing server to lead the charge on this part of the code. As a consequence, it's a mess. It really is. We have objects with really questionable APIs, very bad separation of duties, etc. It's a mess.

But I have to live with it at this point, because so much is done, and the time required to fix it would not be allowed. That's a real shame. I'd like to fix it, and I'm not sure it's all that much time, but it's more than will be allowed, I'm sure. Even if it too no time, they'd oppose it on risk and rewards grounds.

So I have to figure out how to make this work with the tools at hand. Not as fun a job, but it's the job for today.

[12:35] UPDATE: well… I've got two little sections of code to work with. The first I've just put in is the check to make sure that - during market hours - the print must be a "qualified trade" from the exchange to update the 'last' price on the Instrument. The old code:

  if (!isnan(aPrint.getPriceAsInt()) &&
      ((mPrint.last.price != aPrint.getPriceAsInt()) ||
       (mPrint.last.size != aPrint.getSize()) ||
       (mPrint.last.exchange != aPrint.getExchange()))) {
    // update all the components of the print
    mPrint.last.price = aPrint.getPriceAsInt();
    mPrint.last.size = aPrint.getSize();
    mPrint.last.exchange = aPrint.getExchange();
    // ...and don't forget the timestamp
    mPrintTimestamp = now;
  }

became:

  bool  isMH = isMarketHours(aPrint.getCreationTimestamp());
  bool  isLPC = aPrint.isLastPriceChanged();
  if (!isnan(aPrint.getPriceAsInt()) && ((isMH && isLPC) || !isMH) &&
      ((mPrint.last.price != aPrint.getPriceAsInt()) ||
       (mPrint.last.size != aPrint.getSize()) ||
       (mPrint.last.exchange != aPrint.getExchange()))) {
    // update all the components of the print
    mPrint.last.price = aPrint.getPriceAsInt();
    mPrint.last.size = aPrint.getSize();
    mPrint.last.exchange = aPrint.getExchange();
    // ...and don't forget the timestamp
    mPrintTimestamp = now;
  }

The addition should work just as needed - if it's during the market hours, it needs to have the last price changed flag set. If not, take it anyway. Nice to have it be so surgical.

Most of today has been spent working on little updates to the greek engine based on the results of user testing. Some of it has just been on explaining what the engine does and how to get it to do what you want it to do. This last part was really focused around one of the clients of the engine that's a neat little web page that's communicating to the engine directly through The Broker. This is really kind of neat in that the web clients now don't need a back end server to maintain their state - they maintain their own state, and make calls for data as they need it. Very much the spirit of AJAX, but not at all using XML or anything like that.

Anyway, the issue was how to get the right implied vols out of the model in the engine based on what needed to be passed in. It's not all that hard, but it's detail work, and it really helps to know the business domain reasonably well. The guy doing the web coding isn't really strong in that last category, but he'd trying. We had several meetings about what to do, and he wasn't really the best candidate to drive these meetings, but because it was his code the traders see, it was assumed that he was the guy.

So we had to clear all that up in several meetings.

Other than that, I've still been wrestling with the serialization of the data from the exchange feed recorders to the archive server. For some unknown reason, we are seemingly getting scrambled data, and it's amazing that it's just a problematic issue. Most of the serializations I've done are rock solid. It's just this one.

The size isn't horrendous. The contents are bad. It's a simple std::string containing binary data. Should be a slam dunk. But it's not so simple 100% of the time. And that's the issue. It's such a difficult problem to reproduce.

Late today I thought I'd try shifting from serializing a std::string to serializing a msg::ng::bytes_t array. This is a simple byte array that is implemented as a std::vector<uint8_t> - I just typedef it to make it a little easier to use.

What I'm hoping is that this makes the binary data a little more stable in the serialization and deserialization. We'll have to see tomorrow - I don't want to restart the feed recorders now to put in this new data.

Today I have been working on a nasty little problem that I believe is one of my own making. Maybe that's a little harsh… but certainly it's one I have allowed to exist. The problem is that the calculations in the greek engine aren't exactly self-consistent. For instance, if we have a simple AAPL option:

so that if we run the following prices through the engine, we should get some reasonable matches:

or reverse the engine to give you prices for different implied vols:

OK… these are the results after I fixed the code. Before, the initial calculations were looking OK, but when I tried to calculate the implodes for the prices, we'd match on the bid-side, but not the ask-side. And when we tried to get prices for implied vols, the same would happen - we'd match on one side, and miss on the other. Very odd, and not good. After all, this is one data set. One engine. One FE Model.

So I started digging into the code to see what it was that might possibly be causing these differences. The first thing I noticed was that the logic and code flow of the engine wasn't as clean as it could have been. There were entirely different code paths for the different types of calculations. This wasn't necessary because the calculation type was a bit-flag that was passed into the one main calculator object's calculate() method.

This screamed out for a clean-up.

To be fair, I hadn't put it this way, but I had known this was the case, and hadn't cleaned anything up to this point. After all, it all seemed to go back to the same calculate() method - how different could it be? Well… it turns out that it could be very different - depending on the conditions.

So I started collapsing the code. One at a time, I'd take out a distinct code path, replace the hard-coded values with variables, and use the generic code path. I had to add a few variables, sure, but it wasn't too bad. After the first couple, the rest just fell into place. In the end, it was about 100+ less lines of code, and it only took about 30 mins to do.

The results were dramatic. As seen above (boy! I wish I'd kept the bad data!), the numbers match on the different forms of the calculation. This was a big win for me as it had been bothering the testing for a few days. I'm glad to be done with this one.

Today was a lot of little fixes to the Greek Engine - better formatting of the timestamps on some log messages, more logging on reconnection attempts, stuff like that. Things that don't really effect the overall performance, but it makes debugging a lot better, and I'm in the midst of trying to finish up this new big feature, and it's proving to be a little tougher than I'd have hoped. Still, I know I'll get it - eventually, but in the meantime, these little things are going to make figuring out what's wrong a lot easier.

Every little bit helps.

Today has been another hard day… specifically, for me in the realm of emotional turmoil and personal happiness. Yesterday, I got my yearly bonus number, and it was less than 50% of what I had expected. This expectation was based on deals and promises and last year's numbers, and frankly, I was more than a little disappointed. Yet, after I heard the number, I was calm, and collected on the outside, and asked if that was it.

My manager then wanted to know how I felt.

Hmmm… first, I think this is a very silly move on his part. He knows what he said, he knows it's less than 50%, and he expects anything but anger from me? Foolish man. But he wants to know. So I tell him Hey, it's not important to the discussion, is it? Just to try and say "You don't really want to know - you're just trying to give me a chance to vent."

But he pressed on, and said that he cares about how I feel.

Not true, I think. Not really. If he really cared, he'd have made the number bigger, and if it wasn't going to be bigger, he wouldn't try to make it out to seem like anything other than a kick in the pants. You hired me because I was smart… why are you now treating me like I'm an idiot? I know what's going on here. I've owned a business. It's not about good/bad. It's about compensation. Period.

So we had a talk, and the more we got into it the less honest they seemed to be with me. Maybe it was because I was asking a lot of uncomfortable questions of them - like if the compensation isn't important, or I can do nothing to impact it, then what's my motivation for working hard? Why not just put in a decent day's work and be done? Why kill myself?

They had no answer, because there is no defensible answer to that. In this business, they pay you for killing yourself. I've heard them say it time and again, place after place. That's why the bonus is so much of your yearly compensation. They want to see you earn it every single day. I get it. No problem. But then when you come through, they better pay you, or someone else will.

Which brings me to my point… If I'm getting no personal gratification from this job - or let's say I'm not getting any additional personal gratification from this job for working the extra hours, then why do it? It's a simple problem. I'm sacrificing my home life for this job, in the thought that I'd get paid for it and make my home life better. But if that implicit contract is broken, then what's the point? I might as well live on less, and be happier because I'm not killing myself.

Better yet - get a different job and work for someone that honors that contract.

It's a classic blind spot… companies spend gobs of money finding the "right people", and then some will cheap out on the bonus. This makes the person leave, and the replacement cost is far far greater than the difference in the bonus. It's simple business sense. Dollars and sense. If you pay a good person well, they will stay. Pay them poorly, and they leave, and you have to spend even more money to get the next good one.

Don't forget that the bad reputation you're building with employees as a cheapskate firm will make it even harder to get good people in the door. It's just bad business.

If, as an employer, you have set expectations, then you better meet them, or get pretty close. And less than 50% of expectations isn't even remotely close.

I've been trying to deal with a few dodgey disk array for a few weeks. This was a consequence of the recent floods in Thailand and we were unable to get the high-capacity drives to make the 2TB array for the server, so they pressed an old email server's drive array into use, and it's been a bit dodgey to say the least.

To be fair, I'm glad we had the old array to press into service. If I had been forced to wait for the estimated 3 months, that would certainly have been worse. But I still have to say that bad device drivers are a pain, and I would really like them fixed.

So here's what's been happening… I come in in the morning and I see the mount point for this drive array is there in the filesystem, but all the jobs referencing it are failing. Wonderful. So I try to take a look at it:

  $ cd /plogs/Engine/dumps
  $ ls
  ls: cannot open directory .: Input/output error

No amount of un-mounting and re-mounting will work as the OS simply cannot see the drive array. We have to reboot the box and then it comes back online.

The problem with this approach is that I've got a ton of exchange feed recorders running on this box, and it's the only backup we have to production. If we miss recording one of these feeds, then it's gone as the exchanges aren't in the business of replaying their entire day just because we had a hardware problem.

So I'm trying to get a few things done - the first is get a real backup to the recorders in a second datacenter. The second is getting this drive array working properly on Ubuntu 10, hopefully with a kernel update that's in the offing. It is a decent array. I like it. But it's got to work first, and then I'll be happy.

This afternoon I've put the final touches on the sync start to the greek engine. Basically, when we restart the greek engine, it's possible that we are going to miss messages from the exchange because we're down/restarting. This option allow the app to recognize that it might have missed messages, and hit the archive server and ask it for any possible messages for that time frame. If there are some, we'll work them into the message stream.

It's a very nice feature to have as it means that a mid-day crash or reboot is not going to loose anything. But it's a huge load on the archive server, and it's really not been hit all that hard, so the testing is going to be a really big part of this. Fair enough, it's going to take some time to work out the kinks, but at least now we have what we think we need, and it's up to testing to either confirm or deny those beliefs.

It'll be nice to get this tested and into the main codebase. It's been a ton of work to get all the pieces working and working well.

I spent the morning today exposing another Broker service for my greeks engine - this one for stats on the running process. In the last few days, the operations folks, who have had months to decide what support tools they need, have put a halt on the deployment to production of my greek engine because they now need to have these stats for monitoring. Currently, they are running a script on the output of an IRC bot that's hitting the engine, but that parser bot depends on getting data in a specific format, and that's brittle, and doesn't allow us to expand the logging on IRC. So I built the better solution this morning.

It's all based on maps of maps, and I just put the data in what I felt made sense. It's organized by feeds and then the general engine, and within feeds, there are the stock feeds and the option feeds, and so on until you get all the data as values of leaf nodes in the maps. It's pretty simple, the only real issue was that there were several metrics that they wanted to see that I hadn't put in the code, and the person that had failed to make proper getters for the data, which meant that I had to make those before I could get at the data.

Not bad, but it took time.

The testing went really well, and they should be able to gather the stats they want at their convenience. Not bad.

As a personal aside, it really makes me wonder why it is that this is coming up right now, and why it's a show-stopper? I mean if it's a show-stopper, why wasn't it stated months ago at the beginning of testing? I think the reality is that it's not that critical, but the folks are starting to panic a bit, and are looking for the usual suspects to slow things down, or try to make this new system fit the same mold as the previous one.

It's kinda disappointing.

Today I spent the vast majority of my day today trying to make this one client application of my greek engine a lot faster. I mean a lot faster. Friday afternoon, I was running some tests on this usage pattern, and realized that the client really was seeing some massive delays in getting data from my engine when dealing with very large, very active families. Using SPY as the example, there are some 2500 derivatives on SPY, and calculating their data and returning it to the caller was taking from 1800 to 2200 msec. That's a long time. The problem was magnified because all they wanted was three of the 2500 options, and they had to wait for all 2500.

Not good.

So Friday I jotted down a few ideas to try today and spent the first few hours doing just that. Each one was a little better, but I was still looking at 1300 msec, and that's just too long. I needed to chop out an order of magnitude or two. So I started doing the profiling. What was it that was taking so long?

Well… it's the calculations. That's no surprise, but it's a real bottleneck too. We can't really afford to make the calculations tie up multiple threads. That'd kill the box with some 50 clients each needing multiple threads for their calks. Not good. I tried to look at other things, but in the end, it always came back to the calculations.

Along the way, however, I did come up with a few really fun optimizations. I was able to look at a continually updating profile of the instrument and use those values to 'seed' the request, but the updates from the market were just so frequent, it was impossible to stay ahead of the updates. It was a real problem.

So I did what I should have done first - go and talk to the coders writing the client app.

I found out that all they really wanted were the implied vols and they only wanted two or three options in each call. Well… now that's very interesting. That's a use-case that I hadn't expected. The reason it's very interesting is that the implied vols can be calculated independently of each other, which means that by telling me you're interested in only the implied vol calculations, I can look at the three options you're asking for, and calculate just them. Sweet.

I had to work into the API the idea of the type of calculation, but we had something pretty much like that already in the API - it just needed a simple extension. And then I had to get the different type handled in the code. In the end, it wasn't too bad, and the time savings were amazing!

The 1800 msec went to 20 msec. That's something that's more than fast enough for what we need. All because I listened to what the client specifically needed. Simple way to be faster? Just do less.

Excellent.