What's it all about, Alfie?

Today I have been working on a nasty little problem that I believe is one of my own making. Maybe that's a little harsh… but certainly it's one I have allowed to exist. The problem is that the calculations in the greek engine aren't exactly self-consistent. For instance, if we have a simple AAPL option:

so that if we run the following prices through the engine, we should get some reasonable matches:

or reverse the engine to give you prices for different implied vols:

OK… these are the results after I fixed the code. Before, the initial calculations were looking OK, but when I tried to calculate the implodes for the prices, we'd match on the bid-side, but not the ask-side. And when we tried to get prices for implied vols, the same would happen - we'd match on one side, and miss on the other. Very odd, and not good. After all, this is one data set. One engine. One FE Model.

So I started digging into the code to see what it was that might possibly be causing these differences. The first thing I noticed was that the logic and code flow of the engine wasn't as clean as it could have been. There were entirely different code paths for the different types of calculations. This wasn't necessary because the calculation type was a bit-flag that was passed into the one main calculator object's calculate() method.

This screamed out for a clean-up.

To be fair, I hadn't put it this way, but I had known this was the case, and hadn't cleaned anything up to this point. After all, it all seemed to go back to the same calculate() method - how different could it be? Well… it turns out that it could be very different - depending on the conditions.

So I started collapsing the code. One at a time, I'd take out a distinct code path, replace the hard-coded values with variables, and use the generic code path. I had to add a few variables, sure, but it wasn't too bad. After the first couple, the rest just fell into place. In the end, it was about 100+ less lines of code, and it only took about 30 mins to do.

The results were dramatic. As seen above (boy! I wish I'd kept the bad data!), the numbers match on the different forms of the calculation. This was a big win for me as it had been bothering the testing for a few days. I'm glad to be done with this one.

This morning I wasn't at all surprised to see that Google Chrome dev 19.0.1036.7 was released, as I'd noted a few days ago that the beta channel had a more recent version than dev, and that's just not how they typically release things. So they've upped the major version number and included the latest V8 javascript engine (3.9.4.0) and a few UI issues. It's interesting to see that there are more than a few Windows-only fixes, but that's nice to see too - Mac OS X was a little more stable - or is still buggy, hard to say.

Great to see the improvements keep coming!

Today was a lot of little fixes to the Greek Engine - better formatting of the timestamps on some log messages, more logging on reconnection attempts, stuff like that. Things that don't really effect the overall performance, but it makes debugging a lot better, and I'm in the midst of trying to finish up this new big feature, and it's proving to be a little tougher than I'd have hoped. Still, I know I'll get it - eventually, but in the meantime, these little things are going to make figuring out what's wrong a lot easier.

Every little bit helps.

Today has been another hard day… specifically, for me in the realm of emotional turmoil and personal happiness. Yesterday, I got my yearly bonus number, and it was less than 50% of what I had expected. This expectation was based on deals and promises and last year's numbers, and frankly, I was more than a little disappointed. Yet, after I heard the number, I was calm, and collected on the outside, and asked if that was it.

My manager then wanted to know how I felt.

Hmmm… first, I think this is a very silly move on his part. He knows what he said, he knows it's less than 50%, and he expects anything but anger from me? Foolish man. But he wants to know. So I tell him Hey, it's not important to the discussion, is it? Just to try and say "You don't really want to know - you're just trying to give me a chance to vent."

But he pressed on, and said that he cares about how I feel.

Not true, I think. Not really. If he really cared, he'd have made the number bigger, and if it wasn't going to be bigger, he wouldn't try to make it out to seem like anything other than a kick in the pants. You hired me because I was smart… why are you now treating me like I'm an idiot? I know what's going on here. I've owned a business. It's not about good/bad. It's about compensation. Period.

So we had a talk, and the more we got into it the less honest they seemed to be with me. Maybe it was because I was asking a lot of uncomfortable questions of them - like if the compensation isn't important, or I can do nothing to impact it, then what's my motivation for working hard? Why not just put in a decent day's work and be done? Why kill myself?

They had no answer, because there is no defensible answer to that. In this business, they pay you for killing yourself. I've heard them say it time and again, place after place. That's why the bonus is so much of your yearly compensation. They want to see you earn it every single day. I get it. No problem. But then when you come through, they better pay you, or someone else will.

Which brings me to my point… If I'm getting no personal gratification from this job - or let's say I'm not getting any additional personal gratification from this job for working the extra hours, then why do it? It's a simple problem. I'm sacrificing my home life for this job, in the thought that I'd get paid for it and make my home life better. But if that implicit contract is broken, then what's the point? I might as well live on less, and be happier because I'm not killing myself.

Better yet - get a different job and work for someone that honors that contract.

It's a classic blind spot… companies spend gobs of money finding the "right people", and then some will cheap out on the bonus. This makes the person leave, and the replacement cost is far far greater than the difference in the bonus. It's simple business sense. Dollars and sense. If you pay a good person well, they will stay. Pay them poorly, and they leave, and you have to spend even more money to get the next good one.

Don't forget that the bad reputation you're building with employees as a cheapskate firm will make it even harder to get good people in the door. It's just bad business.

If, as an employer, you have set expectations, then you better meet them, or get pretty close. And less than 50% of expectations isn't even remotely close.

This morning I saw that Google Chrome dev 18.0.125.7 was out, and the release notes say there are a few bug fixes with regards to the settings pane. Good enough. I like reading in the comments the praise for the speed of fixing these issues. It shows me that the folks finding the issues are really happy with the team fixing them in a reasonable time frame. Says a lot, because if I had a problem, I'd like to know it was being addressed. Just nice to see.

[2/11] UPDATE: Interestingly, this morning I saw that the beat channel was updated to 18.0.1025.25 - a jump over the dev channel. This makes no sense as the dev channel is supposed to the be more often updated and less stable of the channels. So something much be going on… like maybe the dev channel is going to 19.x.x.x, or something. In any case, it's an odd move that certainly means something… I'm just not sure what it is.

I've been trying to deal with a few dodgey disk array for a few weeks. This was a consequence of the recent floods in Thailand and we were unable to get the high-capacity drives to make the 2TB array for the server, so they pressed an old email server's drive array into use, and it's been a bit dodgey to say the least.

To be fair, I'm glad we had the old array to press into service. If I had been forced to wait for the estimated 3 months, that would certainly have been worse. But I still have to say that bad device drivers are a pain, and I would really like them fixed.

So here's what's been happening… I come in in the morning and I see the mount point for this drive array is there in the filesystem, but all the jobs referencing it are failing. Wonderful. So I try to take a look at it:

  $ cd /plogs/Engine/dumps
  $ ls
  ls: cannot open directory .: Input/output error

No amount of un-mounting and re-mounting will work as the OS simply cannot see the drive array. We have to reboot the box and then it comes back online.

The problem with this approach is that I've got a ton of exchange feed recorders running on this box, and it's the only backup we have to production. If we miss recording one of these feeds, then it's gone as the exchanges aren't in the business of replaying their entire day just because we had a hardware problem.

So I'm trying to get a few things done - the first is get a real backup to the recorders in a second datacenter. The second is getting this drive array working properly on Ubuntu 10, hopefully with a kernel update that's in the offing. It is a decent array. I like it. But it's got to work first, and then I'll be happy.

This afternoon I've put the final touches on the sync start to the greek engine. Basically, when we restart the greek engine, it's possible that we are going to miss messages from the exchange because we're down/restarting. This option allow the app to recognize that it might have missed messages, and hit the archive server and ask it for any possible messages for that time frame. If there are some, we'll work them into the message stream.

It's a very nice feature to have as it means that a mid-day crash or reboot is not going to loose anything. But it's a huge load on the archive server, and it's really not been hit all that hard, so the testing is going to be a really big part of this. Fair enough, it's going to take some time to work out the kinks, but at least now we have what we think we need, and it's up to testing to either confirm or deny those beliefs.

It'll be nice to get this tested and into the main codebase. It's been a ton of work to get all the pieces working and working well.

I spent the morning today exposing another Broker service for my greeks engine - this one for stats on the running process. In the last few days, the operations folks, who have had months to decide what support tools they need, have put a halt on the deployment to production of my greek engine because they now need to have these stats for monitoring. Currently, they are running a script on the output of an IRC bot that's hitting the engine, but that parser bot depends on getting data in a specific format, and that's brittle, and doesn't allow us to expand the logging on IRC. So I built the better solution this morning.

It's all based on maps of maps, and I just put the data in what I felt made sense. It's organized by feeds and then the general engine, and within feeds, there are the stock feeds and the option feeds, and so on until you get all the data as values of leaf nodes in the maps. It's pretty simple, the only real issue was that there were several metrics that they wanted to see that I hadn't put in the code, and the person that had failed to make proper getters for the data, which meant that I had to make those before I could get at the data.

Not bad, but it took time.

The testing went really well, and they should be able to gather the stats they want at their convenience. Not bad.

As a personal aside, it really makes me wonder why it is that this is coming up right now, and why it's a show-stopper? I mean if it's a show-stopper, why wasn't it stated months ago at the beginning of testing? I think the reality is that it's not that critical, but the folks are starting to panic a bit, and are looking for the usual suspects to slow things down, or try to make this new system fit the same mold as the previous one.

It's kinda disappointing.

Today I spent the vast majority of my day today trying to make this one client application of my greek engine a lot faster. I mean a lot faster. Friday afternoon, I was running some tests on this usage pattern, and realized that the client really was seeing some massive delays in getting data from my engine when dealing with very large, very active families. Using SPY as the example, there are some 2500 derivatives on SPY, and calculating their data and returning it to the caller was taking from 1800 to 2200 msec. That's a long time. The problem was magnified because all they wanted was three of the 2500 options, and they had to wait for all 2500.

Not good.

So Friday I jotted down a few ideas to try today and spent the first few hours doing just that. Each one was a little better, but I was still looking at 1300 msec, and that's just too long. I needed to chop out an order of magnitude or two. So I started doing the profiling. What was it that was taking so long?

Well… it's the calculations. That's no surprise, but it's a real bottleneck too. We can't really afford to make the calculations tie up multiple threads. That'd kill the box with some 50 clients each needing multiple threads for their calks. Not good. I tried to look at other things, but in the end, it always came back to the calculations.

Along the way, however, I did come up with a few really fun optimizations. I was able to look at a continually updating profile of the instrument and use those values to 'seed' the request, but the updates from the market were just so frequent, it was impossible to stay ahead of the updates. It was a real problem.

So I did what I should have done first - go and talk to the coders writing the client app.

I found out that all they really wanted were the implied vols and they only wanted two or three options in each call. Well… now that's very interesting. That's a use-case that I hadn't expected. The reason it's very interesting is that the implied vols can be calculated independently of each other, which means that by telling me you're interested in only the implied vol calculations, I can look at the three options you're asking for, and calculate just them. Sweet.

I had to work into the API the idea of the type of calculation, but we had something pretty much like that already in the API - it just needed a simple extension. And then I had to get the different type handled in the code. In the end, it wasn't too bad, and the time savings were amazing!

The 1800 msec went to 20 msec. That's something that's more than fast enough for what we need. All because I listened to what the client specifically needed. Simple way to be faster? Just do less.

Excellent.

This morning I thought that git on my MacBook Pro might be a little behind the times. I don't honestly think there's a huge difference from 1.7.4.1 to 1.7.8.4, but you never know, and it's simple using the Mac OS X installer. Just download it, double-click, and it's ready to go.

It's nice to see:

  $ git --version
  git version 1.7.8.4

Nice. Love it when things "just work".

Interesting… I just noticed that Mac OS X 10.7.3 comes with git - and it makes perfect sense that it does. Xcode uses git now, and so it'd require that the OS - at least the developer tools, would have to have it. So it's not necessary for me to worry about updating this any more. It's nice to have a secondary source, should Apple decide to drop it's support, but I'm guessing that's not going to happen anytime soon.

Interesting stuff…