What's it all about, Alfie?

I was talking to a friend today and realized that I'm really suffering a mini-crisis of faith in the work I'm doing. There are plenty of times that I've really loved the work I was doing, but when I stepped back, and took a real look around, I wasn't as thrilled with the environment I was working in. While no place is perfect, I've found myself more than a little disappointed with the environment on issues that are becoming fundamental to my opinions about work and management and responsibilities.

Maybe it's because I'm getting older. It's certainly possible. I also don't claim to be the most patient person in the world. But recently, the effort I've been putting in seems to be very one-sided, and as I heard a friend say a few weeks ago -- relationships need to be give-and-take, not one-sided, to last. And right now, I feel very one-sided on this working relationship.

So rather than think about the situation I'm finding myself in, which I know I can do precious little about at this time, I'm trying to find the passion I had for the work so that I can at least feel good about the work I'm doing. Unfortunately, I'm finding it harder and harder to do.

I'm finding myself thinking more about writing code for my Mac, and even if it's financial, it's mine, and it's doing work on a platform I really like, and the undercurrent is that it's for something that matters to me, not The Shop. But that's not happening these days… I'm working just too much, and there's no time at all for extra coding.

Maybe I just need a vacation. When this project is done, maybe I just need to take some time off and come back when I feel ready to enter this place again. Hard to say… the concern would be that I never wanted to come back, but I guess that's something to think about as well.

For now, it's something I'm working on… I just wish I were having more success.

I was working on my Greek Engine today and ran across a problem that I didn't know the answer to. I had a base class: instrument, and on this base class I had a virtual method: getBestPrice(). This method looked at the last trade price and last quote price for the instrument, and based on that, returned the very best price for the instrument. It's pretty standard stuff.

I also had a derived class: option, based on instrument, and it had an overridden version of the getBestPrice() method. The problem was, I was in a situation where I wanted to always use the instrument version of the method - even when I had an instance of option. How do I do it?

Well… I knew that if I was in a method on option, I could simple say:

  double option::doThis()
  {
    return instrument::doThis();
  }

and it would run the super class' version. But that's not what I had. I had a pointer to an instrument instance - or maybe it was an option instance. Who knows? But the instrument method had to be run.

My co-worker googled this and found that the correct syntax for this is quite simple:

  instrument   *inst = NULL;
  ...
  double  price = inst->instrument::getBestPrice();

and even if inst points to an instance of option, we'll be executing the instrument's version of the method. Sweet!

I'll be checking it on Monday. Pretty nice thing to remember.

I was trying to speed up the conversion of floating point numbers to integers today, and I came across some really interesting data. My initial code was very straight forward: I'd multiply the double by 1000, make it an integer, snprintf() it, and then look at the last character to see if it's a '0' - if it is, I remove it, if not, I keep it. Then I place the decimal point where I know it needs to be.

The code is simple, and not very creative at all:

  char      buff[16];
  bzero(buff, 16);
  snprintf(buff, 15, "%d", (uint32_t)(value * 1000));
  size_t  len = strlen(buff);
  size_t  lm1 = len - 1;
  size_t  lm2 = len - 2;
  size_t  lm3 = len - 3;
  if (buff[len] != '0') {
    buff[len] = buff[lm1];
  }
  buff[lm1] = buff[lm2];
  buff[lm2] = buff[lm3];
  buff[lm3] = '.';

This runs, and while it's not perfect, it's pretty fast. It's certainly faster because I precompute the index values, but that's just smart work. What I found was that using pointers and not doing the references really isn't any faster than using the references.

Then I looked up fast C++ integer to string and saw what a was out there. One guy wrote about this amazingly fast system he had written, so I gave it a look. What was interesting to me was the second place in the speed race was won by an STL string method that built up the number in reverse order by dividing by 10 and then doing a modulo on each digit, and finally reversing it.

So I tried it.

Not even close to the snprintf() version I had.

His results could have been for an older version of the compiler, or something, but I didn't see snprintf() as slow -- in fact, it was faster than anything else I tried. Amazing.

You just have to trust the compiler, and the OS. It'll get you there most of the time.

I have been a big fan of the KyngChaos builds of PostgreSQL for quite a while. After Marc L. stopped posting his, it made sense to find another source, and while I've heard that Apple's Mac OS X Lion Server has PostgreSQL as opposed to MySQL which it had in Snow Leopard, I haven't seen a lot about this part of Lion Server, and I'm consequently a little nervous about getting Server on my laptop and not being happy with the burden it places on my box. If I had a nice Mac Pro, I'd get it - but that's not (yet) the case.

So here's what I noticed in the update from PostgreSQL 8 to 9.0.4, and on Mac OS X Lion. There's a lot of little things, and it made for a somewhat hybrid scheme that I'm going to try and get rid of today with the upgrade to PostgreSQL 9.1.

First off, Mac OS X 10.7 has psql on it already. It's got a complete PostgreSQL client in it. It happens to be 9.0.4, which is nice, as that matched the version I was targeting. Given that, it's only the server I need to install. Not bad, but it's all a package anyway, so no big deal.

Secondly, in 10.7, the PostgreSQL user is _postgres and it's not a user that can login. This means that a lot of the update schemes that I used to use don't work because you can't log in as the postgres user any more. However, there seems to be a better scheme - assuming you're migrating from a recent version. The update procedures are detailed in a file in the disk image you download from the site, but I haven't been able to get it to work. I'll try on the next update again, but this dump-and-load works really well for me and does not require that we have a postgres user to log in as.

Step 1 - we need to get a complete dump of the database. It's pretty easy, just get to anyplace you can write a file - I used my home directory, and issue the following command as yourself:

    /usr/local/pgsql/bin/pg_dumpall -U _postgres -o > pgbackup

Step 2 - stop (unload) the existing version of PostgreSQL. This is going to shut it down, but it's possible to pull it back if you need to:

    sudo launchctl unload \
      /Library/LaunchDaemons/org.postgresql.postgres.plist

Step 3 - install the new PostgreSQL package from the site. I noticed that I had to install it twice in order to get the contents of the /usr/local/pgsql-9.1/data directory, so if you don't get that, then install it again, and it should get installed properly.

Step 4 - start the new version. The launch daemon file should be installed properly and you should be good to go, but it never hurts to check:

    sudo launchctl load \
      /Library/LaunchDaemons/org.postgresql.postgres.plist
    ps -ef | grep post
      …lots of postmaster processes listed...

Step 5 - reload your database. Be in the same place as Step 1 and you can simply issue:

    /usr/local/pgsql/bin/psql -U _postgres -d template1 \
      -f /full/path/to/pgbackup

Check with psql and maybe a simple PHP script, and you should be good to go. Once it's all OK and running, you can remove the old install:

    sudo rm -rf /usr/local/pgsql-9.0

At the same time, if you have the old postgres user still installed, you can delete him as well - there's nothing you need from him any longer. Lion's _postgres user can handle all the chores from here quite nicely.

If you run into problems, check the permissions on the directories. Because I was switching from the postgres user to the _postgres user, I needed to make the old install usable by the new user:

    sudo chown -R _postgres:_postgres /usr/local/pgsql-9.0/data
    sudo chown -R _postgres:admin /usr/local/pgsql-9.0/var

PostgreSQL 9.1 has a ton of interesting features. I'm using the replication at The Shop, and it's working like a charm. I'm a big fan, and these updates are really exciting to see.

Seems the posts were right, Google Chrome dev 16.0.889.0 was released this morning and I picked it up. I had suspected that they'd move a major version number soon, but as I've known for a long time, it's the third number in the version that really matters, and it'll be interesting to see what happens when it tops 1000 - will they move up the next number or not? I'm guessing they're a bunch of engineers, and because of that, they'll have numbers like 20.0.1011.0… we'll have to see. For now, it's nice to see that they have updated the V8 javascript engine and done more with the Mac port.

Today started out badly. I get to the Naperville Metra train station on time (4:30 am), and they aren't running the trains this morning. At least not yet. Seems there's a freight train that's in the way of all the commuter lines and nothing is leaving Aurora on the way to Chicago. Since I'm not about to want up Liza to take me downtown, and I'm not about to drive myself, it's time to settle in for a little delay.

I had no idea it would be 90 minutes before we'd see a train!

When it arrived, the people who were waiting less time than I had been waiting, decided it was time to play NBA, and started throwing elbows to get on the train. It's an amazing display of a complete lack of decency, which is why I like catching the early train as it's at least got decent people. I guess getting up that early, you appreciate what it takes for others to do the same, and you are a little more respectful of your fellow man.

So over 90 mins late, I arrive at The Shop, and I'm settling in. Trying to put my day back on track, and get into the swing of things so I can forget about this horrible commute.

In general, things are going well until I have to get in on a phone call with the lead developer for another group. This call is about the fact that in order to save space and make comparisons easy, virtually all of the floating point numbers in the system are available as doubles and as integers - signed or unsigned, as the case may be. The decimal representation in an integer is pretty easy - just multiply by a constant and then you have a fixed-precision decimal number. Easy.

So about six or seven months ago, I was building exchange data feeds, and we needed to identify the instruments in a way that allowed the value to be generated at the source - regardless of the instrument. This precluded the existing method which was a database IDENTITY column value that needed to have a stored procedure run if a value didn't exist for a given instrument. This database dependency was horrible, and that's why I invented the SecurityID and SecurityKey. The ID is a 128-bit number that packs in all the details of the instrument in a 128-bit number, based on the UUID, that is suitable for putting in tables, hashes, etc. The Key is a string representation of the same information, and the SecurityID class can take a Key in it's constructor, and generate a Key from a toString() method.

Since OSI came into being, the strike has been a decimalized integer with an offset of 1000. That's what they say, plain and simple. In my SecurityID, I decided to use the same thing so that I could accurately represent any strike without wasting space - after all, I know had 128 bits.

The point of this is that I defined it for the feeds.

Fast-forward to today.

I wrote the C++ classes and gave them to the Java developers for their conversion. I could have written it in Java as well, but they wanted to do the conversion. Turns out, they can't read code to save their lives. They implemented it completely wrong, and yet never asked for verification of the information. They didn't even check the SecurityKeys (string representations) to make sure they were equivalent.

So today this lead developer asks about the decimal multiplier for the strike. They chose to use 10⁴ for all their decimal numbers, and even though this should be an easy conversion for them if they extract the strike from the SecurityID, they wanted the multiplier in the SecurityID to be the same as theirs -- as opposed to the value it's been defined to be for more than 6 months.

The catch was, they already built a lot of code based on this "implementation" they had, and rather than fix it, or correct it, they wanted us to change it. I immediately knew I was messed over because in this place, expediency wins out over everything else. The existing code - forget that they had something to work with for months - forget that they should not have been hitting the SecurityID for the strike in more than one place in the code int he first place - all that mattered was that I now appeared to be asking them to change.

So I had to change instead.

I was silent, for I knew there's nothing I could do or say to change the facts. This was going to be something I had to change. But I'd change it on my terms.

The multiplier was still 1000, but when you asked for the integer value of the strike, it'd add that extra factor of 10 for them. In the string representation, we weren't going to leave it an integer, for it might be OK to look at O:AAPL:20111022:365000:P and know what it was, doing so for another factor of ten on the strike was too much. So I went decimal on the strike, and told them this is what we're doing and why.

I feel that O:AAPL:20111022:345.00:C makes it very clear what the individual components are and what the value of the strike is. If there's a need for the third decimal digit to the right of the decimal point, it will be there, but if it's a zero, then it's drooped. I made the changes to my code, and then had to retrofit it into my Greek Engine and the Ticker Plants.

But wait, there's more…

When my co-worker and I went to tell our boss about this, I was very quiet about the situation because I knew my opinions and feelings had no relevance to this issue, and in fact, were counter-productive to getting things done quickly and efficiently. I could, and would, write the code, but I wasn't going to like it, and I certainly wasn't going to appreciate the way in which this was handled.

After the situation was explained, my boss asked me "What do you think, Bob?"

"My thoughts on this would be counter-productive at this point." I said.

"No, really, what do you think" he reiterated.

"Really. Seriously. They'd be counter-productive" I tried to stress.

"He's right", my co-worker chimed in. "They'd be counter-productive."

"No, really, what are they?"

And so I started to tell him. I was trying to stay a little on the humorous side, but there was no way to hide the fact that I was not happy with the fact of what was happening and why. Midway in this tiny rant he puts up his hands and says "Wow! Bob! I get it!"

Here's the thing: If you don't want to hear the uncomfortable answer, then just don't ask! If you do, then at least be courteous enough to shut up and listen! To forcibly ask me to answer a question that I've already said would be "counter-productive" and then not listen to the answer is, in my opinion, unkind.

Ask or not. But if you ask, listen.

It's then when I came back to my desk and just wanted to scream. But I wrote code instead, and that's what I have to focus on: I write code. I create. The rest of this is lunacy. Stay focused.

Today has been a mix of a lot of changes - some small, some very large, but what really sticks with me about all these is that they are moving the Greek Engine project forward a great deal. It's really pretty amazing, and it all started because I wanted to make sure that I was getting accurate Future prices into the mix for the pricing of the underlying for some options.

What I found was that the Futures weren't getting quoted at all, and that lead to a dive into the code to see where it was they were getting held up. What it turned out to be was a one-line change - I wasn't "wiring up" the output of the Futures feed to the next step in the processing. I was getting the quotes and trades, but I just wasn't doing anything with them. Silly. Easy fix, and then I started seeing data.

Then I noticed that I was saving the state of the feed component to the cache service on red is, but I wasn't reading it back in. Once again, an easy one-line fix, but amazing that I hadn't taken the time to see whether or not this was really working. Got that down, and then things got really big.

One of the other developers let me know that the spreads he was trying to push through the system were going to have negative prices. Since all the code was using uint32_t for a price, that was a major change, and took me quite a while.

While I was moving from unsigned to signed integers for the price, I decided that it was probably a good idea to do the same for the sizes. You just never know, and the last thing I wanted to do was to do this again because some Bond or something will have a negative price to indicate some side or the other. So I changed both. It was massive. But I got it done.

Then it was a few more little things - make the query of Future data correct - I wasn't properly picking up the instruments, and then allowed for the data (quotes, trades) on Futures to create them in the environments if they don't already exist. I'm thinking that new trades and quotes on Futures is good data, and it makes sense to have too many than too few, so let's auto-add them.

In the end, a lot of changes, and I'm going to need to let it run for a bit to make sure things are working, but the effect is profound. We have far better data, better recovery, and things will tick very nicely. Lots of good work makes for a good day.

This morning I saw that Google Chrome dev 15.0.874.21 was out, so I picked it up. It looks a lot like it's getting close to the promotion stage, as backed up by several of the comments on the release notes page. I like that they are working hard on the Mac OS X Lion compatibility, and in recent weeks they've been updating the V8 Javascript engine as well. All good news.

Just wondering what they are going to do next. With the binary-mode web sockets, it's possible to do a lot with this… I just don't know what they are planning on doing. Hope it's interesting.

Today I spent the entire day working on the code to wire in The Broker's service that will inform me when updates to the underlying PostgreSQL database I've converted to, has changed. It's something like an extended trigger for clients of the data. I can read the data from the database and then subscribe to change notifications on a table-level basis. When I get a notification, it'll be up to me to read in the changed data, understand the scope of the change, and then apply the changes to all relevant instruments in the Engine.

It's a lot of detail work, and I had to work out a lot of the details today to get the roles and responsibilities right, but in the end, I think it's all there. The key data tables are in a pretty efficient scheme - only updating those complete units that had some change as opposed to trying to change individual values or blasting everything out on any update.

I still have a lot of testing to do, but I'm going to wait for my co-worker to get back as he's the one setting up the replicated PostgreSQL databases, and I'd like to have him helping me run the tests. SHouldn't be hard, I just want to be able to focus on my end of things while someone else focuses on the source-data end of things.

It feels good to get this all done.

This morning I finished up the code to put the market schedule in for every instrument in my Greek Engine. The idea is that there are default hours for the pre-open market, and then the standard trading day, and finally, the post-close market trading. Since these can vary by instrument type and Underlying, it made sense to make this a little more complex logic than a simple table look-up.

My co-worker had placed all this data into a PostgreSQL table in the replicated database that is the new data service for the application, and so I had to read that in, throw it into decent data structures for looking things up, and then adding methods that made it easy to get at it. Not bad, but it took most of the morning.

Now I need to get some odd pricing logic for certain kinds of instruments from the Quants, and also write the code to take push notifications on the changes to the static data and update as necessary. It's getting close, and I'm very glad to see it finish.