Of Setbacks, Redirections and Fantastic Days
Today was one of those days that you remember for a long time. It has re-defined my ideas of flexibility and gifts presenting themselves as problems in disguise. It's been a big day, but it started out small, and got ugly before it turned around. And all this is a matter of perspective. Mine.
The day was going to be a continuation of the work done yesterday on the Broker by some of the guys in the group. It's undergone a massive re-write in order to give it better functionality and speed, and it's just not quite all back - yet. So I thought it'd be a simple thing to get it all finished today. Unfortunately, I'm a user of the Broker, not one of it's developers, and while I could possibly dig into the code, it's not what I probably should be doing. At least not today.
So I'm waiting for the lead developer on the Broker to get in, and talked to a few of the other developers. It's still broken like it was late last evening, but it's not horrible... it'll get fixed today.
Well... maybe.
Turns out, it's the lead developer's birthday. Yeah... and he's not coming in. FANTASTIC! What am I going to do all day? I can't run any of my ticker plants as the Broker is an integral component of the way they load their configuration, etc. and without the Broker, it's impossible to run anything I have.
I'm dead in the water for an entire day. I hate being unproductive for an entire day. I might as well be home, but I can't leave... I have to stay and try to make the best of it.
Really crappy start to the day.
But then I realize that I might as well look at a few things to see what I can get done that has nothing to do with the running ticker plants. I look at my little notes and see that I might be able to have a look at my SecurityID code to see if there's possibly room in the 128-bit integer for a few more bits.
The problem I was facing was that I needed to have a conflation key for each message coming out of the ticker plant. This key needed to be message-specific, and include (at a minimum) the following components:
- The security key, or complete name of the instrument. This has to include all the relavent facts about the instrument (in the case of an option) and it needs to be two-way - meaning lossless and easily recovered.
- The type of the message - Quote, Print (Trade), etc.
and in the case of an Quote from certain feeds, we needed to include:
- The Exchange the quote is coming from.
The point is that two quotes for the same instrument from difference exchanges cannot conflate one another. They have to remain separate. This is the big reason for the ConflationKey to be defined in terms of the message, as the Quote message needs to add the Exchange to the key, and the other messages don't.
Anyway... I was starting to look at the key and saw that it seemed to be using up all 128 bits for even the simplest of stocks. This shouldn't be. So I looked at the constructor for the SecurityID class, and it's default constructor was blank:
SecurityID::SecurityID() : UUID() { }
but the default constructor for the UUID class is to fill it with a random UUID pattern:
UUID::UUID() : UINT128() { // simple... fill in this bad-boy with 16-bytes of something fill(); }
Lo and behold... that was the problem. I needed to simply clear out the 128-bit int for the SecurityID class, and things would be fine.
SecurityID::SecurityID() : UUID() { // make sure we start with 0 - like any other integer you might use clear(); }
Sure, we end up calling fill() and then clear(), but it's only on construction, and it's for the sake of the natural state of the two classes.
Now, at least, I was seeing the "empty" space in the SecurityID code. What I found was really amazing - there was tons of space left in even the most complex of instrument names. All I needed for the message type was 4 bits, and if there was space (and there was), another 4-bits for the exchange code.
This was a major bonanza! I had several nibbles to work with and I only really needed two! I was then able to go back into all my code and where I had used the SecurityID and Type as two separate values, I could drop the Type as it was now embedded in the SecurityID! This was going to make several things in the code a lot faster as I don't have to deal with std::pair of the two values.
Then I could add in the Exchange to the ConflationKey for the Quote and now I could get the one most worrisome feature into the code - that of proper conflation of the Quote messages. But I didn't stop there, I was on a roll.
I ended up cleaning out a lot of the code based on the non-embedded values in the ConflationKey, and with a little extra work, packaged up the OPRA Quotes into multiple messages allowing for the additional BBO appendages. It's nasty, but rather than have an Option NBBO Engine, we can use the data coming from OPRA for the BBO and send out messages for it. It's not 100% accurate as the sizes aren't right, but it's faster and cheaper than putting up the hardware for the Option NBBO server.
In the end, I had made significant progress in some very nasty problems with the ticker plant, and all because I was forced to look at the SecurityID bits because I didn't have "real work" to do. I have a new respect for the subtle ways of fate and how I really need to stay far more open and accepting of what appears to me to be set-backs, as they are often the greatest of opportunities in disguise.
Great lesson to learn.
Great day.