What's it all about, Alfie?

Today I had an other unsuccessful day testing my ticker plants against the open and the OPRA feeds. They were pumping out far more data than my code could process. It was working, but with all the changes required for the necessary functionality, I was falling behind, and it wasn't a great feeling.

I spent the day looking at things like making ivars protected and using them directly as opposed to passing values to protected setter methods. A stretch, yeah, but that's what today has been about - trying to find out what's up with the speed and why it was working fine before, and now it's really pretty bad.

I'll have to try these on Monday and see if they make any difference.

Today I've spent a ton of time with the uint128_t trie to get it as fast as possible. My goal was to beat the old system, and eventually, I think I have. But getting there was a painful process that had a lot of battles along the way.

When I started testing the uint128_t trie, I noticed that the access was significantly slower than the old character-based trie I had been using. Sure, the old one wasn't going to really work for what I needed, but still... to have a 10x speed hit is not what I was looking to do. So I dug into it.

Turns out the new trie is amazingly fast. I mean just blinding. Good for me. The problem was in generating the conflation key that I use in the trie to store the messages. Really... it was around 7 ms to generate the key and then only 0.01 ms to actually put it in the trie. Yikes!

So I had to keep digging. Turns out I was doing a lot more copying than I had to do, and by 'had to do' I mean 'designed into the API'. I was using STL containers for things when it was possible to make the methods really work on (char *) arrays and then wrap those into the ones that used the STL containers for backward compatibility. This netted me a considerable speed improvement, but I'm still not happy.

I also dug into the payload pools and found that I could do a little better job there as well. Again, no major change in the code, but every little bit is going to help in this guy. I've got more feeds than I have CPUs, and it's getting to be a little nasty to worry about the context switching.

In the end, I changed a lot of little things, but got the speed close to what it was. I'll have to wait and see what happens on the open tomorrow morning to be sure. I'm concerned, but cautiously optimistic.

Today was spent working a lot on the efficiency of the ticker plants - most notably on removing the excessive use of malloc() and free() or new and delete and replacing it with some pooled containers for the processing of the UDP datagrams and the payloads of the ZeroMQ messages. I'm not really sure how inefficient linux is for dealing with small allocations, say less than 1kB, but it can't be good to have 20,000/sec flipping through the system.

It's really pretty simple - given the fact that I'm dealing with reasonably small container sizes. I just needed to have an STL list of some kind - a simple spinlock to protect it, and then a simple alloc() method to get one from the pool, or create one if the pool is empty, and recycle() to put it back in the pool, if there's room, or delete it if not. Not rocket science.

In fact, I made a special StringPool just for the ZeroMQ messages as they are being held in simple STL std::string objects. The bonus of all this is that I don't ahve to have a "receive buffer" and then copy the data from the receive buffer into the container for pushing onto the stack. I can allocate one from the pool, have the boost ASIO put the data in the container, and then simply transfer the pointer into the stack.

Far simple. Very sweet, in fact.

Once I had it done, it has to work better even if it doesn't spec out to be any faster - the switches to the system for malloc() and free() have to be making a positive difference. Good enough.

Today I noticed that I had a serious problem with the CPU usage of my ticker plants. Specifically, the OPRA channels were 3x or 4x larger than they had been before I had made the last round of changes. Not good. Many of the ticker plants were above 100% CPU utilization, and when you have that many plants on an 8 core box, it's not good to have four or more of them taking more than a complete CPU. Yikes.

So today was spent trying to find where they problems were. I guess I knew where they were - in the code that changed, but the problem is, that code was for the conflation queue, and that needed to be changed, and the resulting code is just awfully slow.

I have a feeling it's in the boost unordered_map, but I'm going to try and make it work, if possible, for the alternative is a nasty bit of code - another trie and this time very large. I don't want to have to do that.

Today was a day of cleaning up a lot of little nagging issues on the ticker plants. I spent time fiddling with the timing on lockless queue polling, IRC responding, exchange mappings... all these things that I needed to get to - eventually, but today was the day to "clean the decks". Not bad.

Nice day, too. It's great to get that sense of completion.

Today was a long one... very long. But it was for a good cause. Today I finally got the bulk of the direct dial Broker code working in my apps. It wasn't easy, and probably most of that was my fault. I had looked at the code for my TCP clients and proxies and copied that as a starting point. What I failed to realize was that the Broker code needed to have entirely different handshaking on the connections, and while it would appear to work, it was really a hopeless mess.

No more. It's looking pretty good and I'm hoping to get the final touches on it in the morning.

Today was one of those days that you remember for a long time. It has re-defined my ideas of flexibility and gifts presenting themselves as problems in disguise. It's been a big day, but it started out small, and got ugly before it turned around. And all this is a matter of perspective. Mine.

The day was going to be a continuation of the work done yesterday on the Broker by some of the guys in the group. It's undergone a massive re-write in order to give it better functionality and speed, and it's just not quite all back - yet. So I thought it'd be a simple thing to get it all finished today. Unfortunately, I'm a user of the Broker, not one of it's developers, and while I could possibly dig into the code, it's not what I probably should be doing. At least not today.

So I'm waiting for the lead developer on the Broker to get in, and talked to a few of the other developers. It's still broken like it was late last evening, but it's not horrible... it'll get fixed today.

Well... maybe.

Turns out, it's the lead developer's birthday. Yeah... and he's not coming in. FANTASTIC! What am I going to do all day? I can't run any of my ticker plants as the Broker is an integral component of the way they load their configuration, etc. and without the Broker, it's impossible to run anything I have.

I'm dead in the water for an entire day. I hate being unproductive for an entire day. I might as well be home, but I can't leave... I have to stay and try to make the best of it.

Really crappy start to the day.

But then I realize that I might as well look at a few things to see what I can get done that has nothing to do with the running ticker plants. I look at my little notes and see that I might be able to have a look at my SecurityID code to see if there's possibly room in the 128-bit integer for a few more bits.

The problem I was facing was that I needed to have a conflation key for each message coming out of the ticker plant. This key needed to be message-specific, and include (at a minimum) the following components:

• The security key, or complete name of the instrument. This has to include all the relavent facts about the instrument (in the case of an option) and it needs to be two-way - meaning lossless and easily recovered.
• The type of the message - Quote, Print (Trade), etc.

and in the case of an Quote from certain feeds, we needed to include:

• The Exchange the quote is coming from.

The point is that two quotes for the same instrument from difference exchanges cannot conflate one another. They have to remain separate. This is the big reason for the ConflationKey to be defined in terms of the message, as the Quote message needs to add the Exchange to the key, and the other messages don't.

Anyway... I was starting to look at the key and saw that it seemed to be using up all 128 bits for even the simplest of stocks. This shouldn't be. So I looked at the constructor for the SecurityID class, and it's default constructor was blank:

  SecurityID::SecurityID() :
    UUID()
  {
  }

but the default constructor for the UUID class is to fill it with a random UUID pattern:

  UUID::UUID() :
    UINT128()
  {
    // simple... fill in this bad-boy with 16-bytes of something
    fill();
  }

Lo and behold... that was the problem. I needed to simply clear out the 128-bit int for the SecurityID class, and things would be fine.

  SecurityID::SecurityID() :
    UUID()
  {
    // make sure we start with 0 - like any other integer you might use
    clear();
  }

Sure, we end up calling fill() and then clear(), but it's only on construction, and it's for the sake of the natural state of the two classes.

Now, at least, I was seeing the "empty" space in the SecurityID code. What I found was really amazing - there was tons of space left in even the most complex of instrument names. All I needed for the message type was 4 bits, and if there was space (and there was), another 4-bits for the exchange code.

This was a major bonanza! I had several nibbles to work with and I only really needed two! I was then able to go back into all my code and where I had used the SecurityID and Type as two separate values, I could drop the Type as it was now embedded in the SecurityID! This was going to make several things in the code a lot faster as I don't have to deal with std::pair of the two values.

Then I could add in the Exchange to the ConflationKey for the Quote and now I could get the one most worrisome feature into the code - that of proper conflation of the Quote messages. But I didn't stop there, I was on a roll.

I ended up cleaning out a lot of the code based on the non-embedded values in the ConflationKey, and with a little extra work, packaged up the OPRA Quotes into multiple messages allowing for the additional BBO appendages. It's nasty, but rather than have an Option NBBO Engine, we can use the data coming from OPRA for the BBO and send out messages for it. It's not 100% accurate as the sizes aren't right, but it's faster and cheaper than putting up the hardware for the Option NBBO server.

In the end, I had made significant progress in some very nasty problems with the ticker plant, and all because I was forced to look at the SecurityID bits because I didn't have "real work" to do. I have a new respect for the subtle ways of fate and how I really need to stay far more open and accepting of what appears to me to be set-backs, as they are often the greatest of opportunities in disguise.

Great lesson to learn.

Great day.

This morning I was lamenting that the US Options Expiration calendar that I've been subscribed to for a few years now had a problem, and it was no longer available on iCal Exchange. I think from this note, it was lost, and the author never republished it. So I got tired of lamenting and did something about it. I went to the CBOE page and created a new US Options Expiration calendar. It took me all of about 15 mins and I'm sitting here now wondering why I didn't do this a long time ago.

Anyway... it's got the normal equity option expiration, as well as the VIX expiration and even the quarterly expirations. It's nothing fancy, but it's decent. I tried to keep the event titles as small as possible to make them look nice on my small iCal window, but it's a matter of taste, and that's the way it goes.

Use it if you want. I'll update it every year so you won't have to worry about that.

Whew! It's been a heck of a day... again. Today I was able to get a complete first cut of the lisp-like parser all done and checked-in. As with yesterday, the bulk of the work was really on the variant class and then just a little work in the functions of the parser. Today it was math operations and equality tests. I didn't think about it at the time, but my equality tests were more restrictive than I really wanted.

For example, originally, if I had the code:

  variant   v(25.0);
 
  if (v == 25) {
    ...
  }

it would fail. Why? Because the variant is a 'double', and the test is an integer. I had two different data types for these - as opposed to one "number" type. While that seems like a little thing, it's not - the rounding and the storage, not to mention the serialization size, all favor heavily the two distinct types.

So I had to go in and re-write all the equality and inequality operators to take this into account. Not horrible, but again, it was a few hours to get everything done right.

Thankfully, it's all done now, and the tests run great.

Today I've been hard at work filling in the parser I'm creating with the obvious collection of functions that any script is going to need. Things like add, subtract, multiply, divide... plus the more advanced math calculations - they all need to be done, but what makes this all slow going is the fact that to do this right, we need to implement these features in the variant class first, and then use them in the parser. In fact, using them in the parser is the easy part. It's getting them all in the variant class in the first place.

The reason it's slow going is that it's not enough to make a simple inequality set:

  bool operator==( variant & anOther ) const;
  bool operator==( const variant & anOther ) const;
  bool operator!=( variant & anOther ) const;
  bool operator!=( const variant & anOther ) const;

we're going to need the complete compliment of inequalities:

  bool operator==( variant & anOther ) const;
  bool operator==( const variant & anOther ) const;
  bool operator!=( variant & anOther ) const;
  bool operator!=( const variant & anOther ) const;
  bool operator<( variant & anOther ) const;
  bool operator<( const variant & anOther ) const;
  bool operator<=( variant & anOther ) const;
  bool operator<=( const variant & anOther ) const;
  bool operator>( variant & anOther ) const;
  bool operator>( const variant & anOther ) const;
  bool operator>=( variant & anOther ) const;
  bool operator>=( const variant & anOther ) const;

For the most part, this isn't too terribly hard, but you have to remember that I've got more than a dozen different types of values to check in a variant, and then there's the real work - all the convenience methods.

To make it easy to write:

  variant    v(25.0);
  if (v < 12) {
    ...
  }

I need to have a lot more operators as well. If I look at just the equality operator, I end up with something more like this:

  bool operator==( variant & anOther ) const;
  bool operator==( const variant & anOther ) const;
  bool operator==( varmap & anOther ) const;
  bool operator==( const varmap & anOther ) const;
  bool operator==( varlist & anOther ) const;
  bool operator==( const varlist & anOther ) const;
  bool operator==( uint8_t aValue ) const;
  bool operator==( int aValue ) const;
  bool operator==( int64_t aValue ) const;
  bool operator==( uint64_t aValue ) const;
  bool operator==( float aValue ) const;
  bool operator==( double aValue ) const;
  bool operator==( bool aValue ) const;
  bool operator==( std::string & aValue ) const;
  bool operator==( const std::string & aValue ) const;
  bool operator==( char *aValue ) const;
  bool operator==( const char *aValue ) const;
  bool operator==( uuid_t & aValue ) const;
  bool operator==( const uuid_t & aValue ) const;
  bool operator==( secID_t & aValue ) const;
  bool operator==( const secID_t & aValue ) const;
  bool operator==( error_t & aValue ) const;
  bool operator==( const error_t & aValue ) const;

So even if we are as clever as can be, this is a ton of code to write, and test. It's just brutal at times to make sure you haven't made any typos in the code.

I will say that I was very pleased with the one simplification I made. It's pretty easy to see that you can write the inequalities in terms of one another. However, that can take a while, and you might not end up with valid inequalities. For example, when you have two lists - which one is greater than the other? It's not easy.

My solution was to implement the:

  bool operator>=(...) const;
  bool operator<=(...) const;

methods, and then define the others in terms of them:

  bool variant::operator<(...) const
  {
    return !this->operator>=(...);
  }

So that I could be assured of at least having the "equality" in the code. Then, the others are logical combinations of these. It's not perfect, but it saves time and code, and it's a lot less headache than trying to implement a greater than on a list.