What's it all about, Alfie?

Today was spent working a lot on the efficiency of the ticker plants - most notably on removing the excessive use of malloc() and free() or new and delete and replacing it with some pooled containers for the processing of the UDP datagrams and the payloads of the ZeroMQ messages. I'm not really sure how inefficient linux is for dealing with small allocations, say less than 1kB, but it can't be good to have 20,000/sec flipping through the system.

It's really pretty simple - given the fact that I'm dealing with reasonably small container sizes. I just needed to have an STL list of some kind - a simple spinlock to protect it, and then a simple alloc() method to get one from the pool, or create one if the pool is empty, and recycle() to put it back in the pool, if there's room, or delete it if not. Not rocket science.

In fact, I made a special StringPool just for the ZeroMQ messages as they are being held in simple STL std::string objects. The bonus of all this is that I don't ahve to have a "receive buffer" and then copy the data from the receive buffer into the container for pushing onto the stack. I can allocate one from the pool, have the boost ASIO put the data in the container, and then simply transfer the pointer into the stack.

Far simple. Very sweet, in fact.

Once I had it done, it has to work better even if it doesn't spec out to be any faster - the switches to the system for malloc() and free() have to be making a positive difference. Good enough.

He is my one and only son, and while parents and teenagers don't always see eye-to-eye, he is my son, and I wish him a very happy 17th birthday.

Today I noticed that I had a serious problem with the CPU usage of my ticker plants. Specifically, the OPRA channels were 3x or 4x larger than they had been before I had made the last round of changes. Not good. Many of the ticker plants were above 100% CPU utilization, and when you have that many plants on an 8 core box, it's not good to have four or more of them taking more than a complete CPU. Yikes.

So today was spent trying to find where they problems were. I guess I knew where they were - in the code that changed, but the problem is, that code was for the conflation queue, and that needed to be changed, and the resulting code is just awfully slow.

I have a feeling it's in the boost unordered_map, but I'm going to try and make it work, if possible, for the alternative is a nasty bit of code - another trie and this time very large. I don't want to have to do that.

Today was a day of cleaning up a lot of little nagging issues on the ticker plants. I spent time fiddling with the timing on lockless queue polling, IRC responding, exchange mappings... all these things that I needed to get to - eventually, but today was the day to "clean the decks". Not bad.

Nice day, too. It's great to get that sense of completion.

Today was a long one... very long. But it was for a good cause. Today I finally got the bulk of the direct dial Broker code working in my apps. It wasn't easy, and probably most of that was my fault. I had looked at the code for my TCP clients and proxies and copied that as a starting point. What I failed to realize was that the Broker code needed to have entirely different handshaking on the connections, and while it would appear to work, it was really a hopeless mess.

No more. It's looking pretty good and I'm hoping to get the final touches on it in the morning.

I've been playing with the Apple App Store a bit today and I'm really surprised at the number of apps that I use that it contains. I mean really surprised. The problem is that most of them are detected as "Installed", but if the Bundle IDs are different, as it is with MarsEdit and TextWrangler, then it's not. But in the short-run, it'd be better to have them all not recognized. Basically, if the Bundle ID exists, and it's not in the "Purchased" list, then it should be tagged as "Grandfather" or something. It's on the box, but it's not really recognized properly and won't be updated, etc.

This, in my opinion, is the biggest problem with the App Store: the inability to "import" or "register" purchased apps with the App Store so as to "fold them in" to the App Store umbrella. They could do this with a quick serial number check, or a service that Apple provides that checks to see if it's really licensed to a person... something. They really should try to fold in all the legacy folks.

But I'm guessing that they look at this as a transient condition. The next major release of any one of these apps puts me in the position to buy it from the App Store. No upgrade pricing, however, as I'm not "upgrading". But it's probably in their mind that this is transient, and will be a slight grumbling for a little while and then die down.

Maybe... but all the apps I have provide upgrade pricing, so it's not in my best interests now to buy them new. I would like to think they'd figure this out.

I had a nasty little problem the other day, and I couldn't figure it out until I really started googling the problem and saw that the solution was (of all places) in the man pages. Here's the set-up for the problem.

I have a git repo and at the top level it looks like this:

  Makefile
  README
  bin/
  doc/
  java/
  lib/
  logs/
  src/
  tests/

and because the lib directory contains nothing by generated files (the so libraries), it makes sense to have a top-level .gitignore file looking like this:

  *.swp
  *.swo
  *.tgz
  lib
  logs
  __dist
  core.*

But then when we look at the next level, we see that things get complicated:

  Makefile
  README
  bin/
  doc/
  java/
    Makefile
    build.xml
    classes/
    dist/
    lib/
    peak6/
    tests/
  lib/
  logs/
  src/
  tests/

And in the java/lib/ directory we have third-party jars that we want included in the repo. The problem is, git looks at the top-level .gitignore and sees the line with lib in it, and therefore, the lib in the java directory is also ignored.

But there's a fix. Un-ignore the java/lib. How to do that?

Make the .gitignore fine in the java directory look like this:

  classes
  dist
  !lib
  *.classes
  *.swp
  *.swo

and it's the inclusion of the !lib line that's telling git "Hey, don't ignore this guy".

The more I use git the more impressed I am with it. Subversion made me copy the files to ignore to each directory, git is nicer with the directory structure providing the inheritance, and with this, I can now be as selective as I need. Sweet.

It's interesting, I'll give you that. The App Store is very interesting. It's still the first hours of the App Store, but there are several things I like about it, and a few that worry me. First, the things I really like about it:

• Installed apps go into /Applications, just as I had hoped. Good.
• Previously purchased apps can be detected by the App Store, and shown as "Installed". This is not universal, however, as I have BBEdit and TextWrangler from BareBones, and BBEdit was identified as "Installed", but TextWrangler was not. Maybe it had to do with the version... BBEdit was current and TextWrangler was 3.1 while the shipping version was 3.5, but still... it makes me think there's something that has to be done to the app in order to facilitate this feature.
• It's very much like the iOS store, so it's easy to find all kinds of things, but it'll soon fill with all kinds of junk, and the process will be challenging once again. Still... this is the ideal way for my Mom to get an app. It's just so simple.

I'm not sure how things will play out. For example, if you don't have the App Store app running, will it detect the updates? Probably not. It's not like the Dashboard that runs all the time, but maybe that will change in 10.7 (Lion). At that point, I'd expect them to have something small running all the time for the App Store. Then updates are "sent to you" even if it's a pull from the client, as opposed to the "push notifications" on the iPhone.

I'm surprised that as many apps of mine are already there. BBEdit was a shocker. I'm guessing Xcode isn't going out this way, but that might be a mistake on my part. I'll have to spend more time looking for what's there, and maybe it'll be better than I think.

UPDATE: Of the apps I use every day, here's the status with regards to the App Store:

I'm sure I'll be seeing a big change in these as the days and months progress. I'm just not sure how it'll pan out in the end. I'm a little concerned that this is going to be the preferred method, and that's OK, as long as the apps don't suffer. I think in the end, there will be plenty of apps that won't use this delivery system because they won't pass the rules Apple has set down.

[11:28] UPDATE: I got back news from MarsEdit support - the reason that it's not being seen as "Installed" is that the App ID used for the "direct sale" app was not available for use in the App Store version due to Apple's rules. Daniel states that there are differences in the apps as well, and he's waiting/working to see how these issues work themselves out in the coming weeks and months.

[12:26] UPDATE: It appears that this "Installed" status is a bug in the App Store. It's doubtful that the updating will work, and so it may be that I can buy new software in the App Store, but updates and upgrades seem to be out. As it appears now. Sad, but understandable. I wish Apple had made some kind of "Import..." function in the App Store that they have in iTunes to "recognize the pre-installed apps. Even if it deletes them and re-installs them from the App Store. That would have been nice.

I installed something this morning - the latest Citrix ICA plugin for Mac OS X, and then I couldn't seem to find it. I started wondering where it might be, and then wanted to see the contents of what I had installed. It's supposedly all in the Receipts directory, in on of the pkg bundles, but I couldn't remember how. So I did some digging, and figured it out - again.

The first thing is to find the receipt for the package you installed. For the sake of example, let's call this package Squish.pkg, and there are two places it will be found: ~/Library/Receipts/ or /Library/Receipts/. So look these two places and find it.

When you've found it, get to a shell and get into this receipt and find the bill of materials (bom) file like this:

  $ cd ~/Library/Receipts/
  $ cd Squish.pkg
  $ find . -name \*.bom
  Contents/Archive.bom

where the output of the find command should list the one bom file that you are looking for. Then you can list the contents of that file with:

  $ lsbom -fls Contents/Archive.bom

If you're really interested in being efficient, you can do this all with:

  $ cd ~/Library/Receipts/
  $ cd Squish.pkg
  $ lsbom -fls `find . -name \*.bom`

What I found was what I thought had happened - the Citrix client itself was updated, so nothing new was added to the system - only updates to the existing ICA client. Fair enough. But now I'm glad I have this written down so I don't have to hunt for it.

Today was one of those days that you remember for a long time. It has re-defined my ideas of flexibility and gifts presenting themselves as problems in disguise. It's been a big day, but it started out small, and got ugly before it turned around. And all this is a matter of perspective. Mine.

The day was going to be a continuation of the work done yesterday on the Broker by some of the guys in the group. It's undergone a massive re-write in order to give it better functionality and speed, and it's just not quite all back - yet. So I thought it'd be a simple thing to get it all finished today. Unfortunately, I'm a user of the Broker, not one of it's developers, and while I could possibly dig into the code, it's not what I probably should be doing. At least not today.

So I'm waiting for the lead developer on the Broker to get in, and talked to a few of the other developers. It's still broken like it was late last evening, but it's not horrible... it'll get fixed today.

Well... maybe.

Turns out, it's the lead developer's birthday. Yeah... and he's not coming in. FANTASTIC! What am I going to do all day? I can't run any of my ticker plants as the Broker is an integral component of the way they load their configuration, etc. and without the Broker, it's impossible to run anything I have.

I'm dead in the water for an entire day. I hate being unproductive for an entire day. I might as well be home, but I can't leave... I have to stay and try to make the best of it.

Really crappy start to the day.

But then I realize that I might as well look at a few things to see what I can get done that has nothing to do with the running ticker plants. I look at my little notes and see that I might be able to have a look at my SecurityID code to see if there's possibly room in the 128-bit integer for a few more bits.

The problem I was facing was that I needed to have a conflation key for each message coming out of the ticker plant. This key needed to be message-specific, and include (at a minimum) the following components:

• The security key, or complete name of the instrument. This has to include all the relavent facts about the instrument (in the case of an option) and it needs to be two-way - meaning lossless and easily recovered.
• The type of the message - Quote, Print (Trade), etc.

and in the case of an Quote from certain feeds, we needed to include:

• The Exchange the quote is coming from.

The point is that two quotes for the same instrument from difference exchanges cannot conflate one another. They have to remain separate. This is the big reason for the ConflationKey to be defined in terms of the message, as the Quote message needs to add the Exchange to the key, and the other messages don't.

Anyway... I was starting to look at the key and saw that it seemed to be using up all 128 bits for even the simplest of stocks. This shouldn't be. So I looked at the constructor for the SecurityID class, and it's default constructor was blank:

  SecurityID::SecurityID() :
    UUID()
  {
  }

but the default constructor for the UUID class is to fill it with a random UUID pattern:

  UUID::UUID() :
    UINT128()
  {
    // simple... fill in this bad-boy with 16-bytes of something
    fill();
  }

Lo and behold... that was the problem. I needed to simply clear out the 128-bit int for the SecurityID class, and things would be fine.

  SecurityID::SecurityID() :
    UUID()
  {
    // make sure we start with 0 - like any other integer you might use
    clear();
  }

Sure, we end up calling fill() and then clear(), but it's only on construction, and it's for the sake of the natural state of the two classes.

Now, at least, I was seeing the "empty" space in the SecurityID code. What I found was really amazing - there was tons of space left in even the most complex of instrument names. All I needed for the message type was 4 bits, and if there was space (and there was), another 4-bits for the exchange code.

This was a major bonanza! I had several nibbles to work with and I only really needed two! I was then able to go back into all my code and where I had used the SecurityID and Type as two separate values, I could drop the Type as it was now embedded in the SecurityID! This was going to make several things in the code a lot faster as I don't have to deal with std::pair of the two values.

Then I could add in the Exchange to the ConflationKey for the Quote and now I could get the one most worrisome feature into the code - that of proper conflation of the Quote messages. But I didn't stop there, I was on a roll.

I ended up cleaning out a lot of the code based on the non-embedded values in the ConflationKey, and with a little extra work, packaged up the OPRA Quotes into multiple messages allowing for the additional BBO appendages. It's nasty, but rather than have an Option NBBO Engine, we can use the data coming from OPRA for the BBO and send out messages for it. It's not 100% accurate as the sizes aren't right, but it's faster and cheaper than putting up the hardware for the Option NBBO server.

In the end, I had made significant progress in some very nasty problems with the ticker plant, and all because I was forced to look at the SecurityID bits because I didn't have "real work" to do. I have a new respect for the subtle ways of fate and how I really need to stay far more open and accepting of what appears to me to be set-backs, as they are often the greatest of opportunities in disguise.

Great lesson to learn.

Great day.