What's it all about, Alfie?

Today I noticed that I had a serious problem with the CPU usage of my ticker plants. Specifically, the OPRA channels were 3x or 4x larger than they had been before I had made the last round of changes. Not good. Many of the ticker plants were above 100% CPU utilization, and when you have that many plants on an 8 core box, it's not good to have four or more of them taking more than a complete CPU. Yikes.

So today was spent trying to find where they problems were. I guess I knew where they were - in the code that changed, but the problem is, that code was for the conflation queue, and that needed to be changed, and the resulting code is just awfully slow.

I have a feeling it's in the boost unordered_map, but I'm going to try and make it work, if possible, for the alternative is a nasty bit of code - another trie and this time very large. I don't want to have to do that.

Today was a day of cleaning up a lot of little nagging issues on the ticker plants. I spent time fiddling with the timing on lockless queue polling, IRC responding, exchange mappings... all these things that I needed to get to - eventually, but today was the day to "clean the decks". Not bad.

Nice day, too. It's great to get that sense of completion.

Today was a long one... very long. But it was for a good cause. Today I finally got the bulk of the direct dial Broker code working in my apps. It wasn't easy, and probably most of that was my fault. I had looked at the code for my TCP clients and proxies and copied that as a starting point. What I failed to realize was that the Broker code needed to have entirely different handshaking on the connections, and while it would appear to work, it was really a hopeless mess.

No more. It's looking pretty good and I'm hoping to get the final touches on it in the morning.

I've been playing with the Apple App Store a bit today and I'm really surprised at the number of apps that I use that it contains. I mean really surprised. The problem is that most of them are detected as "Installed", but if the Bundle IDs are different, as it is with MarsEdit and TextWrangler, then it's not. But in the short-run, it'd be better to have them all not recognized. Basically, if the Bundle ID exists, and it's not in the "Purchased" list, then it should be tagged as "Grandfather" or something. It's on the box, but it's not really recognized properly and won't be updated, etc.

This, in my opinion, is the biggest problem with the App Store: the inability to "import" or "register" purchased apps with the App Store so as to "fold them in" to the App Store umbrella. They could do this with a quick serial number check, or a service that Apple provides that checks to see if it's really licensed to a person... something. They really should try to fold in all the legacy folks.

But I'm guessing that they look at this as a transient condition. The next major release of any one of these apps puts me in the position to buy it from the App Store. No upgrade pricing, however, as I'm not "upgrading". But it's probably in their mind that this is transient, and will be a slight grumbling for a little while and then die down.

Maybe... but all the apps I have provide upgrade pricing, so it's not in my best interests now to buy them new. I would like to think they'd figure this out.

I had a nasty little problem the other day, and I couldn't figure it out until I really started googling the problem and saw that the solution was (of all places) in the man pages. Here's the set-up for the problem.

I have a git repo and at the top level it looks like this:

  Makefile
  README
  bin/
  doc/
  java/
  lib/
  logs/
  src/
  tests/

and because the lib directory contains nothing by generated files (the so libraries), it makes sense to have a top-level .gitignore file looking like this:

  *.swp
  *.swo
  *.tgz
  lib
  logs
  __dist
  core.*

But then when we look at the next level, we see that things get complicated:

  Makefile
  README
  bin/
  doc/
  java/
    Makefile
    build.xml
    classes/
    dist/
    lib/
    peak6/
    tests/
  lib/
  logs/
  src/
  tests/

And in the java/lib/ directory we have third-party jars that we want included in the repo. The problem is, git looks at the top-level .gitignore and sees the line with lib in it, and therefore, the lib in the java directory is also ignored.

But there's a fix. Un-ignore the java/lib. How to do that?

Make the .gitignore fine in the java directory look like this:

  classes
  dist
  !lib
  *.classes
  *.swp
  *.swo

and it's the inclusion of the !lib line that's telling git "Hey, don't ignore this guy".

The more I use git the more impressed I am with it. Subversion made me copy the files to ignore to each directory, git is nicer with the directory structure providing the inheritance, and with this, I can now be as selective as I need. Sweet.

It's interesting, I'll give you that. The App Store is very interesting. It's still the first hours of the App Store, but there are several things I like about it, and a few that worry me. First, the things I really like about it:

• Installed apps go into /Applications, just as I had hoped. Good.
• Previously purchased apps can be detected by the App Store, and shown as "Installed". This is not universal, however, as I have BBEdit and TextWrangler from BareBones, and BBEdit was identified as "Installed", but TextWrangler was not. Maybe it had to do with the version... BBEdit was current and TextWrangler was 3.1 while the shipping version was 3.5, but still... it makes me think there's something that has to be done to the app in order to facilitate this feature.
• It's very much like the iOS store, so it's easy to find all kinds of things, but it'll soon fill with all kinds of junk, and the process will be challenging once again. Still... this is the ideal way for my Mom to get an app. It's just so simple.

I'm not sure how things will play out. For example, if you don't have the App Store app running, will it detect the updates? Probably not. It's not like the Dashboard that runs all the time, but maybe that will change in 10.7 (Lion). At that point, I'd expect them to have something small running all the time for the App Store. Then updates are "sent to you" even if it's a pull from the client, as opposed to the "push notifications" on the iPhone.

I'm surprised that as many apps of mine are already there. BBEdit was a shocker. I'm guessing Xcode isn't going out this way, but that might be a mistake on my part. I'll have to spend more time looking for what's there, and maybe it'll be better than I think.

UPDATE: Of the apps I use every day, here's the status with regards to the App Store:

I'm sure I'll be seeing a big change in these as the days and months progress. I'm just not sure how it'll pan out in the end. I'm a little concerned that this is going to be the preferred method, and that's OK, as long as the apps don't suffer. I think in the end, there will be plenty of apps that won't use this delivery system because they won't pass the rules Apple has set down.

[11:28] UPDATE: I got back news from MarsEdit support - the reason that it's not being seen as "Installed" is that the App ID used for the "direct sale" app was not available for use in the App Store version due to Apple's rules. Daniel states that there are differences in the apps as well, and he's waiting/working to see how these issues work themselves out in the coming weeks and months.

[12:26] UPDATE: It appears that this "Installed" status is a bug in the App Store. It's doubtful that the updating will work, and so it may be that I can buy new software in the App Store, but updates and upgrades seem to be out. As it appears now. Sad, but understandable. I wish Apple had made some kind of "Import..." function in the App Store that they have in iTunes to "recognize the pre-installed apps. Even if it deletes them and re-installs them from the App Store. That would have been nice.

I installed something this morning - the latest Citrix ICA plugin for Mac OS X, and then I couldn't seem to find it. I started wondering where it might be, and then wanted to see the contents of what I had installed. It's supposedly all in the Receipts directory, in on of the pkg bundles, but I couldn't remember how. So I did some digging, and figured it out - again.

The first thing is to find the receipt for the package you installed. For the sake of example, let's call this package Squish.pkg, and there are two places it will be found: ~/Library/Receipts/ or /Library/Receipts/. So look these two places and find it.

When you've found it, get to a shell and get into this receipt and find the bill of materials (bom) file like this:

  $ cd ~/Library/Receipts/
  $ cd Squish.pkg
  $ find . -name \*.bom
  Contents/Archive.bom

where the output of the find command should list the one bom file that you are looking for. Then you can list the contents of that file with:

  $ lsbom -fls Contents/Archive.bom

If you're really interested in being efficient, you can do this all with:

  $ cd ~/Library/Receipts/
  $ cd Squish.pkg
  $ lsbom -fls `find . -name \*.bom`

What I found was what I thought had happened - the Citrix client itself was updated, so nothing new was added to the system - only updates to the existing ICA client. Fair enough. But now I'm glad I have this written down so I don't have to hunt for it.

Today was one of those days that you remember for a long time. It has re-defined my ideas of flexibility and gifts presenting themselves as problems in disguise. It's been a big day, but it started out small, and got ugly before it turned around. And all this is a matter of perspective. Mine.

The day was going to be a continuation of the work done yesterday on the Broker by some of the guys in the group. It's undergone a massive re-write in order to give it better functionality and speed, and it's just not quite all back - yet. So I thought it'd be a simple thing to get it all finished today. Unfortunately, I'm a user of the Broker, not one of it's developers, and while I could possibly dig into the code, it's not what I probably should be doing. At least not today.

So I'm waiting for the lead developer on the Broker to get in, and talked to a few of the other developers. It's still broken like it was late last evening, but it's not horrible... it'll get fixed today.

Well... maybe.

Turns out, it's the lead developer's birthday. Yeah... and he's not coming in. FANTASTIC! What am I going to do all day? I can't run any of my ticker plants as the Broker is an integral component of the way they load their configuration, etc. and without the Broker, it's impossible to run anything I have.

I'm dead in the water for an entire day. I hate being unproductive for an entire day. I might as well be home, but I can't leave... I have to stay and try to make the best of it.

Really crappy start to the day.

But then I realize that I might as well look at a few things to see what I can get done that has nothing to do with the running ticker plants. I look at my little notes and see that I might be able to have a look at my SecurityID code to see if there's possibly room in the 128-bit integer for a few more bits.

The problem I was facing was that I needed to have a conflation key for each message coming out of the ticker plant. This key needed to be message-specific, and include (at a minimum) the following components:

• The security key, or complete name of the instrument. This has to include all the relavent facts about the instrument (in the case of an option) and it needs to be two-way - meaning lossless and easily recovered.
• The type of the message - Quote, Print (Trade), etc.

and in the case of an Quote from certain feeds, we needed to include:

• The Exchange the quote is coming from.

The point is that two quotes for the same instrument from difference exchanges cannot conflate one another. They have to remain separate. This is the big reason for the ConflationKey to be defined in terms of the message, as the Quote message needs to add the Exchange to the key, and the other messages don't.

Anyway... I was starting to look at the key and saw that it seemed to be using up all 128 bits for even the simplest of stocks. This shouldn't be. So I looked at the constructor for the SecurityID class, and it's default constructor was blank:

  SecurityID::SecurityID() :
    UUID()
  {
  }

but the default constructor for the UUID class is to fill it with a random UUID pattern:

  UUID::UUID() :
    UINT128()
  {
    // simple... fill in this bad-boy with 16-bytes of something
    fill();
  }

Lo and behold... that was the problem. I needed to simply clear out the 128-bit int for the SecurityID class, and things would be fine.

  SecurityID::SecurityID() :
    UUID()
  {
    // make sure we start with 0 - like any other integer you might use
    clear();
  }

Sure, we end up calling fill() and then clear(), but it's only on construction, and it's for the sake of the natural state of the two classes.

Now, at least, I was seeing the "empty" space in the SecurityID code. What I found was really amazing - there was tons of space left in even the most complex of instrument names. All I needed for the message type was 4 bits, and if there was space (and there was), another 4-bits for the exchange code.

This was a major bonanza! I had several nibbles to work with and I only really needed two! I was then able to go back into all my code and where I had used the SecurityID and Type as two separate values, I could drop the Type as it was now embedded in the SecurityID! This was going to make several things in the code a lot faster as I don't have to deal with std::pair of the two values.

Then I could add in the Exchange to the ConflationKey for the Quote and now I could get the one most worrisome feature into the code - that of proper conflation of the Quote messages. But I didn't stop there, I was on a roll.

I ended up cleaning out a lot of the code based on the non-embedded values in the ConflationKey, and with a little extra work, packaged up the OPRA Quotes into multiple messages allowing for the additional BBO appendages. It's nasty, but rather than have an Option NBBO Engine, we can use the data coming from OPRA for the BBO and send out messages for it. It's not 100% accurate as the sizes aren't right, but it's faster and cheaper than putting up the hardware for the Option NBBO server.

In the end, I had made significant progress in some very nasty problems with the ticker plant, and all because I was forced to look at the SecurityID bits because I didn't have "real work" to do. I have a new respect for the subtle ways of fate and how I really need to stay far more open and accepting of what appears to me to be set-backs, as they are often the greatest of opportunities in disguise.

Great lesson to learn.

Great day.

This afternoon I was working on my unsigned 128-bit integer class, UINT128, and realized that I needed to be able to swap byte orders in place, and so I wrote a little method on the class:

  #include <byteswap.h>
 
 
  void UINT128::byteSwap()
  {
    mBlocks[0] = bswap_64(mBlocks[0]);
    mBlocks[1] = bswap_64(mBlocks[1]);
  }

and the thing I love about GCC today is that bswap_64() is a built-in function. Could it be any better? I did a little googling on byte swapping in GCC, and sure enough, there was exactly what I needed. Previously, I had been individually assigning bytes, and while it allowed for optimization, I'm guessing that this is way faster.

I need to remember to google first. It's almost always worth it.

Well, this morning I got another little treat - iTerm2 Alpha 15 is out and has a rather impressive set of features (scraped from the Sparkle update dialog):

Alpha 15
This release has one big feature: split panes! You can divide a tab into rectangular regions, each showing a different session. Also, you can now save your window arrangement and have it automatically restored when you start iTerm2.

Big new features
- Horizontal and vertical split panes. Use Cmd-D to split vertically, Cmd-Shift-D to split horizontally, and Cmd-Opt-W to close a pane. Navigate panes with Cmd-Opt-Arrow keys (but note that you may have conflicting key mappings in your bookmark, as this had been a default setting).
- Save your window arrangement including window positions, size, tabs, and split panes. You can open new windows with the saved arrangement with a keypress or use a new preference ("Open saved window arrangement at startup") to do it automatically.

Small Enhancements
- Make window title and tab labels more configurable so you can remove window number and job name.
- Add global keybindings for cmd-arrow, cmd-pageup/pagedown, cmd-home/end to scroll.
- Make opt-arrowkey send proper escape codes.
- Preferences and Bookmarks windows can now be closed with Esc key.
- When the "hotkey" is pressed, prevent the running app from getting the keypress.
- Made Find With Selection (cmd-E) behave like a normal cocoa app.
- Allow the OS to pick the initial window position if smart placement is off and no saved location exists.
- Improved wording of context menus.
- Fixed context menu's "send email" feature.

Bugs fixed
- Fix arrow key behavior in paste history and autocomplete windows (bug 407)
- Various memory leaks fixed.
- Improved behavior of full-screen tab bar display.
- Fixed applescript bug with setting window size before adding session.

While I'm not a huge fan of the split screens, I have a lot of friends that swear by them - and interestingly enough, they are emacs users. Hmmm... coincidence? Maybe not.

In any case, it's a nice update because I can get rid of the instance number on the window title and just get what I want. Also, the saved window positions is as nice as Terminal.app, and I don't have to deal with the scroll bars any longer. Unfortunately, the window positions don't include the Space they were on. I'm really hoping that Apple fixes this soon as it's one of the most annoying things about the Terminal.app (after the mandatory scroll bars).