What's it all about, Alfie?

Today I spent most of the day adding in the ability to get and set individual instrument values from the Broker API for the Greek Engine. This is really all about the end of day processing where we need to know the last trade prices of all the instruments (that's the "get"), and then match that up with prices from Bloomberg, Thompson, and other sources, to make a unified "close", or mark, for the day. Then we need to set all these values back into the engine (that's the "set") so that they have the proper, verified, close marks for the day.

It wasn't all that hard, I just needed to expand a little on a few things I've already written for the Greek Engine. I had the bulk of the setter code already done, I just needed to make it a little cleaner and make sure it didn't freeze out any instruments from receiving further updates. The getter was a little more difficult, as it is meant to be a map-of-maps, and so I need to be able to build it up a family at a time.

Again, not horrible, but it was a solid day of writing and then updating test apps to test this functionality and then fixing any little issues with the code I'd written. It was a nice thing to finish up the week on a success. I need a nice weekend to recover from the beating I'm taking at work.

I haven't posted any updates to Google Chrome dev for quite a while, but I've been updating every chance I get. The most recent cut is 15.0.874.15 and has the latest V8 javascript engine, as well as lots of fixes for Mac OS X Lion (10.7). It's been at version 15 for quite a while, and they are starting to issue updates to the "fourth octet", so I'm guessing that it's getting close to time to cut version 16 and make 15 the 'beta' channel and keep on going.

This morning was a lot of work that turned out to be something almost funny in retrospect. I had just completed the conversion of the data source from the old MS SQL Server databases being accessed through SQLAPI++/iODBC/FreeTDS to using PostgreSQL using just the SQLAPI++/PostgreSQL libraries. Things were running, but the greek values were all wrong. I mean not even close to right. So clearly, I had messed up on one of the data import statements or values - somewhere. So I had to track it down.

The problem was that there were a lot of values coming out of the database tables, and I needed to very each one to make sure there wasn't an extra factor of 100 or 10000. So I started doing a lot of checking. Pretty soon, it appeared that there was nothing wrong with the data. Not a single thing. Very odd.

So I backed off and started looking a little wider. Maybe it was the set-up outside the calculations themselves. So I looked at the Holiday Calendar. In the previous holiday calendar SQL call, I received the list of holidays. When I looked at what I was putting in now, I realized that I was looking at a calendar - not a Holiday Calendar. Basically, I was telling the FE Model that every day was a holiday. Very bad. So I changed the SQL on the calendar to only pick out the market holidays and BINGO! The numbers fell right back into place. Very nice to see.

But after that I had to giggle… Every day a holiday. Kinda funny.

The next problem that was sitting on my "TODO" list was a core dump that we got from having a web client hit us as hard as possible - pretty hard, in fact. The problem was pretty simple, and as soon as I saw the back trace, I knew what the problem was: concurrent modification of an iterating container.

Basically, I have been using the TBB concurrent_hash_map to hold things, and in this case, it was really back-firing on me. The TBB containers are thread-safe, which is great, but when you're iterating them, it's possible to have one thread think it's got something, and another thread to delete it out from under them. No fun, and almost no way to clear it up with these data structures. What you really need, and what I went to, was a nice read/write spinlock that we can then ensure isn't getting modified during an iterating pass.

With this in place, we don't have to worry about the excessive load of the exclusive lock, we can typically use the read level on the mutex, saving the write for those times we add and remove pairs from the hash map. Then we can back off and use something like a boost::unordered_map for the container and be better off all the way around.

At the end of the day I started working on the next feature I was adding - the ability to pull out of a running Engine a provided list of values for a group (or all) instruments in the engine. For example, a way to say Give me all the last trade prices and sizes for everything in this profile. I'm going to need it, so I got started on it knowing I wouldn't be able to come close to finishing it today.

Today I'm trying to finish up on the migration away from hitting the legacy databases and towards the replicated, local, read-only, PostgreSQL database. It's been nice, but from time to time, it's required data that my co-worker is far better equipped to add than I am, so I've had a few spells of just sitting around. So it goes. Can't fix everything.

I'm hoping to get this done today, but who knows… there may be a few sources of data that are really difficult to get. We'll have to wait and see.

Today has been kinda fun, my Greek Engine reads instrument static data from some legacy databases, and then merges these many sources into one useful source of data. It takes about a minute, which isn't horrible, but it's an expense in getting the logic right, and adding new features based on this data is a serious pain. Thankfully, my co-worker had the idea of making a "data service" that's really just a replicated PostgreSQL 9 database where the master is written to by update jobs, and the data is replicated to read-only slaves on each server box for local access.

It allows us to do all this convoluted logic in one place and let it be shared by all the groups and applications that need it. It doesn't have to be normalized, and it's probably better if it isn't - the queries will be simpler and faster without all the foreign keys.

So he's been spending time getting this master database up and populated and updating with data from a lot of different sources, and today I've been working on integrating this into my Greeks Engine. It's a lot of fun because PostgreSQL 9 is fast, simple, and the support tools are just amazing. It's been a favorite of mine for many years, and the new version, 9.1, is no exception.

Replication is fast, easy, and efficient. It's complete and so far as we can tell now, fool-proof. Wonderful news. The only wrinkle is the development tools - on my linux box, they have both 8.4 and 9.0 and there was a little confusion this morning about what was on and what wasn't. I needed to get that cleared up before anything I wrote worked.

Updating the code was pretty simple - and greatly simplified. No longer do I have to have 50 line SQL queries to get the data. It's there and it works. No more ODBC drivers, either. Holy Cow! What a relief.

I need to do a little more work in the morning, but it's a great start.

So we finally found the problem with the deadlock in the code… and it wasn't a deadlock at all. It was an infinite loop. Holy Cow! This was without a doubt the hardest bug I've ever had to find. And it was all my fault. Silly typo, and I spent days trying to find it. Thankfully, my co-worker was about to ask a few questions and get us on the right track, and we were able to find it, but it was nasty to find, and was a single character long.

Incredible.

In general, when doing atomic operations, when setting a value, you need to have a loop where you look at the existing value, try to CAS in the new one, and if you fail, try it all again. This typically looks like this:

  uint32_t      now = mValue;
  while (!__sync_bool_compare_and_swap(&mValue, now, aValue) {
    now = mValue;
  }

where aValue is the new value of the ivar mValue. And this will work. But if you have a typo in the code, say like this:

  uint32_t      now = mValue;
  while (!__sync_bool_compare_and_swap(&mValue, now, aValue) {
    now = aValue;
  }

then the first failure will put you in an infinite loop until someone happens to set it with the new value you're trying to set. It was a disaster.

When I changed the a to m, we were pulling in the correct values, and things were working just fine. Also, single-threaded tests that I had would not have seen this as they would not have failed in the first place. It requires two threads hitting the same atomic at the same time with different values. Amazing.

There… I feel worlds better because we found it. I know the code is better, and I know why. I know we won't have these same issues, and I know why. What a relief.

For a little while now - starting really in the quiet times, I've been using Intel's TBB Library, and I have to say, it's pretty impressive. There are several things in the library, but the biggies of note are the concurrent vector, the concurrent hash map, and the read/write spinlock. I've used these in my work over the last month or two with great success.

When I initially started looking at the TBB, I seem to remember building and using this was a pain. But maybe it's changed, or maybe it's a bad first impression. In either case, a co-worker decided to get the code in-house and installed on a few boxes, and it's as valuable for it's target as boost is for it's. There are a lot of things in this library, but with 4.0 announced this past week, they've added the Memory Pool (we're using TCMalloc from Google, but have heard there are better out there), full GCC Atomics Support (as opposed to the gcc-specific semantics), and new unordered sets and priority queue. Very nice stuff indeed!

Last evening, I took out a boost::unordered_map and boost::detail::spinlock and replaced them with a single tab::concurrent_hash_map. The difference in code wasn't all that great, but I have to say their Accessor is a bit odd, and possibly far easier replaced with an iterator of some sort. However, I can understand why they keep the two different.

For example, when I wanted to add a key/value to a boost::unordered_map, I'd simply do something like this:

  boost::unordered_map<std::string, uint32_t>    names;
  …
  names["joe"] = 32;

but that's not possible in TBB's concurrent_hash_map. Instead, you need to do something like the following:

  typedef tab::concurrent_hash_map<std::string, uint32_t> ages_t;
  ages_t      names;
  …
  ages_t::accessor  a;
  if (!names.find(a, "joe")) {
    names.insert(a, "joe");
  }
  a->second = 32;

The find() is used to set the ages_t::accessor to the right pair, and lock it for updating. If it returns false, then there is no pair, and we need to insert() one in the map. Again, this will lock that pair for updating. Either way, the accessor is now pointing to the pair, and has it locked. From here, it's just a matter of setting the 'value' part of the pair.

Again, not really simple, but not terribly hard, either. The advantage is amazing. Inserts are not as fast, obviously, but the lookups are very quick and we don't have to worry about using a read/write lock on this map, or a simple mutex, to keep things from getting scrambled.

There are lots of decent tools out there, but a lot more really crummy ones. I'm glad we found TBB and boost. They are going to make things a lot easier to work with.

Today I'm dealing with a lot of issues regarding performance and threading. It's non-trivial, even for as much of it as I've done, as it's all a balance of safety, speed, and need. It's just plain not easy, but hey… if it were, I wouldn't get paid to do it.

Today I ran across an interesting issue… found this code:

  const calc::results_t & Option::getResults() const
  {
    using namespace boost::detail;
    spinlock::scoped_lock  lock((spinlock &)mResultsMutex);
    return mResults;
  }

Now I know that the intended purpose of this code was to make sure that no one altered the mResults value while a copy was made on the calling stack and returned to the caller. But that's not at all what's happening, is it?

Have a look at the return type - a reference. That means that we're passing back a reference to the mResults value, and not a copy at all. This means that the scoped lock is totally useless in this context. Better to just remove it as there are places in the code that expect to get a reference:

  const calc::results_t & Option::getResults() const
  {
    return mResults;
  }

This is far cleaner, and just as "protected" from multiple thread access - which is to say it isn't.

I'm cleaning up things and trying to track down why I'm getting a deadlock, but it's not at all simple. It's not as easy as looking at the code - if it were, I'd have solved it, but some things are just too well hidden to pop out on a simple inspection. Sometimes, it takes a smoking gun to really point out the problem you have.

So I'm looking for the smoking gun. Hope I find it soon.

I'm typically a huge fan of the boost libraries. They have saved me a ton of time in the last year, and they are about as rock-solid as I've seen public domain software be. But today I've been fighting a threading issue that's really been vexing. I'm not 100% positive this afternoon, but the evidence is certainly pointing to a problem with the use of the boost::detail::spinlock::scoped_lock and using it in very tight loop situations like the following:

  size_t MessageSource::getCountOfListeners()
  {
    boost::detail::spinlock::scoped_lock     lock(mMutex);
    return mListeners.size();
  }

where mListeners is just a boost::unordered_set containing pointers of some kind. The problem seems to be coming from the use of this as an argument in a function like the following:

  cLog.info("[process] %d kids", getCountOfListeners());

Now I've clearly simplified things a bit, but I'm guessing that I'd have had a lot better luck had I done the following:

  size_t MessageSource::getCountOfListeners()
  {
    mMutex.lock();
    size_t   sz = mListeners.size();
    mMutex.unlock();
    return sz;
  }

where I'm explicitly locking, getting the value, and unlocking the mutex. But there were a lot of places in the code where this was occurring, and rather than go that route, I chose to try the Intel TBB concurrent_hash_map where the key is the pointer and the value is just a dummy uint8_t. This compiles and runs just fine, with the added benefit that I was able to remove all the locks:

  size_t MessageSource::getCountOfListeners()
  {
    return mListeners.size();
  }

In this implementation, the hash map handles the size for me, and I don't have to worry about the locking or the scoped lock's lifetime. I believe this is going to make a huge difference tomorrow, but we'll have to wait and see. It's certainly in the area of the problem.

A few weeks ago, yeah, I know, I've been busy, I got Lion (10.7.1) from the Mac App Store and installed it on my main MacBook Pro. The upgrade took longer than I really expected - the downloading was not fast at all. But I will say, it was smooth. Very smooth.

The big issues I found with Mac OS X Lion is that Colloquy 2.3 wasn't really working properly. Thankfully, all the other apps that I depend on day-to-day were working fine. I still have the problem that Twitterrific does not work when the video card is changed and it tries to "pop up", but hey, it's a small price to pay, and it's not just Twitterrific - it's all the pop-up Twitter clients I've tried. Kind of disappointing.

Anyway, Colloquy had a new build that does support Lion, and it's available from the Colloquy Downloads Folder. Go there, get the latest, or at least 2.4, and you're in business. Kind of surprised that they aren't more responsive as Lion has been out there for a while, but at least they have something that works for me.

Other than that, Lion is working just fine and the look and feel of the OS is very nice. I especially like the vanishing scroll bars. On Terminal.app, I've been asking for those for ages, and I now have them. Good. Fantastic.