What's it all about, Alfie?

I've been given the opportunity to see if a developer in the Shop would be a better fit for the project I'm working on than the one he's currently on. Well... sure. I think to myself - why not? Well... the answer to that question is, of course, that adding this person to the group would slow me down to the point that I miss every deadline I've promised, and drive me to the brink of sanity.

I'm not sure what I'd call this personality, but it's an interesting mix. There's skill there to be sure. But getting that skill out is a very time-consuming operation. I'm not sure it's really worth it, as the work seems to be something I end up re-writing anyway, but it's based on a decent set of skills. Just bad ideas, I guess.

So I'm working with him to try and see if I can correct the problems - thinking they aren't that severe. After all, if they got here, the theory is they are valuable, right? So they are worth saving, right? At least those are the assumptions I'm working under.

So I'm asking him to have a look at some unit tests, and write a few more. This is how I like to introduce new developers to the codebase. It gives them an idea of how the code looks, how the tests are built, and how to compile and run the code on a very baby steps approach. It's been a pretty good approach for me so far.

For this guy it didn't seem to help a lot. I still had to do a lot of re-writing. More importantly, the designs, which should have been simple and small extensions to the existing objects, were instead complex, tangled messes of code. I don't think I've ever seen as bad an extension to a class as I've seen here. He just didn't even seem to try to look at the existing code and make his like it in any way.

But while this is all understandable, and I can re-write anything, the big surprise to me is the time it takes to talk to this person. It's measured in hours. Really. It's not like this is someone that can take an idea or two and run with it. Nope. It's got to be argued over and over again - and then, he'll do something about it. The code is bad, yes, but the time sink is really what pains me.

I'm throwing away large chunks of my day to this person, and I'm not sure any of it is doing any good. It's as if there's nothing being really transferred from me to him. He's not learning from the corrections I'm making to his code, and he's not learning from the talks we're having. I've had talks with people in his previous group here at The Shop, and they are tickled pink that he's with me now. They are guessing it saves them three hours a day to have this person off their team.

I'm being a good corporate citizen and doing what I'm asked, but I've already alerted my manager and the project lead that this person is going to double all my time estimates. I don't know what they are going to do, but until they tell me something, he's my problem and I need to come up with some way to at least keep the conversations to a minimum.

Yikes. I had no idea what I was getting into.

This afternoon I had the most horrid debugging session in recent memory. It was really just that bad. The problem was that the code the new guy was writing was giving segmentation faults when my code wasn't. Additionally, there seemed to be a serious data problem in the information we were getting from another group's code. Not fun.

The first problem I ran into was the formatting of a timestamp into a human-readable string. The original code was:

  std::string formatTimestamp( uint64_t aTime )
  {
    char   buf[32];
    // see if it's w.r.t. epoch or today - we'll format accordingly
    if (aTime > 86400000) {
      // this is since epoch
      time_t  msec = aTime;
      struct tm   when;
      localtime(&msec, &when);
      // now make the msec since epoch for the broken out time
      msec = mktime(&when);
      // now let's make a pretty representation of those parts
      snprintf(buf, 31, "%04d-%02d-%02d %02d:%02d:%02d.%03d",
               when.tm_year, when.tm_mon, when.tm_mday,
               when.tm_hour, when.tm_min, when.tm_sec,
               (int)(aTime - msec));
    } else {
      // this is since midnight - let's break it down...
      uint32_t    t = aTime;
      uint8_t     hrs = t/3600000;
      t -= hrs*3600000;
      uint8_t     min = t/60000;
      t -= min*60000;
      uint8_t     sec = t/1000;
      t -= sec*1000;
      // now let's make a pretty representation of those parts
      snprintf(buf, 31, "%02d:%02d:%02d.%03d", hrs, min, sec, t);
    }
    // ...and return a nice std::string of it
    return std::string(buf);
  }

My problem typically is that I test some things, but not all edge cases. In this case, I hadn't really tested the "since epoch" code, for there's a few doozies in that section.

First, localtime_r() is seconds since epoch, not milliseconds, like I'm assuming. Duh. That means we need to fix that up:

  std::string formatTimestamp( uint64_t aTime )
  {
    char   buf[32];
    // see if it's w.r.t. epoch or today - we'll format accordingly
    if (aTime > 86400000) {
      // this is since epoch
      time_t  sec = aTime;
      struct tm   when;
      localtime(&sec, &when);
      // now make the sec since epoch for the broken out time
      sec = mktime(&when);
      // now let's make a pretty representation of those parts
      snprintf(buf, 31, "%04d-%02d-%02d %02d:%02d:%02d.%03d",
               when.tm_year, when.tm_mon, when.tm_mday,
               when.tm_hour, when.tm_min, when.tm_sec,
               (int)(aTime - sec*1000));
    } else {
      // this is since midnight - let's break it down...
      uint32_t    t = aTime;
      uint8_t     hrs = t/3600000;
      t -= hrs*3600000;
      uint8_t     min = t/60000;
      t -= min*60000;
      uint8_t     sec = t/1000;
      t -= sec*1000;
      // now let's make a pretty representation of those parts
      snprintf(buf, 31, "%02d:%02d:%02d.%03d", hrs, min, sec, t);
    }
    // ...and return a nice std::string of it
    return std::string(buf);
  }

The next one is about the broken out time struct. The year is offset by 1900, and the month offset by 0, so I had to fix those up:

  std::string formatTimestamp( uint64_t aTime )
  {
    char   buf[32];
    // see if it's w.r.t. epoch or today - we'll format accordingly
    if (aTime > 86400000) {
      // this is since epoch
      time_t  sec = aTime;
      struct tm   when;
      localtime(&sec, &when);
      // now make the sec since epoch for the broken out time
      sec = mktime(&when);
      // now let's make a pretty representation of those parts
      snprintf(buf, 31, "%04d-%02d-%02d %02d:%02d:%02d.%03d",
               when.tm_year+1900, when.tm_mon+1, when.tm_mday,
               when.tm_hour, when.tm_min, when.tm_sec,
               (int)(aTime - sec*1000));
    } else {
      // this is since midnight - let's break it down...
      uint32_t    t = aTime;
      uint8_t     hrs = t/3600000;
      t -= hrs*3600000;
      uint8_t     min = t/60000;
      t -= min*60000;
      uint8_t     sec = t/1000;
      t -= sec*1000;
      // now let's make a pretty representation of those parts
      snprintf(buf, 31, "%02d:%02d:%02d.%03d", hrs, min, sec, t);
    }
    // ...and return a nice std::string of it
    return std::string(buf);
  }

Here, now, we finally have something that's right. The second big issue was with the encoding of integers with Google's varint system. In general, it's vey effective, but I was seeing the timestamps coming from a service as twice what they should be. Made no sense whatsoever - until I saw the code.

In order to efficiently compress signed integers, Google came up with the idea of zig-zag encoding. Basically, you sort the integers by their absolute value, negatives first, and then pick the row in the sort as the coded value. Something like this:

From this table, it's easy to see that if you expect to see an encoded unsigned int, as I did, the signed int can look to be twice the size! When I saw they were using signed values where I was expecting an unsigned value, I knew it was the zig-zag encoding and fixed that right up.

These should have taken me about 30 mins to find. As it was, they took hours. Why? The developer that's new to the team wanted to "watch". Now by "watch" they meant "watch a little, but talk a lot", and there's the rub. When I'm debugging, I need quiet. I need to see what's really going on, why the code isn't acting as I expect it, and what assumptions and ideas I had that were wrong. Having a Chatty Kathy sitting next to me was, to say the least, unhelpful. To be accurate, a hinderance.

If I felt they learned anything in the process, I'd settle for that as a positive outcome. But I'm not at all sure they learned a thing. I think they just don't like working alone, and so saw this as an opportunity to have a "working buddy". Well... today is the last day for that. Never again.

Gotta nip this in the bud.

This afternoon I've spent a lot of time cleaning up someone else's code. This person is new to the group, and I've been asked to see if he will fit into the project. I do have my doubts...

I had him put in a few tests and write a few methods on an existing class. This wasn't meant to be hard, it was meant to be easy, but it turned out to be something that I needed to be more explicit about.

The code was an interesting combination of decent STL C++ code, but completely optimistic, and no comments to speak of. Those comments that were there weren't even complete sentences. While I'm not an english major, I think the comments are the only real help that an author gives to the person trying to figure out what they did. Many think of this as a weakness, but I know better. I'm often the one that reads what I wrote 6 to 9 months after the fact, and I'd like to cut my future self a little slack.

So I had to not only re-arrange things, and comment them, I also had to remove the optimism of the original code, and in many cases, improve the speed as the use of some STL ideas is fine if speed isn't an issue, but in a ticker plant, that's just simply not the case.

What I have come to realize is that the new guy is not all that great a coder. With all the examples they had to work with, they wrote code that was completely out of context with the rest of the class. This shows a real lack of awareness to me. We'll see how he progresses in the coming days.

This morning I've been cleaning up my ticker plant client - making the ZeroMQ listener of messages smart enough so that when the last listener is removed, the ZMQ socket is torn down thus shutting off the reliable multicast stream of messages that no one is listening for any more. It's a bunch of little details like that - things that will make the code nicer, but not really any less functional for the initial testing and rollout.

It's the "second pass" on the code - adding all those touches you'd expect to see, but aren't required in the first workable cut of the code.

I'm relieved to know that I have all the conflation queues and good byte-level queues in all the necessary endpoints in my communications library. The ZMQ receiver and transmitter have them, as to the TCP client and client proxy. This is pretty much all that's needed, and adding them where they aren't necessary is loading the system with threads that are processing data as fast as possible. It gets inefficient.

But for now, I have them in and things are looking and running well. One step closer to a full-up test. Now I need to start pulling these together into more complex components and then testing box-to-box performance. It's getting there.

Today I wanted to just build a conflation queue and integrate it into a few of my endpoints in the message flow so that slow consumers, or very fast producers, don't overwhelm the system. It's a simple design: a queue with a map that are linked so that the map can quickly tell the insert routine if the message already exists in the queue, and so just the contents are replaced. It's standard stuff, but there are always a few tricks.

In my case, I wanted to have the uniqueness of the messages dictated by the type, and unsigned 8-bit integer, and a conflation key, a 64-bit unsigned integer that is returned by each message indicating the uniqueness of the message within that family of message. I decided to use a simple STL std::pair as it could be used in the std::deque as well as the std::map as a key.

When I got into the code, though, I realized that there were plenty of places I wasn't doing any kind of buffering - specifically, the TCP endpoints. So I had to go into those components and put in a simple byte-level buffer with an additional thread for de-spooling. That took time. Then I saw that there were a few more that needed some work, and pretty soon the entire morning was gone.

Finally, though, I got all the components working and testing just fine - which wasn't as easy as it used to be with additional threads running around. In fact, it was a mess of segmentation faults until I changed the way I was terminating the unspooling threads. But I got it all going.

The afternoon was devoted to getting the conflation queue working and tested out. It's nice, and it should be pretty fast, but I'll have to wait and see how it tests out for performance when it's getting hit with some of these exchange feeds. But hey, that's the point - we need to be able to decouple the producers from consumers with this conflation queue.

Tomorrow is going to be putting these into some components.

This morning I was able to finish up the message cache that I started work on yesterday afternoon. It's been pretty straight forward right up until I got to the part about being thread-safe and fast at the same time. This meant lockless data structures, and that meant the compare and swap operations.

Since I'm only having one producer and many consumers, it's a thought that maybe I didn't have to worry too much about this. Well... that's a mistake. I do. But thankfully, there's only one place I need to worry about the complexities of this map - in the "setter" of the individual value.

The code I ended up with looks like this:

  boost::unordered_map<uint64_t, Message *>   mCache;
 
  Message   *oldMsg = mCache[key];
  while (!__sync_bool_compare_and_swap(&(mCache[key]), oldMsg, newMsg)) {
    oldMsg = mCache[key];
  }
  // now handle the old message
  if (oldMsg != NULL) {
    delete oldMsg;
    oldMsg = NULL;
  }

where the value newMsg is the value to place in the map at the key of key.

Sure, this is greatly simplified, but the single core issue is there - we have to check to see if the value has been changed, and if it has, we need to get the new value, and then try and set our value again. The idea is that it's thread-safe, not that it's the right element in the map, but that it's the one that's put there last.

This is working great for me and while I haven't had the chance to run tests against it, it's as fast as I'm going to be able to get regardless if it's fast enough. I think it will be. I've used the boost::unordered_map which is supposed to be faster than the STL std::map which uses a read/black tree for the key space, and since my key space is a uint64_t, it's pretty easy to hash that guy - it's just the value.

I'm going to have to test this guy with the real data feeds, but I think I'm going to be OK. I feel good about it.

The last half of today I spent trying to figure out a really good way to incorporate a good, fast, caching scheme for these messages that are the heart of my ticker plant. The problem is that I know I'm going to be hammering the messages as fast as I can possibly take them, so a cache that locks is just right out. I wanted to make it simple as well - something parametric so it'd be easy to turn on or off and see what the difference in timing and throughput was.

The problem was, it wasn't really obvious how to do this with the given design I had.

Failed Attempts

I thought that it'd be best to put this into an existing class in the design - as low a level as possible. I was thinking maybe the component that handled simple message distribution to a list of registered listeners. It'd be nice there in that any component that "pushed" messages would be able to take advantage of this by simply "turning it on".

But there were a lot of conceptual problems. First, the messaging API used references for the messages, which is what I wanted to use, but that meant that in order to make the caching fit in without altering the API, I'd have to be making copies of the messages. That's not a good plan. Nothing good is going to come from copying a hundred thousand messages a second. That's a recipe for poor performance.

Next, if we changed the API to use pointers, and thought of the send() method as a "sink" of the message, then we'd have a nice, transparent, caching scheme, but the problem is that the messaging system relies on calling send() several times - once for each listener. This means I can't have the one method "eating" the instance as there'd be nothing left for the other listeners.

I messed around with this for a while, each time coming to the conclusion that the design I had wasn't really workable. I finally backed off and tried to think of the problem in vastly different terms.

What came to me was very simple, very clean, and in the end, far better than I'd ever have been able to do in the previous approaches.

The Component Approach

I began to think of the cache as something of a component as opposed to a capability of the objects in the design. I started to look at the cache as something that I could put as an ivar in the few components that needed it, and make it self-contained and easy to integrate and use, and all of a sudden things started to really look up.

Now I didn't have to worry about the passing of references and how many times something is called. In anything that deals with messages, it's possible to have a cache, and that cache will "eat" messages created on the heap. It will either delete them right away (if it's inactive), or it'll save the latest, and delete the older version, so as to keep the most recent in memory and yet at the same time handle all the housekeeping for the memory management of the created messages.

Most of the components that need this caching are things that create messages. Maybe it's coming in from a serialized data stream. In that case, the created message needs to be passed to all the registered listeners, and then deleted - or cached. It's similar for different "translation components", but the model fits very well. So I started coding it up.

I didn't get finished, but it's well on it's way, and it appears to have beaten all the problems that were really dogging me with the other approaches. I know there's more to deal with - like the "fast" part, but I'll deal with that in the morning.

This morning I spent the entire morning messing with getting a new machine at The Shop working for me. I'd spent a little bit of last Friday making sure all the software I needed on it was installed, but that didn't prepare me for what I was to experience first hand with Windows 7.

Normally, I'm a live and let live kind of guy for work machines. It's what the company wanted to get, it's what tools the company has supplied, and so I make due with it because that's really the only possible outcome. If I dislike it that much, then it's my choice to leave and look for a place that provides the tools I like. I've turned down a lot of positions in the last decade for Windows development jobs - I just don't want to do it.

So today when I got the new machine - and by 'new' I mean 'different' with Windows 7 installed, I set about making it workable with the software and set-up I needed. A few things I couldn't have done remotely - like the virtual windows and such, but no matter, I had penciled in the morning for just such activities.

So I got the box, and while I'm not a great fan of the look and feel, I can see it's positives, and understand why they did it. But on the hardware that was provided, it was quite literally, a joke.

I am required to use NX Machine to get to my linux servers, and when I did that, I was stunned to see an 80-line Vim session take 2 seconds to redraw. The full screen was on the order of 10 sec. It was amazing... and totally unusable.

I tried all the things I could find on their web site - nothing to address the horrible slowness of the redrawing. Then someone pointed out a setting to skip DirectDraw on the refreshing. It seems it's the "budget" video card and Windows 7 that are at odds here. Had my machine had an accelerated video card, I'd have been fine. But with the hardware I had, I was forced to change the refresh method.

Thankfully, that worked. A little later the head of desktop support stopped by to ask how it was going. I had shared my problems with him earlier, and I was glad to see he was following up. We talked, and he pointed out that he turns off all the "new" Windows 7 features to get his box working quickly again.

Amazing... I go back to a Windows NT look-n-feel to get Windows 7 working OK. I'd have been better off sticking with XP! Really. The cost, and the time, and the now degraded UI... it's a joke. An honest to goodness joke. I don't expect the company to spring for all new hardware because of Windows 7, but they should have had a far more graceful degradation of performance so I could look like XP, and have the Windows 7 engine, as opposed to dropping all the way to NT.

But hey... I'm sure there's tons of people saying "Buy a $100 video card!", and they'd be right. For me. But for the hundreds of other machines? Now we're talking real money. Not so easy to say "Buy $30,000 in video cards so Windows 7 is nice!" But that's really what Microsoft is expecting you to do - invest in more hardware to get the user experience they ship out of the box.

This morning I finally finished up the client library I've been working on for a while. I finished up yesterday with it functional, but not the way I wanted it. I spent a little time this morning cleaning up the code - adding comments here and there, and clearing out the logging using only during the debugging phase.

Nothing very exciting so far...

Then I was beginning to think that I might be able to get the best of both worlds with the existing design. One of the problems with my one request - one socket is that we can possibly end up with a ton of sockets in use. We can also be spending a lot of time creating them, only to tear them down right away. Not as efficient as it could be.

Yet I didn't want to have the one bad apple problem, so I think I've got a nice idea that I'm going to be putting into the codebase this morning - pooling of sockets. It's a simple thing - have a list of "unused" socket connections, and when a request comes in, grab one from the list, or if there are non there, create one. Then, as you use it, great. Everything is isolated. Then, when you're done with it, return it to the pool, and it'll get re-used the next time someone needs one.

Very simple, but the socket stays open, all the resources stay allocated, and in general, we have the best of both worlds. Very nice. That's going to be my morning - pretty excited about it.