What's it all about, Alfie?

Without question, the best multi-protocol IM client for Mac OS X has got to be Adium, and today they tweeted that a few days ago, they released a release candidate for 1.5. This is just one of those amazing programs for the Mac that Just Works and while it's not on the kind of release schedule that, say, Google Chrome is, it's stable, works well, and I've never had a problem with it. Just amazing.

I've been running the different 1.5 betas for a while, but it'll be nice to see official 1.5 support as it means that it's mainline. Nice to see.

This morning I saw that Google Chrome dev 18.0.1010.0 was out, and the release notes have a few nice things like PDF rotation and a few Mac-specific fixes as well. Good to see these user-experience additions and fixes, but I'm still waiting to see if Google tries to put in something that again opens up the wild development in browser technology again. Can't imagine what it'd be, but that's part of the fun!

A few days ago, I wrote a few static methods on a class that could be used in the testing of changed values. They aren't anything special, but it's something that I know from experience you need to do right, or it's going to be painful, and in order to do it right, you really need to handle all the edge conditions. Normally, this would be in-line code for another developer, but I've learned that it really clutters your code to do it right. Hence, the static methods.

My first cut was:

  double Instrument::pctChange( double oldValue, double newValue )
  {
    double     chg = NAN;
    if (!::isnan(oldValue) && !::isnan(newValue)) {
      if (fabs(oldValue) < 1.0e-6) {
        chg = 1.0e6;
      } else {
        chg = (newValue - oldValue)*100.0/oldValue;
      }
    }
    return chg;
  }
 
 
  double Instrument::pctChange( int64_t oldValue, int64_t newValue )
  {
    double     chg = NAN;
    if (!::isnan(oldValue) && !::isnan(newValue)) {
      if (oldValue == 0) {
        chg = 1.0e6;
      } else {
        chg = (newValue - oldValue)*100.0/oldValue;
      }
    }
    return chg;
  }

Having one that takes doubles and another that takes int64_t is nice in that we store the floating point numbers as integers internally to make math faster and transmission far faster.

I then proceeded to use this in the test of whether or not on instrument needed to be calculated based on the changes in some values of that instrument. The code was pretty simple, but it compounded problems int he above code even more:

  double   spotChg = pctChg(res.spot, getSpotAsInt());
  …
  if ((maxChg > 0.0) &&
      !::isnan(spotChg) && (spotChg < maxChg)) {
    …
  }

It was pretty easy to see, when I got word that this code wasn't working, that the problem was in the loss of the sign in the test. I needed the absolute value of the percent change in the test, not just the value itself. All negative changes were being discarded because they passed this test. Not good.

So I decided to incorporate the absolute value into the method, and at the same time, fix up a few problems I saw with the methods themselves:

  double Instrument::pctChange( double oldValue, double newValue )
  {
    double     chg = NAN;
    if (!::isnan(oldValue) && !::isnan(newValue)) {
      if (fabs(newValue - oldValue) < 1.0e-8) {
        chg = 0.0;
      } else if (fabs(oldValue) < 1.0e-6) {
        chg = (newValue > 0.0 ? 1.0e6 : -1.0e6);
      } else {
        chg = (newValue - oldValue)*100.0/oldValue;
      }
    }
    return chg;
  }
 
 
  double Instrument::pctChange( int64_t oldValue, int64_t newValue )
  {
    double     chg = NAN;
    if (!::isnan(oldValue) && !::isnan(newValue)) {
      if (newValue == oldValue) {
        chg = 0.0;
      } else if (oldValue == 0) {
        chg = (newValue > 0 ? 1.0e6 : -1.0e6);
      } else {
        chg = (newValue - oldValue)*100.0/oldValue;
      }
    }
    return chg;
  }
 
 
  double Instrument::absPctChange( double oldValue, double newValue )
  {
    return fabs(pctChg(oldValue, newValue));
  }
 
 
  double Instrument::absPctChange( int64_t oldValue, int64_t newValue )
  {
    return fabs(pctChg(oldValue, newValue));
  }

and then in my code I simply used the correct form of the method:

  double   spotChg = absPctChg(res.spot, getSpotAsInt());
  …
  if ((maxChg > 0.0) &&
      !::isnan(spotChg) && (spotChg < maxChg)) {
    …
  }

and we should be good to go.

The method now properly detects no change - regardless of the old value being zero, and the sign of the very large change is in accordance with the sign on the new value. Also, the absolute value being backed into the other methods makes it very easy not to forget this going forward.

For the last few days I've been struggling with a really nasty bug. Something that seemingly defies explanation based on the code and lack of error messages. It's been a real pain in the rump. But today I finally cracked it, and it was really fun to discover the subtle bug that had me baffled for a few days.

It all starts with timestamps. The basic idea is that I wanted a simple 64-bit unsigned integer to be a timestamp - and it seemed reasonable to pick microseconds since epoch. It all fits in the uint64_t, and it's all the resolution I'm going to need for the time being. The code to get it quickly from the system is pretty clean:

  uint64_t usecSinceEpoch()
  {
    struct timespec tv;
    clock_gettime(CLOCK_REALTIME, &tv);
    return (tv.tv_sec * 1000000 + tv.tv_nsec/1000);
  }

With this, I can get a nice, simple timestamp for comparisons. I can format it nicely with a little code like this:

  std::string formatUSecSinceMidnight( uint64_t aTime )
  {
    char   buf[64];
    // see if it's w.r.t. epoch or today - we'll format accordingly
    if (aTime > 84600000000) {
      // this is since epoch
      time_t  sec = aTime/1000000L;
      struct ™   when;
      localtime_r(&sec, &when);
      // now  make the sec since epoch for the broken out time
      sec = mktime(&when);
      // now let's make a pretty representation of those parts
      snprintf(buf, 63, "%04d-%02d-%02d %02d:%02d:%02d.%06d",
               when.tm_year+1900, when.tm_mon+1, when.tm_mday,
               when.tm_hour, when.tm_min, when.tm_sec,
               (uint32_t)(aTime - sec*1000000L));
    } else {
      // this is since midnight - let's break it down...
      uint64_t   t = aTime;
      uint8_t    hrs = t/3600000000L;
      t -= hrs*3600000000L;
      uint8_t    min = t/60000000L;
      t -= min*60000000L;
      uint8_t    sec = t/1000000L;
      t -= sec*1000000L;
      // now let's make a pretty representation of those parts
      snprintf(buf, 63, "%02d:%02d:%02d.%06d", hrs, min, sec, t);
    }
    // ...and return a nice std::string of it
    return std::string(but);
  }

And all is well and good. Timestamps made look right, they are fast, and all is good.

Almost.

There are times in the code where I want a fast way to get the date from a timestamp. I really don't care that it's with respect to epoch, I just want to know what the date is so that I can group things together based on the date. What seemed logical (to me) was to simply divide this timestamp by the number of microseconds in a day:

  uint32_t   day = aTime/86400000000;

and for the most part, it worked just fine. I could tell when I "crossed" into a new day, and everything was fine. Until 5:00pm came. Then it wasn't so nice.

Things stopped working. Certain caches were getting cleared, messages were generated but not delivered. Very odd behavior, given that just 10 min before everything was working just fine. To jump ahead a bit, the problem code looked a little like this:

  bool        skip = false;
  uint64_t    day = emsg.getCreationTimestamp()/86400000000L;
  uint64_t    eday = emsg.getTimestamp()/86400000000L;
  if ((day != eday) || (day < mCreationDay)) {
    // this is old data - skip it
    skip = true;
  } else {
    ...
  }

where the code is essentially saying "Hey, get the date the message arrived from the exchange and compare it to the date they sent it, and if they aren't the same, then drop it." Seems to be just fine. Until you take a little closer look at the value of those microseconds. See... the system call includes the timezone offset. So if I'm looking at a message that comes in at 5:00 pm (CST), that's 6:00 pm as we see it, and add in the CST offset (6 hrs), and we're in a new day!

All of a sudden, at 5:00 pm, all my new messages were being decoded, and delivered, but skipped as "bad data" simply because their "days" were different, as one had crossed into the next day. This didn't pop up until this because the localtime_r() call knowns to take out the same offset when it gets a value. My mistake was in trying to assume that I knew the value of the timestamp as opposed to dealing with it as a bit value and using the same methods to get the date.

The solution isn't too bad, but it is hard-coded:

  static uint64_t __cstOffset = 6*60*60*1000000L;
  bool            skip = false;
  uint64_t        day = (emsg.getCreationTimestamp() - __cstOffset)/86400000000L;
  uint64_t        eday = (emsg.getTimestamp() - __cstOffset)/86400000000L;
  if ((day != eday) || (day < mCreationDay)) {
    // this is old data - skip it
    skip = true;
  } else {
    ...
  }

I was so happy to find this bug as it was causing all kinds of problems in the code, and now that I've figured out the offset, it's not bad to use it as-is, it's just something I need to be mindful of when doing math on the timestamp value itself.

One of the things I wanted to get done this afternoon was to clean up the bash script I have that runs every night out of cron to clean up (delete) the data files that I can't afford to keep. These are the data files for me feed recorders, and I can afford to keep a few days, but not a lot. The old code I had was really pretty lame - and it scanned the directory looking for subdirectories and looking for the highest sorted name, and picking that one as the "latest".

I did this several times, each pass getting the next one, and the next one, and in the end, I had a list of directories to "keep". Very sloppy. Here's what I came up with this afternoon after just a little bit of fiddling:

  i=0
  for d in `ls | sort -r`
  do
    # skip anything that's not a directory
    if [ ! -d ${d} ]; then
      continue
    fi
    # track count of dirs to keep - and keep 4
    i=$(($i + 1))
    if [ $i -le 4 ]; then
      continue
    fi
    # whatever is left - delete… it's too old
    echo -n '.'
    rm -rf ${d}
  done

The beauty of this script is that I can easily change the number of days to keep - just change the '4' to a '5', and we have a whole (business) week. Very nice. Also, it's a lot more compact than the original approach.

This afternoon one of the big things I've needed to get done is to try to add to my UDP Exchange Recorders the ability to publish themselves as a service on our Broker system. A little background here, as I haven't written about this yet… we need to record the data feeds from the exchange feeds in order to see what happened in the middle of the day - in a post mortem sense. In order to do that as simply as possible, I wrote a UDP feed recorder. It's a lot like our UDP exchange feeds, but instead of decoding the datagrams, we simply write them out to a disk file in 10MB blocks.

Depending on the feed, these blocks are a minute or two, all the way up to hours of real-time data. We archive these for a few days, and then drop them as the disk space they consume is non-trivial. So… we wanted to be able to at this same data a little easier for the Greek Engines, so that a restart of a Greek Engine will not miss any ticks. By far, the easiest way to do this is to make it a service on out Broker system.

Thankfully, I've done this a lot, and it's a simple matter of creating a new class, filling in some code, tying in the initialization and periodic checks to make sure it's still connected, and we're ready for the meat of the request system. In all, about an hour of work, which isn't bad at all.

Now I need to figure out how these UDP exchange recorders are going to work with my archive server to get the latest data from the feeds in order to catch up the greek engine on restart.

Today I needed to add in an LRU cache to my archive server because we have some number of files loaded - each with about 10MB of messages from the exchange feeds. We don't want to have an infinite number, but we don't want to dump them prematurely. After all, they are immutable, so as long as we can keep them in memory, we should keep them in memory - just in case the next client asks for the data that happens to be in that chunk.

So I decided on a two-step memory management scheme - simple aging, and an LRU for times of very high load. The simple aging is that - simple. Every minute, or so, I'll look at all the chunks loaded into the cache, and see the last time it was accessed by a client. If

that time ever exceeds, say, an hour, we'll drop it. That just says that at the end of the day there's no reason to be a pig about the machine.

The LRU is pretty simple, in theory: every time we go to add a new chunk, we look to see how many we have, and if we have too many, we drop the one with the oldest access time, and continue until we get below the threshold so we can add the new one.

The question is about the determination of the oldest. You can make a list of keys, and each time the chunk is accessed, move it's key to the start of the list, and then take the last element on the list, but that's a lot of list accessing (adds, finds, deletes) when we might not be hitting the cache limit very often at all. So I took a simpler approach: just find the oldest one when I need to delete it.

  bool ArchiveSvr::dropOldestChunk_nl()
  {
    bool      error = false;
 
    // first, find the one to delete…
    uint64_t    when = 0;
    std::string what;
    for (cmap::iterator i = cache.begin(); i != cache.end(); ++i) {
      if ((when == 0) || (when > it->second.lastAccessTime)) {
        when = it->second.lastAccessTime;
        what = it->second.filename;
      }
    }
 
    // if we have anything to delete - do it
    if (!what.empty()) {
      cache.erase(what);
    } else {
      error = true;
      // we were supposed to delete SOMETHING…
    }
 
    return !errror;
  }

Now, in the code, I can simply do:

  while (cache.size() >= MAX_SIZE) {
    if (!dropOldestChunk_nl()) {
      error = true;
      cLog.error("can't delete what we need to… Bad!");
      break;
    }
  }

and we're good to go! Not bad at all. Simple, but effective.

Yes, today we jump the century mark! The Google Chrome dev release 18.0.1003.1 marks the first release of the 18.x.x.x branch, and answers the question I've had for a while when it came to the third number - would they go past 1000? And the answer is Certainly! They are engineers!

So we have several things in this release - a new version of the V8 javascript engine (3.8.4.1) as well as several under-the-cover items, better zooming, and several crashing bugs. All in, I'd say it's nice to see that they are still working on major issues. Good to know.

Today I needed to be able to read compressed files for a service I was writing. Sure, I could have shelled out and run gunzip on the file and then gzip-ed it up after reading it, but I wanted something that would allow me to read the gzipped file in-place and uncompress it into a simple std::string for processing.

Enter boost to the rescue.

This is one of the more difficult things to get right in boost… OK, I take that back, it's pretty easy by comparison to the serialization and ASIO, but it's not something that is simple to see from their docs. Also, some of the more obvious attempts to use the boost filtering iostreams yielded some pretty bad results. Still… as I kept with it, success emerged.

Here's what finally worked for me:

  #include <zlib.h>
  #include <boost/iostreams/filtering_stream.hpp>
  #include <boost/iostreams/filter/gzip.hpp>
  #include <boost/iostreams/copy.hpp>
 
 
  std::string     contents;
  std::ifstream   file(aFilename.c_str(),
                       std::ios_base::in | std::ios_base::binary);
  if (!file.is_open()) {
    error = true;
    cLog.error("can't open the file %s", aFilename.c_str());
  } else {
    using namespace boost::iostreams;
    // make the filter for the gzip with the right args…
    filtering_streambuf<input> gin;
    zlib_params   p;
    p.window_bits = 16 + MAX_WBITS;
    gin.push(zlib_decompressor?);
    gin.push(file);
    // now let's get a string stream for a destination
    std::stringstream  ss;
    // copy the source to the dest and trap errors
    try {
      copy(gin, ss);
      contents = ss.str();
      cLog.info("read in %u bytes from %s", contents.size(), aFilename.c_str());
    } catch (zlib_error & err) {
      error = true;
      cLog.error("decompression error on %s: %s (%d)",
                 aFilename.c_str(), err.what(), err.error());
    }
  }

What's the point of all this? Well, it turns out that boost isn't about the general decompression file streams, it's about pipelined filters and one of the filters is a gzip compressor and decompressor. It's more flexible, yes, but it's a little harder to do, and it ends up with an intermediate std::stringstream that we don't need. But in the end, this is only about 100msec slower than reading the file uncompressed. That's a reasonable performance hit for the fact that I didn't have to do the messing with the zlib libraries.

Yeah boost!

It's been a very long time since I wrote a post. Work has been as difficult as I can ever remember it being. I've written at times about how it's killing my ability to write anything, and while the same it true today, I'm not going to let that possibility crush the life out of me. Not now. And shame on me for letting it get to me before.

Life is a tragedy for those who feel and a comedy for those who think - Fortune Cookie

I'm tired of being in the midst of a tragedy. I don't have to be, and if I simply refuse to be, I won't be.

Oh, my circumstances won't change because of this, but the way in which I interpret them certainly will. Again, tragedy or comedy - the choice is ultimately mine. And starting today I'm choosing comedy. And there's a very good reason for it.

I've been at The Shop for more than 18 months, and I've completed a few really solid products. It's nothing that someone else couldn't have done, but I was here, I did it, and they know it. Good work so far.

I was originally hired to work with a really sharp, funny, guy. I've done all kinds of things in my day, so I know it's not the what you are doing it's the who you are doing it with that matters. My previous place had been a really nice job with some decent people but a really difficult manager. The great work environment, good perks, and nice company didn't even come close to offsetting the bad manager, and so I tried to learn from my mistakes and this job was about the guy I'd be working with.

So fast forward to the present day. My boss/partner is now heading up IT for The Shop, and I am lucky to talk to him once a week, and it's been months since I really worked with him. I have been lucky to find someone else to work with for the last few months, but now it seems I'm being reassigned, and he's not coming with me. So the reason for me taking this job is really gone. What's left is just coding - and I can do that anyplace.

So the reason for me coming to work at The Shop is gone, and my attempts to resurrect it have been failures. I've found someone good to work with, but all attempts for me to continue the working relationship seem to have failed as well. Now we get to the kicker - I'm being re-assigned to the one group I didn't want to work in. And I'm being given no choice int he matter.

It seems there's always one no matter where you go. The group that's hand-picked by someone high up in the organization, to make some kind of Untouchables squad. They are smart folks - to be sure. But probably not as smart as they think they are. Certainly not as smart as they are telling others. But because they were hand-picked, their egos go pretty much unchecked, and that becomes a significant problem.

We're a trading company, but some of the Untouchables come in at 10:30 am - two hours after the open. I'm sure there's a type of job that this works for, but one of the key developers in a trading organization it's not. Yet nothing is done because he's one of the special few. And that's just one of the problems.

My concern, when I was offered a position in this group a few weeks ago, was that my style of work would not mesh at all with this group. I was reading rands' tweets the other day and came across this article that very much defines how I see myself and how I work: meaningful and mindful work. It is a very nice little statement of the principles of some individuals - but I can see it's not for everyone.

Yet for me, it's gospel. I don't need this particular job. I'm fortunate to have found myself in an industry, in a time, with a skill set that is currently in high demand. Another time, another place, another set of circumstances that brought me here, and I'd be in trouble. But I'm lucky that I'm not. And thankful for that.

This group I'm slotted to go into has picked on one of it's own to the point of bringing a nice, reasonably happy man, to sobbing tears in the middle of a workday. That's not the kind of group I'm going to fit into easily as I'm not the type that's going to sit and take it. I'm as likely to lash out and inflict wounds on those intending to do the same to me. It's not a good plan, in my book. Which is exactly why I asked not to be placed in that group.

But my request went unheeded.

So today, in a few hours, in fact, I'm going to have my first meeting with the leaders of the Untouchables, and we'll see how things go. I'm not looking for a fight, but I also know that I simply cannot trust any one of them. They may turn out to be decent folks to work with, and everything I've seen from the outside might be explained logically by the different view from the inside. But the coming in at 10:30 am and making a grown man cry are going to take a lot of explaining, and I'm not holding my breath that I'm going to buy their explanations. But they deserve a chance.

If I am not convinced that this is a good thing for me, it'll be time to talk to a few partners - my original boss, and my new boss. I will certainly give them a chance to explain the logic of this move, and their plans moving forward. But given that I've been through a lot in my processional life, and only recently has it become so depressing that I've been unable to write in my journal, I'm not interested in the same old stories that I've heard from them in the past.

After all… I don't really need this particular job.