What's it all about, Alfie?

I was messing around with trying to create a feedback mechanism from one application (server) that I had no control of whatsoever, to my application that was feeding this app tick data. The problem was I seemed to be able to flood the input and create a back-up by sending in too many price events a second, and the only monitoring I had was what I was going to make. So I had to come up with a clever idea.

My clever idea was really pretty simple - monitor the log of the second process, and see if it was done processing the last bit of data I sent. If not, then wait for it, if so, then continue on. It's not rocket science, but the devil is in the details. What to do if the log message is not something I expect? What to do if the log isn't moving? All these details pop up when you put a feedback loop into your code and don't want it to hang when the other app is hanging.

But the interesting part is the reading of the log. It's a fast moving system, so I can't be assured that the file is going to stay the same for any length of time, but I need to get the snapshot of the last line in the file as it's being written. Not easy. But I came up with what I think is a clever solution:

  CKString readLastLine( const CKString & aFilename )
  {
    bool      error = false;
    CKString  retval;
 
    // now let's open this file for reading
    std::ifstream      src;
    if (!error) {
      // open it for reading
      src.open(aFilename.c_str(), std::ios::in);
      if (!src || !src.is_open()) {
        /*
         * While this is an error, I don't want to throw an exception
         * because it's possible that this file just isn't there. In
         * those cases, log the error and let it go.
         */
        error = true;
        std::ostringstream  msg;
        msg << "[readLastLine] While trying to open the file '"
            << filename << "' for reading the last message, an error "
            "occured: " << strerror(errno);
        SPLog::error(msg.str());
      }
    }
 
    // let's back up to the last '\n' and read what we have up to there
    if (!error) {
      char    buff[256];
      int     i = 0;
      // zero this out to make sure it's clean - no matter what
      bzero(buff, 256);
      /*
       * We need to get to the char *before* the end, and the way this
       * is done is to have an offset of (-1) and the location the 'end'.
       * We need to do this as opposed to start at the end because if
       * we do the latter, we'll never get off the end. This allows us
       * to back up the file to get what we need.
       */
      // get to the last character in the file
      src.seekg(-1, std::ios::end);
      // take a look at it, and if it's not a '\n', keep it
      buff[i] = src.peek();
      if (buff[i] != '\n') {
        ++i;
      }
      // back-up past this last character so we can loop
      src.seekg(-1, std::ios::cur);
      // keep backing up until we hit a '\n' and then stop
      while ((buff[i++] = src.peek()) != '\n') {
        src.seekg(-1, std::ios::cur);
        if (i == 255) {
          break;
        }
      }
      // see if we backed up so far that we missed the last full line
      if (i == 255) {
        error = true;
        std::ostringstream  msg;
        msg << "[readLastLine] While trying to read the last "
            "line of the file '" << filename << "' for the last "
            "message, an error occured and a complete log message "
            "couldn't be read.";
        SPLog::warn(msg.str());
      } else {
        // get back to the last good char we read
        --i;
        // now, strip the beginning newline - if it's there
        if (buff[i] == '\n') {
          --i;
        }
        // reverse the line into a CKString
        for (int j = i; j >= 0; --j) {
          retval.append(buff[j]);
        }
      }
      // close the file
      src.close();
    }
 
    return retval;
  } 

The interesting stuff starts with the open() on line 10. Typically, you might write that as:

      // open it for reading
      src.open(aFilename.c_str(), std::ios::in | std::ios::ate); 

but if you did that, you'd end up at the end of the file - as if to append to it, and you would not be able to "back up" into the file. This is the key to this code - find the end, sit on it, and then even if the file is added to, the pointer you have won't move and you can back up to get a line.

The seekg() in line 40 puts the 'marker' at the end of the file, and then reversing the buffer as you walk 'up' the file, 'peeking' the data out of it is pretty easy. The tough part was opening and quickly getting to the end while being able to back up.

In the end, this is working wonderfully for me. It's fast, stable, and as long as the log writer is buffered on newlines, we're going to get nice, complete, lines. Sweet.

It's an exciting morning for me... I updated Liza's MacBook Air to 10.5.5 last night, and did my MacBook Pro this morning. I'll get to the kid's machines, and my iMac later. The release notes indicate "graphics updates" and I'm really hoping that the ATSUI renderer in MacVim is fixed, but I'm going to have to work with it to know for sure. Also, there were a few updating bugs in the visual appearance of BBEdit 9.0, so maybe these two are related. Be interesting to see.

One of the really nice things in 10.5.5 is the speed that the Terminal.app windows open and get a prompt. Historically, this has been a real stinker in my book. Probably 80% of the time I start Terminal.app prior to this morning, I'd have some - up to 70%, of the windows simply not get a prompt. It's annoying, and I don't understand why they hadn't fixed it. Well... it looks like someone else had this issue and the first restart of 10.5.5 was a perfect launch of Terminal.app. I love it!

I'll be seeing what's up with the changes as the hours and days progress, but what a nice shot in the arm. I can see that Snow Lepoard (10.6) is going to be a nice upgrade when they focus on all these infrastructural components and get them faster, better, and more reliable.

This last episode with my back is proving to be a real hassle to deal with. The chair in my home office was a $99 chair (I was cheap) purchased many years ago (we had no money) and when I've had to sit in it these last few weeks it seems to make my back problems worse. I know it's the chair because I can sit in other chairs and don't get these aches that I'm getting from my office chair.

Yet I can't help it - I have to be able to get some work done. It's just a fact of the business I'm in. So I'm thinking of springing for a refurbished Aeron chair. I can get one for about $700 - half off. That's not a bad deal, and it's the exact model I've used at a previous job for a while and loved every minute of it. Excellent chair.

I have a hard time spending that much money on a chair for an office that I don't sit in on a daily basis. It is, after all, a lot of money for a chair. Heck, I can get a Lazy-Boy for that money... but it's not a good, working, office chair. I think it's time to realize that I'm not getting any younger and it's time to invest in a good chair for the rest of my professional life.

This morning I saw that BareBones released BBEdit 9.0.1 with tons of little bug fixes. Looks very much like a maintenance release and that's great news. I have to say that I've been playing with it for a while and while I understand the type-ahead and hints, I'm not sure that I'm the kind of guy that really likes to use them.

I love ctags, and that gets me a long way, and I work around a lot of guys that probably couldn't write a method without it. I know they couldn't write C++, so all that's left is Java. OK... I'm probably being a little mean here, but I'm certain all but one couldn't do it. The days of IDEs have taken away the skills of real programming in favor of click-n-drag development. It's a trade-off, but I think they are the worse for their dependency.

As for me. I want to know my code. Understand it inside and out. That's how you debug something. Single stepping is the last resort as it modifies the execution in all but the simplest of applications.

Anyway... kudos to the guys are BareBones for keeping BBEdit going. If only the Coding Monkeys updated SubEthaEdit...

I've been thinking about Subversion today. It's been brought on by the fact that the main Subversion server for the Shop has been down due to some reason that they never explained. At the same time, my local CVS pserver has been working just fine. But it got me to thinking. svn is nice for the ability to do svn status and get diffs and such. That's really nice. But if that's something you want to have, why not take it another step and make it all on your box - there's a Distributed SCM system. Like Git.

So... is it time to get into something like Git? Not sure. I was talking to a friend a while back and he's going in that direction because there are a lot of advantages to this scheme. Face it - the one thing CVS, Subversion, ClearCase, PVCS, etc. all have in common is that they have a central repository. This is great for a company that wants to have a 'vault' for their code. Some place to lock down, back-up, secure and control.

But if you say that's not important because there will be enough copies of the repository around, you don't have to worry about losing the whole thing. You can get it from your buddy. OK, "bad" for corporate-types, but for guys writing code on Open Source projects, it's ideal. I've got it - you've got it, and I don't have to constantly be connected to "The Source" to work and have version control. I only need to be connected to "The Source" if I want it to have what I've been doing.

Think of the case where one developer fixes one file and some of the other members need to get this file. With CVS you all get it when you do anything SCM-related. Nice if you have a fast network connection to "The Source", but if you don't it's painful. And if you don't need the change, you get it anyway, most likely.

Git (as an example) would let a group develop different parts of the project without having to connect to one another until they decided that it was time to share code - then they'd do it. And only then, share what they wanted. You could choose to have Git hosted in a central location for synching - much like a traditional 'primary' repository, if you wanted. There's no one way to do it.

I'm getting interested in the idea of having a central Git repository for historical records and then using Git locally to do most of the work. It would mean larger disk space usage, but I'm guessing with decent laptop drives it's not that big a deal. And there's TimeMachine to back it up. Might be a good idea. Have to give it more thought.

I have been reading good and bad things about the new Genius in iTunes 8 and so this morning I was looking for a specific song in my collection I hadn't listened to in a while (Please Come to Boston) and I decided to turn on the Genius and see what playlist the Genius would make with that song as the key. Low expectations... more interest than anything else.

But I was wrong... it's pretty decent. Then again, I have a pretty simplistic taste and so there was a lot of material that would be similar. Still... I have to say that their goal of trying to make easier for me to pick a song and get a neat playlist out of it is admirable.

I'm sure it'll get better, but even now, it's pretty darn good. Thanks, Apple.

Bean 1.3.3 was released recently (I read about it today) and I used to be a big fan - until I got iWork '08. At that point, I thought "Hey, I've got Pages - what do I need Bean for?" And the blog posting I read makes a good point - Bean is just lighter weight, and there are plenty of times that you need TextEdit and Pages, and there are plenty of times you don't. For those other times, Bean is a great alternative.

There is a school of thought that says Don't get three different tools when two will do. Or one. And I can certainly agree with that. To date, I can't say as I've had a need to use Bean, but I can see the logic. In reality, there have been precious few times I've needed Pages, but I like to have it just in case.

Maybe that's the reason for Bean - just in case. Good enough for me. Great for free.

Now that I updated iTunes to 8.0, I could update my iPod Touch to 2.1 - I tried earlier and it said "no go". Fair enough. It was a simple thing - plug in the iPod and a little while later it's updated. There are a fair number of 'backups' and such, but in the end, it's what needed to be done (I suppose) and that's what matters.

It's updated, has all my stuff on it and there we go. Nice.

I'm not sure I'm going to be using a lot of the features other than "faster" and "faster backups", but I have it and there was no reason not to upgrade.

Today one of the more annoying developers I work with reported a 'bug' in the BKJEP parser's getSymbols() method - saying that the symbols 'null', 'e', and 'pi' weren't supposed to be there. Well... those are the constants I added a little while back to allow the testing for null as well as the basic math constants. So they are symbols in the JEP sense of the word, but this guy's code wasn't smart enough to realize that not all symbols are those that he supplied.

What I think he's doing is assuming that all symbols are, in fact, tickers to be processed. He then tries to process these tickers, doesn't trap for the possibility that the user entered bad data, and then it bombs because he's done all this optimistic coding and this is just the first place it's been caught.

My first response was to say "Nope. These are symbols and you need to deal with the fact that users might enter bad data as well." But then I got to thinking. Were constants the equivalent of variables?

I had thought BKJEP overrode the getSymbols() method from JEP, but it hadn't. So I really was free to think of this in terms of what they user might really want. When I took the annoying developer out of the mix, and thought about the role of constants in the expression parsing I came to the conclusion that it probably is better to think of them as "invisible" to the variable space. They aren't to be changed... they support the processing, but may take no part in it's execution... in many ways they aren't the same as the other variables returned from getSymbols().

So I filtered them out of the getSymbols() call. I suppose that the better thing to add would be a getConstants() and then maybe a getAllSymbols() or something like that. It you break down the JEP symbols in a way that I thought it was originally, but wasn't: variables, constants, and both.

So, I guess the lesson is try to keep an open mind and even an annoying jerk of a programmer can get something right - even if by accident.

This morning (finally) the Software Update had iTunes 8.0 and Quicktime 7.5.5 which was mentioned yesterday in the 'event' with Steve. It's no fun to reboot, but since Quicktime is such an integral part of the OS, it's inevitable. Anyway, we'll see if I like the new features in iTunes 8.0 as much as they think I will.

Next up is the new iPod Touch software, but I had to get this first as it was required for the latter.