What's it all about, Alfie?

I was messing around with trying to create a feedback mechanism from one application (server) that I had no control of whatsoever, to my application that was feeding this app tick data. The problem was I seemed to be able to flood the input and create a back-up by sending in too many price events a second, and the only monitoring I had was what I was going to make. So I had to come up with a clever idea.

My clever idea was really pretty simple - monitor the log of the second process, and see if it was done processing the last bit of data I sent. If not, then wait for it, if so, then continue on. It's not rocket science, but the devil is in the details. What to do if the log message is not something I expect? What to do if the log isn't moving? All these details pop up when you put a feedback loop into your code and don't want it to hang when the other app is hanging.

But the interesting part is the reading of the log. It's a fast moving system, so I can't be assured that the file is going to stay the same for any length of time, but I need to get the snapshot of the last line in the file as it's being written. Not easy. But I came up with what I think is a clever solution:

  CKString readLastLine( const CKString & aFilename )
  {
    bool      error = false;
    CKString  retval;
 
    // now let's open this file for reading
    std::ifstream      src;
    if (!error) {
      // open it for reading
      src.open(aFilename.c_str(), std::ios::in);
      if (!src || !src.is_open()) {
        /*
         * While this is an error, I don't want to throw an exception
         * because it's possible that this file just isn't there. In
         * those cases, log the error and let it go.
         */
        error = true;
        std::ostringstream  msg;
        msg << "[readLastLine] While trying to open the file '"
            << filename << "' for reading the last message, an error "
            "occured: " << strerror(errno);
        SPLog::error(msg.str());
      }
    }
 
    // let's back up to the last '\n' and read what we have up to there
    if (!error) {
      char    buff[256];
      int     i = 0;
      // zero this out to make sure it's clean - no matter what
      bzero(buff, 256);
      /*
       * We need to get to the char *before* the end, and the way this
       * is done is to have an offset of (-1) and the location the 'end'.
       * We need to do this as opposed to start at the end because if
       * we do the latter, we'll never get off the end. This allows us
       * to back up the file to get what we need.
       */
      // get to the last character in the file
      src.seekg(-1, std::ios::end);
      // take a look at it, and if it's not a '\n', keep it
      buff[i] = src.peek();
      if (buff[i] != '\n') {
        ++i;
      }
      // back-up past this last character so we can loop
      src.seekg(-1, std::ios::cur);
      // keep backing up until we hit a '\n' and then stop
      while ((buff[i++] = src.peek()) != '\n') {
        src.seekg(-1, std::ios::cur);
        if (i == 255) {
          break;
        }
      }
      // see if we backed up so far that we missed the last full line
      if (i == 255) {
        error = true;
        std::ostringstream  msg;
        msg << "[readLastLine] While trying to read the last "
            "line of the file '" << filename << "' for the last "
            "message, an error occured and a complete log message "
            "couldn't be read.";
        SPLog::warn(msg.str());
      } else {
        // get back to the last good char we read
        --i;
        // now, strip the beginning newline - if it's there
        if (buff[i] == '\n') {
          --i;
        }
        // reverse the line into a CKString
        for (int j = i; j >= 0; --j) {
          retval.append(buff[j]);
        }
      }
      // close the file
      src.close();
    }
 
    return retval;
  } 

The interesting stuff starts with the open() on line 10. Typically, you might write that as:

      // open it for reading
      src.open(aFilename.c_str(), std::ios::in | std::ios::ate); 

but if you did that, you'd end up at the end of the file - as if to append to it, and you would not be able to "back up" into the file. This is the key to this code - find the end, sit on it, and then even if the file is added to, the pointer you have won't move and you can back up to get a line.

The seekg() in line 40 puts the 'marker' at the end of the file, and then reversing the buffer as you walk 'up' the file, 'peeking' the data out of it is pretty easy. The tough part was opening and quickly getting to the end while being able to back up.

In the end, this is working wonderfully for me. It's fast, stable, and as long as the log writer is buffered on newlines, we're going to get nice, complete, lines. Sweet.

This entry was posted on Wednesday, September 17th, 2008 at 3:47 pm and is filed under Coding, Cube Life. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.