What's it all about, Alfie?

Today I've been working a little under the gun. OK... a lot under the gun. Another group is looking to use the ticker plant I'm building and they want to be using it "right now". Well... it's not exactly built right now, but I'm doing my best to provide them with something they can test with. What ensues is a lot of pressure that I usually don't like working in. But... I know these guys, and they'd love to have an excuse why not to use it, so I have to suck it up and get things done as fast as possible.

Today was spent resolving a few issues about the ticker plant. The first issue that was a hold-over from yesterday was the fact that the FAST (FIX Adapted to STreaming) decoder for strings was returning more than it was supposed to. I was seeing a trailing asterisk (*) in some of the strings in the position after the last character I was supposed to see. So if the string was a maximum of 5 characters, then I'd see an asterisk in the sixth position.

In order to fix it, I went into my wrapper class and did a little more defensive coding:

  std::string codec::decode_string( fast_tag_t aTag, uint32_t aSize )
  {
    // let's give the decoder a little safe room - it needs it at times
    char    buff[aSize + 8];
    decode(aTag, buff, aSize);
    // make sure they didn't run long (and they do often)
    char  *ptr = &buff[aSize];
    *(ptr--) = '\0';
    // trim off the excess spaces on the right-hand side
    while ((ptr >= buff) && (*ptr == ' ')) {
      *(ptr--) = '\0';
    }
    return std::string(buff);
  }

I needed to give them a little headroom, and then truncate it at the maximum length, and then do a simple right trim of the data. Not hard, but it's amazing that their own decoder has these problems. Yikes.

The next problem with the ticker plant was the CPU usage. When it's just the ticker plant - without the cached ticks, it idles around 10%-20%. With the cache it was up around 70%. I started playing with the cache (it's lockless but uses boot's unordered_map) and saw that we were in another pickle. The general operation of this guy is to replace the old message with the new, and delete the old. But if we had people looking at the old, we'd mess them up something fierce.

I've written this up on another post, but the idea is to allow them to remain valid for as long as the client needs, and then trash them. It's not hard, and I don't think it slows things down much, but it's absolutely vital for proper operation.

Unfortunately, with all this work, I got to 3:15 before I could get a lot more done. At that point, I have no ticks, and I'm stuck. Kind of a drag to have to depend on other systems like this. But so it goes. Tomorrow I'll hit the CPU usage harder and see what I can find out for certain.

This was actually the most fun part of my day. I was worried about how to allow the quick cache's clients to run iterators over the cache data and not get nailed when the data in the cache is updated by the exchange feed. In general, it's hard to imagine. You have a cache that by design doesn't lock or notify anyone, and a reader that also by design is going to be slower than the ticker feed and will, very likely, always be looking at the entire cache and getting into a lot of trouble with it's iterators.

I was trying to think of a clever solution to the problem when I had another really nice eureka! moment: What I'd have is a temporary "trash can", and when the client asks to keep things "stable", the cache tosses the old values into the "trash". When the client is done, it'll "throw the trash away", and the cost of the deletes will be on the client's thread.

If I built it with a simple STL __gnu_cxx::slist, then I'd have a very fast queue. I don't need to have any specific order, just a place to hold these guys until they are no longer needed. So rather than calling delete on the 'old' message, the put() method on the cache will instead do a push_front() to the 'trash' queue. It's fast, clean, and should work wonderfully.

I have two methods that control an atomic boolean. It's a little utility class I wrote that wraps an atomic uint8_t as a simple boolean so it can be toggled/set/read atomically:

  void QuickCache::saveToTrash()
  {
    if (!(bool)mUsingTrash) {
      // first, clean out anything that might be lingering in the trash
      Message  *m = NULL;
      while (!mTrash.empty()) {
        if ((m = mTrash.front()) != NULL) {
          delete m;
        }
        mTrash.pop_front();
      }
      // now set the flag that indicates that the trash is ready to use
      mUsingTrash = true;
    }
  }
 
 
  void QuickCache::takeOutTrash()
  {
    if ((bool)mUsingTrash) {
      // first, set the flag that indicates that the trash is NOT in use
      mUsingTrash = false;
      // next, clean out anything that might be lingering in the trash now
      Message  *m = NULL;
      while (!mTrash.empty()) {
        if ((m = mTrash.front()) != NULL) {
          delete m;
        }
        mTrash.pop_front();
      }
    }
  }

And then in my put() method, when I'd normally have deleted the 'old' message from the cache, I simply:

  // the new one is in the cache - let's dispose of the old one
  if (oldMsg != NULL) {
    if ((bool)mUsingTrash) {
      mTrash.push_front(oldMsg);
    } else {
      delete oldMsg;
    }
    oldMsg = NULL;
  }

I haven't had a real chance to test it, but I'm hoping that tomorrow morning when I get ticks from the exchanges, I'll be able to see that this is going to work. If not, then it's back to the drawing board and figure something else out.

Today was spent getting the timing of events finished up. Yesterday, I took a lot of time getting the shutdown working, and then I had to make the movement of messages work. It's pretty funny to look at the code in retrospect - I had just assumed that things would be there when I needed them, and in my development case with hard-coded values, it was. But making it work with the actual timing of events was a lot harder than I had thought.

Basically another day spent on getting the start-up and shutdown right for all the components of the system. Ick.

Today I spent a lot of time working on getting rid of core dumps when I shut down the ticker plant. This seems very obvious, but after I just did all the work to fold in the configuration and then even more work today to get the authentication token into all the configuration code, it required that I change a lot of little things around. The upshot of that was that the subtle interactions on the shutdown were quite broken and I had to fix them all up again - but not revert to the old system due to the configuration and authentication changes.

Little things I ran into made this a lot harder than it sounds. We have boost's asio to contend with, and the fact that we need to cancel any pending I/O with them on the shutdown. However, that's going to generate an error in the socket communications, and from that we'll try to shutdown the socket. This is a nasty loop that makes it very difficult to unravel.

What I had to do was to be very careful about the two different conditions, and if I'm being interrupted, then I need to assume I'm being told to shutdown and not assume there's something wrong I need to attempt to recover from. Thankfully, the error messages are clear enough to make this possible, but it's still a lot of detail work about when things are happening and who's responsible for doing what.

All this took me the better part of the day - with the initial part being trying to get the timing of the initialization of everything right as well. It's like I spent two days getting nowhere as I didn't advance the codebase one feature - I just made it possible to use the configuration and authentication tools we have to use. That's important, do be sure, but it didn't feel like I got a lot done.

I've been working hard all day trying to work the configuration system into all the classes that it really should be used. It was really my fault for not putting the configuration service into place before writing all the code that should use it, but it just wasn't there, and I didn't ask enough questions about how it was going to work to get all the parameters, etc. in-place. Just means the job of getting the configuration into the system is a lot harder than it could have been.

Thankfully, and this is a really weak silver lining, I had a good start at what I wanted to use, but I hadn't taken it nearly far enough to make the process simple. It's further complicated by the fact that I need the configuration data at the lowest level components, and that means passing it down, down, down... Not horrible, but it takes a little bit of work to make sure that you put the configuration data in as few a places as possible with maximum coverage.

Not horrible, but not trivial. Just takes a lot of time.

I've been really slugging it out today. I mean really. Today has been a lot of work writing JSON configuration files for my ticker plant that will be read into a MongoDB that is the back-end for the configuration service. There's a lot of things I need to set up for this system - all the exchange feeds, all the distribution of the messages, all the mappings... it's a lot of stuff. In addition to all this, I really need to find a way to create the service coverage maps...

These things really got me down, I have to say. Not ashamed to admit it. There's something to be said about not getting bogged down, but when you're in the middle of it, the best thing you can do - I have found - is to just push through it. Taking a break isn't going to help - you have to come back to the exact same place. That's no good. There's no way to go around the problem, you have to go through it.

Kind of a drag, but it's what you have to do. And in the end, it's really what separates the good from the middle-of-the-road. If you push through when you're sick and tired of doing something, then you're going to get it done faster, and move on to something that you like to do. Let it really slow you down, and you're spending more time doing this thing you're going to hate even more.

Well... gotta get back to it. Pushing through it.

I've spent most of today doing two things: weaving the authentication data into the Ticker Plant client code, and making a nice package (tarball) of the client include files and libraries for other groups at The Shop. Over the last few days, I've been doing the "second pass" on the code - looking at those supporting sub-systems and components that I've been "faking" with simple hard-coded values. For example, the original authentication system was a very skeletal 'client' of the authentication service and a simple 'token' that contains the important data, but the tests for the validity of the token was always true, and the client just created an empty one.

Like I said, very incomplete - but it worked and wasn't crucial to the functional aspects of the codebase. It's just the supporting stuff that the app had to have before it went live, but could always be done later. Well... now is that later.

Interestingly, putting the AuthToken in the code was surprising harder than I had expected. I simply hadn't exposed the ability to set the AuthToken at the highest levels of the code, which, it turns out, is exactly what I needed. Unfortunately, to find this, I had to do a lot of digging because I wasn't logging the requests that failed due to invalid AuthTokens.

First, I had to log things a little more, and then I saw the problem. Then I needed to see how often I was doing the same things in different places in the code. These weren't always done the same way in the code, so I took the time to standardized the way I was doing the authentication work. This took even more time. In the end, I was able to get it all working, but it took a lot more time than I had expected.

After that, I got pulled into a few meetings that pointed me in the direction that we might have a C++ client sooner rather than later. What I wanted to do was create a package - a tarball, that contained the libraries and include files so it's easily used by other groups. Making it was really a process of finding the minimum set of include files that I would need. Then I just had to stage them into a good directory structure and tar them up. I made it clean with some nice dependencies so that if we change one of the headers, we'll be able to know to make the package again.

Lots of good things done... but it was a lot of details to get done.

This afternoon was spent testing the ticker plant in all it's glory - that means loading data from the configuration service, checking the login service, and getting ticks moving. I realized that there were several parts of the system that weren't as polished as they should be - specifically in the configuration service client, and even a little in the Broker client code.

The configuration client didn't have a way for me to ask for a series of keys, and wait until all of them had returned. I had to do it one at a time. While it's not horrible, it's not as efficient as the system could be - after all, we created the asynchronous calls to the Broker just for this case.

So I added in the ability to get() a list of keys and have it returned in a variant list. The code actually sends out all the requests for the values asynchronously, and then waits for them all to return. There's a timeout, and a logging, if the complete set of values doesn't come back, but that's not something that's likely to happen.

In the end, I cut 40% off the time needed to get the configuration values this way. Nice.

This morning I finally was able to push through and finish the code for my persistence system for the ticker plant. We have a cache of messages in each ticker plant, and they are typically fed from the exchange ticker feed, but in the event that we need to restart the server, it'd be nice to be able to save and load the state of the messages (ticks) so if we have to restart, we can keep the cache in roughly the same state after the restart as before.

This is very important when you need to have a stable price flow. You can't have a restart take out the messages that came in the morning and haven't been seen since. Well... I'm counting on a service hanging off the Broker that is going to handle all the saving and loading for me. I'll simply send maps/lists of data (serialized messages) to the configuration service and when I restart, I'll read from the same. It's fine in that it'll work regardless of the machine, and we have all the tools at our disposal, but the problem is that we need the Broker and it's services up before we can start.

Minor point. If we can't get to the Broker and it's services, we're in a lot bigger problems.

So I got that written up, and a good bit of testing on the authentication service and the basic configuration service. I need to go back to the code now and start putting in calls to the configuration service so that these components and apps can load their configuration from the right source.

Never a dull moment.

I just got out of a meeting, and in the beginning things were going better than I thought. I was getting an idea on how their system was put together, listening to what they were doing, seeing where they thought they were going wrong. It was all constructive - up to a point. And at that point, I was very saddened to see "chips" appear on shoulders. Because I knew that the moment those chips went up, the communication broke down, and the meeting, and probably the entire collaborative project was over.

I understand the emotion. You're a professional. You care a lot about those things you've been asked to care about. When something happens that rumbles the ground you walk on, it's not a "good" feeling. It's OK to handle it for a while, but after a time, you really want to regain some sense of control. Like I said, it's understandable. I just wish people didn't get so upset about things.

But even that, I suppose, is understandable. I get upset at a lot of things. That I don't get upset about someone offering advice is my little filter. Maybe these other folks don't get upset by being in glass elevators - one of my phobias. It's just the want people are wired together.

The problem is that as soon as the offensive behavior goes up, the blinders go on and no amount of talk is going to bring them down. This person is now shut-off to me, and they aren't about to open up again until I agree I'm wrong, or apologize, or am reprimanded - something that means I'm addressed in the proper format, as they define 'appropriate'. It's just so counter-productive.

Sad to see it happen.