What's it all about, Alfie?

While I know it's not possible in a lot of cases, it really is nice to be able to go back and revisit code you've written several months before. You can see a lot of things that seemed obvious at the time - and now they aren't so obvious, which means you might need to recode them to make them easier to understand, or at least add docs in the code to make it easier to understand why the code looks the way it looks.

Case in point: today I found a minor bug in the pricing logic of the server. When a quote and a trade come in with the exact same timestamp, the logic should have been that the trade 'wins'. What was happening was the quote was winning. Seems like a trivial change - somewhere a 'greater-than' needed to be changed to a 'greater-than-or-equal-to' and that should have been it. Oh... if only it were that easy.

The pricing logic in the server is really, unfortunately, quite complex. We need to look not only to see if the quote is good - meaning does it have the requisite parts, but also does it look right? Meaning, is the bid/ask spread too far to be considered a good quote? We have tons of rules like this in the pricing engine part of the server. So it's not going to be one place. But it was certainly more complicated than it needed to be. So I dug into it.

I was able to simplify it a lot by taking the price and the instrument the price was for as two different logical entities. First, see if the price itself had a 'good trade', and a 'good quote' - regardless of the instrument it might be applied to. Then look at the instrument and see if it was supposed to be quotable, and if so, was the quote more recent that the trade? If not, then don't use the quote.

If I got no quote, then see if I got a decent trade for this instrument. The logic sounds easy, and is... now... but the original code combined too many things at once - trying to see if the price/instrument combo was good. This meant that a lot of checks were duplicated for different instrument types. That's just bad.

So even though it looked logically fine to me 4 months ago, it really needed to be re-written to make it even easier to understand. Now, when we run into issues it'll be a lot easier to follow the logic of the trade/quote part of the pricing section. Almost everything can be improved upon.

Since last Friday I've been really out of it. Bad, bad sickness. That's the primary reason that I haven't posted anything. I haven't written much code, either. It's been a hunker down siege mentality - just get through the days. Then, when I started to feel about 75% recovered yesterday, I decided to come in to work and see what's what at the Shop.

Several meetings - unusual for me, but it's all part of a new 'interest' in the goings on of the development staff. It's gotten big enough that the business (traders) don't know all the faces and therefore have no idea what each one is doing. This shouldn't apply to me, and when I asked, I was told only to include a few broad strokes as they knew what I was doing, but even so, it's a level of inspection that shouldn't apply to someone who deals with the traders on a daily basis. I know my Boss isn't filling out what he's been doing.

Anyway, that kind of micromanaging might be perfectly normal in a lot of places, but here, it's not normal at all. I fill out a journal with everything I do on a daily basis for this exact reason, but they don't want to read that - they want the executive summary. I don't like doing paperwork for the sake of paperwork. They know what I'm doing as I'm doing it in front of them, but with the new org-chart, I've been put in a 'box' that requires that we fill this out. It's more than annoying to me.

If my 6+ years here haven't proven themselves as trustworthy and doing the "right things", then let me know and I'll move on. After a time, asking me about what I'm doing it really quite insulting. Enter the 'disgust'.

More meetings... and then one with a data vendor. As a result I'm supposed to change the utility of one of the market data server's data providers and change it to conform to their new requirements for reporting. It comes down to this: I have to know who's asking for what and ensure that that's the only person looking at it. Well... in an of itself, that's not a bad request, but in comparison to what's out there now, it's a major step backwards. For example, all automated jobs getting data are now dead. Period. I have to know who it's going to, and that person needs to be logged in right them. That's the part that kills the jobs. What if the job takes an hour to get all the data? Today, we have that hour-long job start at 3:00 am so the data is sitting there waiting for them when they arrive.

Tough luck, they say... they have to be logged in and then they can start the hour long request.

It's unreasonable, and rather than allow me to point these things out, they say "Sure, OK" and then it'll be up to me to make it work. Which means I'm going to have to find some silly way to circumvent the literal rules of the license just because they didn't say "Hey, we love this data... but it takes too long to get it how you want us to get it. Make it take 30 sec and not an hour and then we're OK." But no, they know nothing, and yet they are the ones making these decisions.

Classic Dilbert.

I know it happens everywhere, it's just that I've been sick and have no stomach for this today. Gotta get some rest and let it go. Problem is I care, and that's a key reason I'm as good as my job as I am. Who wants to have someone that's not committed to the job? Not me. No way to live.

OK, maybe birth is more than this warrants, but I've been working on the local price injector for a new application and I've finally today gotten it to the point that it's talking to me (via chat)... that it's getting prices from the price feed, checking to see if they need to be sent along the way, and cycling through all it needs to do.

The backups are looking good... heck, I've even managed to have nice sub-second times for the log messages. It's getting close. It's to the point that the major issues remaining are the eternal questions: Where are we getting the symbol set? and the maintenance issues like: Are we going to need web access to this data?... that kind of stuff.

When we get the group together next week it'll be easy to hammer out these issues and then fold this into the code base. But it's a great feeling to get past the first phase of development and see a 'heartbeat' from the app. Sweet.

Today I needed to change our CVS pserver from an old Sun workstation to a linux box because some goobers at work decided to change the network taps on the Sun box and hook it to a non-responsive network port. Thanks, guys. Anyway, I took this opportunity to move the pserver from this old Sun workstation to a linux Blade in the server room. The steps are pretty simple.

First, make sure that /etc/services has the following line in it:

    cvspserver      2401/tcp

Then in the directory /etc/xinetd.d create a file called cvspserver and in it place:

    # default: off
    # description: A CVS pserver
    service cvspserver
    {
        disable = no
        socket_type = stream
        protocol = tcp
        user = root
        wait = no
        server = /usr/bin/cvs
        server_args = -f --allow-root=/path/to/your/cvsroot pserver
        log_on_success += USERID
        log_on_failure += USERID
    }

and then restart xinetd with:

    service xinetd restart

That's it. It works just as well as a pserver on Solaris because it's the same bloody thing. Now I don't have to worry about the goobers and network taps.

Possibly the most useful thing I learned while getting my Ph.D. was totally unrelated to the work. It was simply this: Often the difference between 'decent' and 'great' is a matter of endurance. The movie Galaxy Quest said it more compactly: Never give up. Never surrender.

Today is one of those days.

Several hours ago I realized that I had no motivation for the work I'm doing. I just didn't care. But it's at these times I remember the final year of my Ph.D. where I was simply wishing for the degree or death - I didn't much care some days. But as I got within the last months I saw that there really was a difference between those that kept giving it everything they had even when they didn't believe they had it.

You see, it really is just an impression... a feeling of desperation. Not that I was any really less capable of writing code this morning. Or that code on one project was inherently harder to write than another. Nope, it's all in the emotions of the situation. Take them out of it - even for a little and you'll be surprised that you can really keep going when you think you really can't.

So I kept pushing today. Take a walk to get a drink ever now and then, and get back at it. Never give it a break. Don't stop even when you want to. After a bit you start to see real progress. This may not make things that much better, but it's going to be reassuring tomorrow when you might feel the same, and it'll be nice to know that you did it today. If you did it then, you can do it again.

After a while things will turn around. They always do. If you can keep pushing through the hardest of times you'll feel better about the good times. Just don't give up.

Late last week, Rock Star - the fallen one came by my place and asked me why the two different source of historical prices were taking different amounts of time to return from my market data server. I explained about the caching of the data, and he mentioned that even on the second and subsequent hits, the one source within the Bank was considerably slower than the source outside the Bank. Of course, I tended not to believe him, or put his musings down as experimental error.

But as he showed me the data he was getting, I began to think that he might be onto something. So I wrote up a test case of my own and checked the two sources for the same data for a long enough date range to make the timings significant of the work and not of the overhead involved.

What I found was that the one was faster than the other. And immediately I thought I knew why. The historical data from any source has to be understood within the confines of the requested date range. Say I ask a source for 10 years of data for IBM. It's not going to return anything for holidays and weekends, so it's very possible that the first available day of data is not eh first day requested. If you have the logic of "check the cache for data, fill as necessary, and then respond" you end up asking for the same little bits over and over again. But they never amount to anything.

So I had to modify the general historical cache code that I had written for the second source, and add in the knowledge of the requested first and last dates - in addition to the actual first and last data dates. This made the code hit the cache completely which is a big win, but there was still a little difference in the cache return speed.

What I had done was to get the list of dates between a range of dates, and then using those dates, get the data from the time series for those dates. Over the weekend, I realized that the way I was getting the dates was just inefficient - scanning a std::map is not as good as using lower_bound() and then using that iterator to move through the map until we get to a point where the date is outside our range.

Likewise, it made more sense to subclass CKTimeSeries and provide a 'response filling' method so that we could use the same idea to directly fill the response from the cached data. Something like this:

  void MDTimeSeries::addToResponse( CKTimeTable *aResponse,
                                    const CKString & aSymbol,
                                    const CKString & aFieldName,
                                    int aStartDate, int anEndDate )
  {
    if (aResponse != NULL) {
      CKStackLocker          lockem(getTimeseriesMutex());
 
      std::map<double, double>::iterator    i;
      for (i = getTimeseries()->lower_bound(aStartDate);
           i != getTimeseries()->end(); ++i) {
        // see if we've gone past the limit of the data we want
        if (i->first > anEndDate) {
          break;
        }
        if (!isnan(i->second)) {
          // copy in the data as it's relavent
          aResponse->setDoubleValue((long)i->first, aSymbol,
                                    aFieldName, i->second);
        }
      }
    }
  }

This turned out to be a big win for the speed and it's now faster than the older implementation, which means I can get rid of it at sometime in the future if I want. Not bad.

I'll admit that I'm about as clueless of my surroundings as one grown-up individual can be, but today had to be right up there with the best of them. I was working late today working on the efficiency of my historical cache in my market data server and went to the bathroom before leaving. I ran into a friend there who happened to say "Have a nice extended weekend."

What?

I asked him, disbelieving what he was telling me, and sure enough. Monday is a UBS Holiday - no work. I was floored.

Of course I had to ask a few others, because you never know when people are trying to pull a fast one on you, but the information checked out. I have Monday off.

Had I left on time I would have come in on Monday and been all alone.

Well... almost. When I told one of my co-workers he was floored too. He didn't know either. We both had to laugh. But then we realized that this is exactly the way we like it. Don't tell us too far ahead... then it's not a fun surprise. Now, I feel like I really have something to be happy about. Like Christmas all over again.

What a sweet surprise!

Today I was working on my latest new application for price feeding. Part of this app is a crontab-like system. I've put several into several apps I've built in recent years, but most of them have been database-driven primarily because I thought it would be easier to use, and I could get something far better than the conventional crontab system from Unix. I was wrong.

Oh, the database-driven crontab system works fine, but it requires a database connection to work, which has it's pros and cons. The pros are obviously that it can easily be global, easily modified, thread-safe, and secure. You can write any tools you want to edit it, and it can be as complex or as simple as you want.

The cons are that unless you really take the time to write the tool, using SQL to modify the jobs is a real pain in the neck. My problem has been that I change the database crontabs so infrequently that building a tool has never really been necessary.

So this time, with this price feeder, I decided that a file-based crontab was going to be easier to use, easier to understand, and just as flexible. Face it... it's a bunch of fields where the first five are the when and the sixth is the what. So today I built a nice little crontab system that easily fits into applications.

The format of the crontab file is the same as unix crontabs, so it's easy for people to know what to expect when they look at it. The thread that runs in the app that checks for differences is simple - simply using stat() to get the last modification time, and checking to see if it's after the last known modification time of the file. If it's been updated, then we drop all the loaded jobs, reload and reparse the crontab into a series of jobs, and then check what jobs need to be done this minute.

I keep the time of the last minute we ran, sleep for 2 sec between checks, so we're going to be firing off the at-the-minute-jobs pretty darn close to the top of the minute. Also, we're not going to be firing off the same job more than once in the target minute. Pretty simple.

The most interesting thing was the parsing of the time field codes. Crontab allows three basic styles that can be put together with commas, separated by any whitespace:

• n - a single entry. For all fields it can be numeric, and for the months and days of the week field it can be the names.
• n-m - a range, inclusive of the endpoints. Again, for all fields it can be numeric and for the month and day of the week it can be the names.
• */i - every i units. This requires the i be numeric on all fields and will indicate valid times every i units. Therefore, */5 in the minutes column means every 5 minutes.

It was really surprisingly fun to have each of these cases handled nicely in the code. The logic was pretty simple - parse these fields into lists of applicable values and then put them into lists. When checking a time, simply break the actual time into the necessary components and see if each component fits into the approved values for that field for that job. If all match, then the job should be executed. If one misses, then skip it and go to the next.

With the caching of the cronjobs, we're not doing more reading and parsing than we need to. This makes the system less consuming of resources, which is always good. The command portion of the job can be anything we want - complex or simple, it's just up to what I need the app to be able to do.

When I get this app all done and everything is working, I'll probably pull this out and put it into CKit as a more reusable component. I just want to make sure that it's got all the features I need before I put it in CKit.

This is the second time this has happened to me - maybe more, I'm only starting to notice it today. I'm running Adium 1.2 on my MacBook Pro with 10.5.1 (fully updated) and when I put my laptop to sleep I think it turns off the sound on Adium until I restart the app. I'm going to check the website to see if they have any bugs reported on this - I know the last update included a new QuickTime - maybe that's to blame.

I just know that I don't like not being able to hear the chat announcements. I count on them to get my attention when I'm working at another machine. Crud.

UPDATE: I put in a trouble ticket for this bug. We'll see what they have to say when someone gets a chance to look at it.

[1/22/08] UPDATE: Adium 1.2.1 is out and it fixes this bug nicely. Don't know if it's something else, or my post, but I'm guessing that it was already in the works.

Last night I got a call from Hong Kong about slow ticks in the server. I talked to the support guy there for about 30 mins to get this out of him, but in the end, I figured out that it was a ticking issue, and further discussions with this guy were going to be pointless and frustrating. So I got online and checked things out.

Turns out there were a few things that weren't right, but nothing pointing to the real problem - which was that the ticks were seemingly working, but not as fast as they should and not as regularly as the ought to. I looked at a few of the logs and one of the instruments had no position on it, but positions on it's options, and so should have been in the 'positioned' poller queue - as opposed to the 'non-positioned' poller queue. I put in a fake position on the instrument and it moved to the right queue, but didn't start ticking like it should have.

I told the support guy over chat to simply hand mark these guys for the night, and I'd get to the issue in the morning and send an email about what I found. I needed to get some sleep or I wouldn't be able to really find the solution to the problem in the morning.

What I found was most interesting. Three problems in two systems caused this issue. The first one was a fix I had put in for the price feeder yesterday for augmenting the trade date/time with the current date/time when no trade date/time was sent by the exchange. The problem with the implementation was that it was also allowing ticks with no trade information to update the trade date/time. This meant that quotes were updating the trade date/time and that's no good. Not at night in Chicago when the ticks are for the next business day in Hong Kong. I was 'stomping' on the trade date/time and move it back a day. Bad move.

This meant that the server was looking at these prices and saying "Hey, this is old data... I need data for 'tomorrow' now." and throwing the ticks away. This explained the hit-n-miss ticks - some came in on the right date, and then the date was overwritten with the quote and no more ticks. Unfortunate, but that meant that I really needed to fix two things - the price feeder and the interpretation of 'today' for ticks.

Fixing the price feeder wasn't bad - I simply made sure that there was some evidence of a trade before defaulting to the current date/time for the trade time. This will make things much more reasonable. At the same time, I checked to see if the current date/time was before the trade date/time existing on the price. If so, then I didn't overwrite it as that would be moving it back in time.

The problem in the server was that I needed to have some knowledge of the region of the instrument. Specifically, I needed to know if it was an Asian instrument. For if it were, the checks on the date of the tick needed to be relaxed by the time difference. So I got a list of all the Asian currencies, made that a configurable option on the server, and then checked each instrument to see if it's currency was an Asian currency. If so, then I relaxed the time check to be after the start of the server so that we would not be in a position to throw away ticks when I knew they were really good - just in the wrong time zone.

The final problem was a simple fix to make sure that when I looked at an instrument to see if it needed to be in the fast tick pool or the slow tick pool, I used the method on the base instrument that checked to see if there were any positions on the instrument or on any of it's children. This fixes up the mistake of improper classification in the first place.

One nasty problem with three important changes. Not a bad morning.