What's it all about, Alfie?

Today I got a somewhat sketchy bug report that, if it was true, would possibly have pointed to a section in my code where I was deleting things from an STL std::map using the erase() method whose argument was an iterator on the std::map. The original code looked something like this:

  InstrumentAliasMap::iterator    ia = mInstruments.begin();
  while (ia != mInstruments.end()) {
    if (ia->first == anInstrument) {
      // found a match - erase it
      mInstruments.erase(ia);
    } else {
      // no match, check the next one
      ++ia;
    }
  }

where my thinking was that the iterator would be updated to the next, valid element after the erase. I think that was a major boo-boo. In all honesty, I didn't really know, but it seemed to test OK, and so it went. The problem is that after the call to erase() the iterator, ia, is invalid and can't be moved to the next valid element in the iteration.

I did a little searching on this as I was sure there was something simple about this. What I read was that the right way to do this was to take advantage of the properties of the post-increment operator which will increment the value (move it to the next element) but return the original value. So the code becomes:

  InstrumentAliasMap::iterator    ia = mInstruments.begin();
  while (ia != mInstruments.end()) {
    if (ia->first == anInstrument) {
      // found a match - erase it
      mInstruments.erase(ia++);
    } else {
      // no match, check the next one
      ++ia;
    }
  }

In this code, the iterator is moved to the next valid element and yet the original iterator value is returned to the argument of erase(). This then removes the element from the map and we're sitting on the next, good one.

I like this a lot more, and it's clear what I should have done all along. I won't forget this guy.

I was working way too fast yesterday and skipped over some obvious bugs in the parser that were related to the addition of the JEP and BKit constants to the mix. The first was that pulling back a null valued variable returned a String with the value "__null__" - our tag for the null variable. This isn't right. I had to put the same value-mapping on the getVarValue() method that I had on the output of the expression parser. With that, it was fixed.

The second one was that the addConstant() method wasn't properly mapping incoming null values to the null tag, which was a problem I was having yesterday and simply wrote it off to the super's method getting called. I should have known better, but I was in a big hurry to get this done for the developer. The problem was in the missing null mapping, and when I added that in properly, everything worked just like I had expected it to.

I'm glad I went back and scanned what I was doing to make sure it was all done. These little loose ends are annoying and erode your credibility with others. Don't be sloppy - even if you have to be fast, go back and make sure it's all cleaned up.

I got a visit from a developer who is using a shared Java library of mine - BKit, and two components of it, specifically: the BKTable and the expression parser - BKJEP, itself based on the JEP parser while it was Open Source and free. They have since closed the source on JEP and upgraded it's capabilities, but for what we need it is more than adequate, and I really don't want to get into a licensing situation with a group that started Open Source and then closed it down. They sound like they're on the brink of closing shop.

Anyway, the developer wanted to return a null value for the computed column. Originally, the idea of the computed column was to give the ability to the user of the BKTable a way to create a simple computed column without having to do it themselves. They could say "Give me a column called 'Area' that is 'width * height'" - where the columns called 'width' and 'height' were already defined in the table. When either 'width' or 'height' of any row in the table changed, the resulting 'Area' column was also updated. Nice, but not Rocket Science.

It got more complicated when they wanted to have Excel-like functions. Things like dateFormat() to turn a Date object into a String for comparison... isNull() to test if the value is null... and then if_then() to give it a simple conditional. And it's this last one that's opened the particular can of worms I'm dealing with right now.

When data is set in the expression, it's impossible to give it a true null value. The core JEP parser simply doesn't allow it. So we encoded the null to be a special String (__null__ if it matters, and it doesn't) so that it could be 'set' and 'get' from the expression. We also created the isNull() function so that you could test for it. So far so good. But what if the output of the if_then() is to be a null? Ah... there we have an issue.

There was no way to create a properly encoded null value within a JEP expression. This developer was the first to ask for it. There's a lot of good that could be done with this variable defined in all expressions... but there's a lot of danger as well.

Say the user snuck an expression past us that generated a true null value. Then used that value in the test: x == null... well... the value of 'x' would not have been encoded to the proper string constant and so it'd fail. So we have the problem that there are cases where something that should work - won't work. The easiest way to think about this is to just be careful and use the variable 'null' only when you have to. And test... test... test.

But with that, I think this adds more than it harms, so I added it in.

UPDATE: wow... that was a lot harder than I had originally thought. The first cut I had seemed to work, but it really didn't. I had assumed that within JEP the constants and the variables were kept apart as the first are immutable and the second aren't. But in fact, they are the same storage class. So... when I scanned through the variables in the BKTable's method, I still caught them and they didn't appear as column headers and so the expression was tossed out.

I had to change the way the BKTable's method worked. When I got a variable from the expression I had to check to see if it had a defined value. If it did, then it was a constant as only constants had defined values at this point in the processing of the expression. This skipping was essential to get the expressions not only passed in, but also properly evaluated. I ran all the tests on the table and they all came up OK. Whew! That was a lot more complex than I had originally thought.

My favorite IM client - Adium, released version 1.3 this morning and I had to get it. The release notes indicate a ton of improvements, but there's enough bug fixes to make it worth the time to download the version.

They have made it a big deal to revamp the entire contact manager system in the app. Now you have much more control over your buddy list as well as being able to set different activity alerts (sounds, events, etc.) for each buddy's different actions. While I don't do that now, I can certainly see why that would be a very nice thing to have.

There's also now Facebook chat as well as new MSN chatting libraries to make that smoother than before. It's a heck of an upgrade, and if you haven't tried it, you really should. Highly recommended.

I haven't had a lot to do today on current projects as I'm waiting for others to get things lined up or deployed for me to do my stuff. So I've spent most of the day transferring the contents of my HTML journal at work to the WordPress weblog I've got there. I accomplished about 4 months of postings - quite a lot of work, actually. At this rate, I should be able to get all the entries into WordPress in another week.

But even making this nice progress is tiring. I'm really tired. Sitting still... hammering in changes. Trying to stay focused so as not to make mistakes... it's just plain tiring. But it's still good to see the "To Do" list shrink. My goal is still to get it all into WordPress - even if it takes longer than a week to finish.

It's not been a particularly great day... nothing really outstanding has happened at all. I'm keeping busy moving things from my old journal to the WordPress blog... I'm getting faster at it. It's pretty interesting when you make competitions with yourself to make the time pass more quickly. It's doubtful that anyone is going to say "Hey, this is great!" when I'm done, but it keeps me from sitting doing nothing.

Some days, it's just a matter of slugging through the day-to-day mud of a job. It's not glamorous, it's not even really interesting. It's just the job. Today is one of those days.

I'm taking off a little early because I'd like to have a nice long weekend. Nothing planned... just no need to be here, as evidenced by the fact that there's nothing really to do.

Today I've spent far more time than I'd like to have on a problem that has seemed to plague me for months. There's a perl client app to my market data server and it is run to get end-of-day prices for a variety of instruments that we don't get (easily) any other way. Seems reasonable.

Oh, how I wish it were...

I've been hearing that it's working, and then it's not. I look into the data, and it's fine. Then it's not. I don't want to take over this application because it's something that the guys should be able to deal with, but it has been getting closer and closer to that point because I keep thinking it's clear, and then there's an issue.

Just yesterday, I had an issue with a wild bug in the market data server. I deployed it and all looked OK. Then today it's broken again. I'm not sure exactly why this is happening, but I wasn't going to take any unnecessary chances: get the latest driver libraries, and make it from scratch. Maybe there's a header file/library mismatch? Who knows, but I was getting tired of this.

I also got their perl application - including the data, and stripped it down to the point that it's getting the data, testing it, logging failures, and then skipping the updating of the database. I've run this and it's returned good data for everything - save the ones where the ticker is bad, or there's not sufficient data in the database to generate a good ticker.

At this point, I'm convinced that the market data server is OK, and the perl interface into it is also OK. The application should run without problem, but if it picks up another mysterious problem, I'm going to be looking at the machine it runs on to see if there's any problems. It's got to be something, and the code just isn't it.

Digging into other people's code and not re-writing it is one of the less fun things I have to do from time to time. It's like a tacit approval of the horrible way they have put together this application. It was (and is) a mess, and should be recoded to be a lot more streamlined and efficient, but that's not my call. Thankfully, for now, it's working. We'll see what tomorrow brings.

This afternoon I found a wild bug in my MarketData server related to the symbology mapping and F/X rate conversion code for my ticker plant. I know this may seem to be a little 'specialized', but finding it was one of those 'Eureka!' moments that comes with a flash of insight.

This particular data provider for the market data server was the ticker plant I'd written to interface to the Reuters RMDS system. The symbology in our Shop for international tickers has an '=' separating the base ticker from the primary country of trading. So Vodaphone in Great Britain is VOD=GB while Google in the US is just GOOG - the '=US' is implied.

These need to be converted into RIC-like aliases for the ticker plant. I say RIC-like because there are synthetic prices in the system created by basket prices and/or mathematical operations done on 'natural' tickers. Say we had an F/X rate in GBP/USD, but we wanted it in USD/GBP - we'd make a synthetic price that's essentially '1/x' where x is the original F/X rate.

Well... it was this '=' that was giving me the problems. When a user asked for a piece of data for a symbol, they could ask for 'native' or dollarized (USD) figures. This means that I needed to have the F/X rate for all symbols at hand for returning the dollarized figures. But if the user happened to give me an alias as opposed to a symbol, I would happily process that and return the values. But there's the rub... if I were given the alias - that happened to include an '=', I'd try to get the F/X rate and that would be impossible.

An example is JPY=.RTL the USD/JPY F/X rate. If I were asked for that, I'd look at is as if it were a non-US symbol with the country code of '=.RTL' when in fact, it was an alias for an F/X rate and as such had no business being converted to dollars. I would try to get this mythical F/X rate (for this F/X rate) and it would be zero - causing the data to be zero.

What I needed was to be more aware of the symbols coming in and to be able to detect which were tickers and which were aliases and not try to get the F/X rate on the aliases. Once I did that, I was golden and everything worked fine. But it was interesting that I was looking at the data and the '=' sign hit me like a ton of bricks. I'm sure glad it did. The fix took all of 5 mins and everything is working fine. Whew!

Because I have had some spare time today, I've been transferring some of the old journal entries from my work journal to a WordPress blog I created for just that purpose. The old system was a single, giant HTML page, and while it loaded quickly, and you could do searches of sorts, it wasn't as flexible as a WordPress system. Which is, of course, why I made the change.

Anyway, it's not all that bad if you get the data formatted reasonably well. Sure, it's tedious, and it's not something you'd do for fun, but if you want to move the entries into the system it sure beats re-typing all those pages. I'm close to getting 2008 into WordPress, but that still leaves 2006 and 2007, and that's a ton of entries.

Baby Steps... just keep taking baby steps.

The MacVim Team has cut Snapshot 34 that includes the ability to turn on/off the ATSUI renderer for the app. The ATSUI renderer is the Apple Type Services for Unicode Imaging system - basically, the (old) Apple way for rendering Unicode. It's supposed to be significantly faster for rendering text but in the past it's had the limitation that the mouse events simply weren't supported in the MacVim code. What's happened is they added a superclass that had all the standard mouse support and then sub-classed the standard renderer and the ATSUI renderer. I'm hoping that it's faster, but it's awfully fast already.

I'll be working with this guy for a while and trying to find any problems but it'll be nice if it's faster and not picking up any bugs.

UPDATE: interesting note: I read about ATSUI and it seems it's rather old (Mac OS 8.5) and has been replaced by Core Text in 10.5 (and beyond). So... while it seems we're one step up in MacVim, it seems it's come right at the time when it's really a generation behind. Shucks. We'll see what happens in the future with SubEthaEdit and BBEdit - see if they pick up on this.