What's it all about, Alfie?

A few days ago, a couple of developers stopped by to ask me if I'd seen a problem with the VantagePoint graph initialization when used in applets. Basically, an applet built using the VantagePoint graphs would initialize quickly (sub-second) when only one core was available to IE. Add a second core, and the initialization time stretched out. In fact, adding successive cores stretched out the time so much that at four cores the initialization time was measured in minutes. Clearly, there was a problem. Their initial assessment was that the locking and unlocking of the VantagePoint data tables was the culprit, but I told them that knowing the guys who built it, I doubted that - but I sent them an email just in case.

Today, one of the developers gave me an update and it turned out that it wasn't just IE. Firefox had the same behavior. The other developer had created simply HTML files that would show the problem. So I decided to get that file and see what I could see for myself.

Interesting problem.

It turns out that it's not the locking and unlocking, it's the redrawing that's done after the unlocking that's causing the problem. What was happening was that every additional data point added (parsed from the applet's param tags) was causing a redraw, and Windows was trying to slice things in between the redraws and that was causing the slowdown. Even after I put a lock around the entire data parsing, the delay simply moved to the parsing and setting of the graph's structure.

The solution was to hold-off on all redraws while parsing all the applet's param tags. Then, after it's all set-up and ready to go, unlock the graph and let it redraw the viewport once. This made the four core initialization faster than the old single core because no redraws were being done at all. Since this is a time that you really don't need to look at the intermediate results, it seems like a reasonable compromise.

What amazes me is that Windows isn't smarter about this. When multiple cores are available, Windows should improve performance not have it get worse. This might be as simple as upgrading the video drivers - who knows, but the fact that an applet in a browser is causing this kind of problem for Windows is amazing. Anyway... the code in BKit is fixed and the developers are back at it loading faster than ever.

I learned a new thing - at least it appears to be new, about crontabs on Solaris 8 - maybe later versions as well. It seems that if you have blank lines in a crontab it's going to give you errors. This is news to me for I've had a crontab on a Solaris 8 box for about 4 years that has blank lines in it and it's working fine. But today I had to update my crontab on my main Solaris development box and it was giving me fits about an illegal end of line.

I tried a ton of things, finally I dug into the man pages on the Solaris box and it said right there that blank lines should be avoided. Odd. I looked at another Solaris box and it's crontab has blank lines and it's working fine. But sure enough, take the blank lines out of the crontab and it's working fine.

Odd... but at least I know what to avoid on Solaris.

Today I've spent a good bit of time adding two interesting (and useful) data storage template classes to CKit. The reason I added them today was that I had noticed their use in a few other projects I've been working on, and they are just complex enough to cause 'clutter' in the project and not enough to be a significant part of the project. They're right where I find the things to strip for re-use.

The first is a 1:1 map where the data is added with a simple put() command but the underlying data structure is really two STL std::map objects - one going from keys to values and the reverse going from values to keys. This 'bi-directional' map allows me to be able to quickly and easily get a key for a value and a value for a key. In a few of my projects I do encoding mapping or translational mappings - where the map is 1:1, but it's as often that I need to go in the 'reverse' direction as the 'forward' direction. Scanning the map was far too inefficient and the size of these guys wasn't an issue, so it made sense to have two 'opposing' maps and place (and remove) the data in pairs using both maps.

The second is a many:many cross-reference map where the pairings do not have to be unique - similarly to the STL multimap, but in this case, the data is maps of std::sets which makes it easy to get iterators on the values for a key - or the keys for a value. This guy is used in a project of mine where there are several representations of the same thing, and a 1:many is a simplistic case of the many:many. I'm not sure if I'll go back and retrofit the code in the projects for these new template classes, but it's nice to have them should the need arise again.

I was looking at a problem today and saw a code snippet that looked a lot like this:

    try {
      if (timeToDoA) {
        ...
      } else if (timeToDoB) {
        ...
      } else {
        ...
      }
    } catch (Exception & e) {
    } catch (...) {
    }

where there were several if-then clauses each trying to determine what needed to be done based on the input conditions. Standard stuff.

Except for that try/catch block. That wasn't right at all.

While it was nice that the try/catch block was there to catch exceptions, I realized that it wasn't doing anything with them. This was most likely someone else in the group who put this in - if for no other reason than that's not how I would have structured it. In the end, I had something like:

    if (timeToDoA) {
      try {
        ...
      } catch (Exception & e) {
        // log error about doing 'A' with exception
      } catch (...) {
        // log error about doing 'A' with unknown exception
      }
    } else if (timeToDoB) {
      try {
        ...
      } catch (Exception & e) {
        // log error about doing 'B' with exception
      } catch (...) {
        // log error about doing 'B' with unknown exception
      }
    } else {
      try {
        ...
      } catch (Exception & e) {
        // log error about doing 'C' with exception
      } catch (...) {
        // log error about doing 'C' with unknown exception
      }
    }

With this approach, I get the logging and proper error handling with the focus on the events that precipitated the exception(s) in the first place. This level of logging might seem a bit much, but if there is a section of code that's too broad and an exception is thrown, it's going to be very difficult to find what exactly caused it. Better to invest the time and keep things tight and contained rather than too loose. Certainly in this codebase.

So watch out for those try/catch blocks that go nowhere. You may bot get an exception, but you may and not know it.

This morning I'm still thinking about the in-memory database. As the first step towards that, I realized that my CKTable didn't have indexing on the column headers or row labels. This is in the Java version of the table - BKTable, but not yet in the CKit version. So I took the time this morning to add those indexes into the CKTable. I also took the time to add the operator() to the table with two arguments - one for the row, the other for the column. This makes indexing into the data in the table very easy, and isn't available in the Java version.

The last thing I did this morning to move a step closer to the in-memory database is to add the cast operators for the CKVariant. This makes it a lot easier to use the variant in places where an int is need - or a string, assuming that the variant is holding the type of data you're casting it to. If not, it'll throw an exception, which is essentially the same as the ClassCastException in Java.

With these two things done, I'm off to thinking about how I might create an object that takes a table and does the extraction from it for a complex SELECT/WHERE clauses. It would be nice, but there may be real problems still ahead.

Today I realized that one of my price injectors wasn't properly updating the data structures. Well... that's not really right... it was working, it just wasn't doing what I needed. Basically, the first cut I had of the data structure was to have a map where the key was the price identifier (RIC for Reuters, BBG Symbol for Bloomberg, etc.) and the value was an array of instruments that would need to be updated if a price with this identifier came into the injector. Pretty simple. Price comes in... we pick off the identifier... we hit the map and get the array of instruments, and then for each instrument we send an update. Simple.

But it's got a flaw.

What if I received a new price identifier for an existing instrument? Then, I'd add another key to the map with one instrument on it. I wouldn't remove the old one, and so there might be two price injections for the same instrument. Bad idea.

The simple fix would be to remove the old instrument from the array in the map - but that would require a large scan - first, of all the price identifiers, and then for each element in the array associated with the identifier. I didn't like this scanning as it was bound to be inefficient when the numbers got very large. So I had to change, or at least augment, the data structures.

What I chose to do was to have another map - this one from the instrument to the identifier so that I could easily look up the identifier given an instrument. This would then allow me to quickly find all the instruments for an identifier, and then the identifier for an instrument. With this, I was able to quickly remove an instrument if the identifier changed, and also easily send out the instrument updates when a price (with identifier) came in.

But it got me thinking... what I really wanted was a simple database table. Something where I could say 'SELECT identifier WHERE instrument=blah', and then 'SELECT instrument WHERE identifier=zip'. This would allow me the freedom to look at the same data two ways, and even if I didn't have the complete relational database, a simple SELECT on a table would be all I'd really need.

There's a lot to think about here. Maybe it would be easier to just use the multimap in STL and see if that doesn't handle all my needs. If I did a multimap of identifier to instrument, I could easily find all the instruments for a given identifier, and with a reverse map, I could find the one identifier for a given instrument. I guess that making a simple template might be all I'd need and then I'd have all this functionality.

But that in-memory database table would be really nice. I can think of a lot of uses for it. I may have to spend more time on this tomorrow. It's a really interesting idea.

UPDATE: I looked more at the STL multimap and I don't like it's insertion methods at all. Yuck. Why not put in the operator[] like the map? It's got to be possible - but I can't fit it in after the fact. The best I can do is to subclass it or something. I'm really a bit surprised at this. In any case, I'm not going to keep going after the multimap for the price feeder. Ick.

I just happened to be checking on a few things this morning, and one of the little unsung heros of my day is Growl. I really think it might be the next thing Apple pulls into 10.6. It's just an exceptionally handy tool to have at the system level - like tabbed terminal sessions (from iTerm) and virtual desktops, it's something that a lot of people like myself use.

Anyway, the point was that for the longest time, Growl had the same web site that didn't look all that great on Safari - at least how I looked at it. Today I noticed that they totally revamped the web site and it's looking much better. No new releases, but now they show the different styles as well as better docs, more screen shots, and a better look to the whole thing.

Nice update.

I've spent most of today debugging a problem I saw in a vendor's API to a messaging system. It's not the best vendor I've ever worked with, in fact, I don't think they are even in the top 80%, but they are the vendor I have to work with at the time, and I've had to try and make the best of it. So here's the problem we ran into today.

We have a price injecting system. It gets price data over a custom socket interface (I wrote) from a price feed server and the reformats it into the format needed by the vendor's product and sends it on it's way. It's a simple transformation system. Nothing fancy. But the vendor's API is socket-based as well, and as we were to learn, not done nearly as well as mine.

When the transformer/feeder app was running in Chicago, and the price source was in Chicago and the database for the vendor's product was in New York, everything worked fine. When the feeder was in London and everything else was the same, my feeder missed the messages coming from the vendor's messaging system when I was injecting prices.

Inject prices - miss messages. Stop injecting prices - get messages. Make another test app subscribed for the same messages and it always gets them - because it injects no prices. This was getting crazy. So on a whim I decided to make use of a price source in London for the London feeder - Bingo! Now it worked. It appears that the vendor's API can't handle delays in the socket delivery from another completely different source. Yup. It's got nothing to do with the use of the vendor's API - it's the activity on a completely different socket that's effecting the vendor's API.

Note that in all this, my code is working fine. It's the vendor's that's stopped working properly. Nowhere in the documentation do they say that excessive waiting on socket communications will invalidate the delivery of messages - why should they? They probably never tested it, as they probably never had reason to. But when you charge $20 mil for something, you really ought to take a more pro-active view on things. For example, don't use a home-grown messaging buss when there are so many commercial ones that you can include in your $20 mil cost and not effect the bottom-line much.

In the end, I think I'm stable now, but there's really no way to know. They aren't going to fix this, I didn't expect them to. They took almost 2 months to fix the last bug we pointed out and that was a simple recompile with the right data type for a 64-bit version of the API. This would require real changes in how they do things, and change is not a word I'd use with this vendor.

I just have to say that I really hate the fact that it's expected that we figure this out. I'm not getting any part of that $20 mil, and yet I've saved their bacon by figuring out how to make it work in our environment. Crummy vendor.

I was talking to a friend today and he was fuming over a decision that one of the other guys he works with made. Of course, this other person really isn't in a position to make these kinds of unilateral changes, and then have other systems change as well, without at least some discussion with the lead developer (my friend), but that's what he did. It's the Better to ask forgiveness than permission style of work.

My friend tells the story that this guy needed to add a field to a RPC call, and while it was meant to be an integer, he decided to make it a string because he didn't want to get into parsing problems. The "fear of the integer" so consumed this guy that he told others down-stream of the system he was working on to use the string and then parse it as necessary. There were problems because this is clearly a dumb solution to the problem, and they had to change it to what it should have been all along - an integer. But the guy didn't want to change the RPC interface - 'leave it a string' he says. My friend was fuming.

Since this isn't happening to me, I can giggle about this to no end. There's a guy that shouldn't be developing because he's not really a developer, and doesn't really understand why you'd want to make the interface match the datatype - he's one of those guys that would make all the data in a database table two varchars() - one for the name, and the other for the value, and put everything in there. It's almost comical, if you didn't have to deal with the fallout.

So my friend has to spend the next several days unwinding all these changes and getting the right data type in there - in multiple systems, all the while cursing under his breath. Yeah... it's funny if you don't have to be involved in it.

There were a few things that I had wished were different about MacVim - specifically, the filenames in the tabs had the complete path name, nicely abbreviated, but still, there. I asked the support group about that and they came back with the most incredible answers. Now it's true, I've been using vi/vim for about 23 years, and I'm by no means a wizard at Vim - I use it, and I'm quick with it, but there are a lot of things about it that I'm not aware of. Today was an education.

The Current Directory of the File

The first thing that struck me was that when you opened a new tab with the File Open dialog, you'd get the complete path to the file as part of the name in the GUI tab. Yes, the path was nicely abbreviated, but it was still there, and if I had tabs that I'd opened from the command line, they didn't have the path. When I read the answer it was obvious - the current working directory of the command-line tabs was well know - the ones from the file open dialog wasn't. So what to do? Well, the answer was to change the directory on those files as well.

The brilliant answer came from one of the contributors on the mailing list: have an auto-command change the directory for each buffer you enter. The lines in my .vimrc that controls this are:

    if (has("gui_macvim"))
        :autocmd BufEnter * :cd %:p:h
    endif

The Format of the Tab's Filename

The second thing that can help is to force the tab to have just the filename and not the complete abbreviated path. This is done with a simple set command:

    if (has("gui_macvim"))
        :set guitablabel=%t
    endif

With these two additions I have two things that I really didn't like fixed. When they get the ATSUI renderer working with the mouse support, this is going to be one incredible editor. It's pretty awesome already.