What's it all about, Alfie?

Today I've spent far more time than I'd like to have on a problem that has seemed to plague me for months. There's a perl client app to my market data server and it is run to get end-of-day prices for a variety of instruments that we don't get (easily) any other way. Seems reasonable.

Oh, how I wish it were...

I've been hearing that it's working, and then it's not. I look into the data, and it's fine. Then it's not. I don't want to take over this application because it's something that the guys should be able to deal with, but it has been getting closer and closer to that point because I keep thinking it's clear, and then there's an issue.

Just yesterday, I had an issue with a wild bug in the market data server. I deployed it and all looked OK. Then today it's broken again. I'm not sure exactly why this is happening, but I wasn't going to take any unnecessary chances: get the latest driver libraries, and make it from scratch. Maybe there's a header file/library mismatch? Who knows, but I was getting tired of this.

I also got their perl application - including the data, and stripped it down to the point that it's getting the data, testing it, logging failures, and then skipping the updating of the database. I've run this and it's returned good data for everything - save the ones where the ticker is bad, or there's not sufficient data in the database to generate a good ticker.

At this point, I'm convinced that the market data server is OK, and the perl interface into it is also OK. The application should run without problem, but if it picks up another mysterious problem, I'm going to be looking at the machine it runs on to see if there's any problems. It's got to be something, and the code just isn't it.

Digging into other people's code and not re-writing it is one of the less fun things I have to do from time to time. It's like a tacit approval of the horrible way they have put together this application. It was (and is) a mess, and should be recoded to be a lot more streamlined and efficient, but that's not my call. Thankfully, for now, it's working. We'll see what tomorrow brings.

This afternoon I found a wild bug in my MarketData server related to the symbology mapping and F/X rate conversion code for my ticker plant. I know this may seem to be a little 'specialized', but finding it was one of those 'Eureka!' moments that comes with a flash of insight.

This particular data provider for the market data server was the ticker plant I'd written to interface to the Reuters RMDS system. The symbology in our Shop for international tickers has an '=' separating the base ticker from the primary country of trading. So Vodaphone in Great Britain is VOD=GB while Google in the US is just GOOG - the '=US' is implied.

These need to be converted into RIC-like aliases for the ticker plant. I say RIC-like because there are synthetic prices in the system created by basket prices and/or mathematical operations done on 'natural' tickers. Say we had an F/X rate in GBP/USD, but we wanted it in USD/GBP - we'd make a synthetic price that's essentially '1/x' where x is the original F/X rate.

Well... it was this '=' that was giving me the problems. When a user asked for a piece of data for a symbol, they could ask for 'native' or dollarized (USD) figures. This means that I needed to have the F/X rate for all symbols at hand for returning the dollarized figures. But if the user happened to give me an alias as opposed to a symbol, I would happily process that and return the values. But there's the rub... if I were given the alias - that happened to include an '=', I'd try to get the F/X rate and that would be impossible.

An example is JPY=.RTL the USD/JPY F/X rate. If I were asked for that, I'd look at is as if it were a non-US symbol with the country code of '=.RTL' when in fact, it was an alias for an F/X rate and as such had no business being converted to dollars. I would try to get this mythical F/X rate (for this F/X rate) and it would be zero - causing the data to be zero.

What I needed was to be more aware of the symbols coming in and to be able to detect which were tickers and which were aliases and not try to get the F/X rate on the aliases. Once I did that, I was golden and everything worked fine. But it was interesting that I was looking at the data and the '=' sign hit me like a ton of bricks. I'm sure glad it did. The fix took all of 5 mins and everything is working fine. Whew!

The MacVim Team has cut Snapshot 34 that includes the ability to turn on/off the ATSUI renderer for the app. The ATSUI renderer is the Apple Type Services for Unicode Imaging system - basically, the (old) Apple way for rendering Unicode. It's supposed to be significantly faster for rendering text but in the past it's had the limitation that the mouse events simply weren't supported in the MacVim code. What's happened is they added a superclass that had all the standard mouse support and then sub-classed the standard renderer and the ATSUI renderer. I'm hoping that it's faster, but it's awfully fast already.

I'll be working with this guy for a while and trying to find any problems but it'll be nice if it's faster and not picking up any bugs.

UPDATE: interesting note: I read about ATSUI and it seems it's rather old (Mac OS 8.5) and has been replaced by Core Text in 10.5 (and beyond). So... while it seems we're one step up in MacVim, it seems it's come right at the time when it's really a generation behind. Shucks. We'll see what happens in the future with SubEthaEdit and BBEdit - see if they pick up on this.

Today a fellow developer came to me to ask how difficult it would be to make the scatter graph rescale the axes when the z-axis selection changed. This is the BKit graph that has the optional z-axis grouping selection list so that a third column can be used to group the data points defined by the x and y axes for easier viewing.

The problem is that VantagePoint (while a great package) is not universal in it's feature set. For example, the Bar and Combo charts allow for the ability to set the specific Observables - basically, the points in a data set (or Variable). If I could have applied this to the scatter graph, then I'd be set - simply make the Observable visible, or not, based on the state of the z-axis selections and the value corresponding to that data point. But the class hierarchy of the scatter graph did not include this functionality. I'm sure there's a reason, but I with they had included this in all graph types.

So I had to come up with another scheme. The problem is, the obvious one is to have a copy of all the data, and then when any of the selections change, we remove the old data and copy in just the data set necessary to plot the complete two-columns on the graph. The problem there is that's a lot of data moving of which a ton will be unnecessary. So I didn't want to do that unless I absolutely had to.

What I decided to do was nearly as compute intensive, but it turned out to be reasonable in the context of the GUI event. That was to look at the data values of the 'visible' points and then set the limits on the data in the graph to these, and let the graph refresh accordingly. The upside of this is that there's no data moving. The downside is that we're getting limits over and over again when they aren't necessarily changing. There's also the possibility that we might mess up the nice axes selections that VantagePoint does automatically.

Luckly, that last part had already been solved in the base graph class, so when I tried this approach, I was very pleasantly surprised to see that everything worked as it should. Not half bad, in fact. Thankfully, this is only active when the user specifies a z-axis column for the graph, otherwise this would impact the performance of the applet at all times. Not really great. Good enough.

I've been working with this software package for months now, and while I can't use the name for fear of lawsuit, I think I'll borrow a page from J.K.Rowling and call it The Software Package Which Must Not Be Named or maybe just You Know What for short. OK, it's silly, but here's another silly lesson to be learned from this million-dollar fiasco: Sometimes, if things are slow enough, they will be technically working, but the users will think them broken.

Case in point with You Know What was a certain instrument's trade ticket. There were supplemental data fields on the form, and they appeared not to populate after being programmatically set on the committal of the trade. I was testing this very thing this morning and after seeing it happen in front of my own eyes, I reached for my pencil to make notes of the fields I'd populated, and the ones that hadn't been correctly populated with the calculate values.

After a few seconds of writing the notes, I looked up to see the empty fields and amazingly, they had been filled. I'm convinced that this was what was happening all along. The way in which this system is wired up is to save to one database, replicate all changes to another and then reach from the replicant. If the changes went in programmatically, then the GUI might not have known to update them, and so they had to go through the replication chain and that was going to take several seconds. It might just be that the users/testers were impatient, and when they didn't see the data in these fields immediately after the saving, they assumed the fields weren't going to be populated and closed the window.

So... this morning I'm going to sit with the tester and make them wait to see if the fields don't appear. I figure that it's about a 30 sec wait, and if they show up for him like they did for me, then we know it's not a bug... it's a feature. Unfortunately, this is a nasty feature, but there's no helping that short of a significant reconfiguration of the system. Don't know if and when they might happen, but if it does, then this delay should lessen. But we'll have to see.

For now, You Know What seems to be OK. You just have to be patient. I wish I felt that described more of the users, but I don't.

Today I was trying to look at the chat interface of one of my apps that uses CKit and has a chat interface. I wanted to make sure that we'd be able to enable logging of individual price updates during the day because some of the support staff in London were getting a little panicked because they're used to this level of logging in another app I've written and frankly, they've come to depend on it.

So I was checking and realized that the chat interface was offline. Odd. It should work. Then it hit me... there was a change I'd made to the chat interface not so long ago (July 23, 2008: Lots of Trouble with MindAlign These Days). But I couldn't remember it. Thank goodness for WordPress! I found the entry where I'd changed the logic on the getReply() method on CKIRCProtocol. I'd had a problem of an infinite loop where I wasn't checking for an error when nothing was returned from the socket. So I changed the code to look like this:

    // now read up to the "\n" NEWLINE that the IRC server sends
    if (!error) {
        retval = mCommPort.readUpToNEWLINE();
        /*
         * It's possible that the data is empty - but the only way for
         * that to be acceptable is for a timeout to have occured. So,
         * if the data is empty and a timeout *didn't* occur, then we
         * need to disconnect this guy and the next pass through, we'll
         * be able to connect again and set things up properly - we
         * hope.
         */
        if (retval.empty() && (errno != ERR_READ_TIMEOUT)) {
            disconnect();
        }
    }

but that left me in the case today where the return code wasn't an error or a timeout - but in interrupt. What to do? Well... the point of an interrupt is that something caused it to bail, but it's not a problem of the socket per-se. So I needed to really change the code to read:

    // now read up to the "\n" NEWLINE that the IRC server sends
    if (!error) {
        retval = mCommPort.readUpToNEWLINE();
        /*
         * It's possible that the data is empty - but the only way for
         * that to be acceptable is for a timeout to have occured. So,
         * if the data is empty and a timeout *didn't* occur, then we
         * need to disconnect this guy and the next pass through, we'll
         * be able to connect again and set things up properly - we
         * hope.
         */
        if (retval.empty() && (errno == ERR_READ_ERROR)) {
            disconnect();
        }
    }

so that I'm sure to only bail when there's an error. Timeouts and interrupts don't count. This fixed up the problem and I got it out to the effect apps. Interesting little thing that turned up on linux. Solaris didn't have this issue.

I've been working with a system today that has nearly driven me to drink with it's hidden features. I know each complex system will have them, but when you are paying millions of dollars for a serious system, you should have as few of these hidden gotchas as possible. I submit that the higher the quality of the product, the fewer the number of hidden gotchas it has.

Take a good messaging system like IBM MQ or Sun ONE MQ. Both have serious price tags, both do a serious job, and while I may not like the configuration of one, or the other, I have to say that there are so few hidden gotchas in these two products as to really make them almost perfect. Ideal? No... but they do exactly what they say with no ambiguity and very little learning curve.

In comparison to the product I'm working with. They have a feature where you can define supplemental fields to the existing fields (attributes) to the stock objects in their system. You can add strings, doubles, dates, etc. Seems reasonably flexible, until you have to actually have to use them.

Say you have an object that has a built-in value (attribute) called 'date', and you want to add a supplemental field for 'expiration'. You can define the 'expiration' field just fine - as a date, even. You can then use the fields like:

  print obj.date
  print obj.expiration    // <- WRONG!!

You cannot "just use" the field you've defined even though the system has all the data it needs to work this out. No, you have to read it through a special method:

  print obj.read_supp('expiration')

and writing is even worse. You have to scan through all the supplemental fields on the type of object and see if there's the one you want, and then scan the object itself for this 'id' as you can't look it up on the name.

In short, this is an enormous pain, and there's even more. Even if you define a supplemental field as a double, you have to store it as a string! All supplemental values are going to be strings. The only advantage the 'type' has in the definition is the input checking in the GUI. It's a joke. I'm stunned.

In a good product there aren't any gotchas like these. Things work in a similar and symmetric manner. Sure, it takes more work on the part of the developers, but that's what it takes to have quality software - work. I'm continually amazed that this product sells any copies at all. (which is why I'm not referring to it by name, clearly)

If I"m lucky, this will die soon enough.

One of the things that has historically been a problem with NeXTSTEP/OPENSTEP and now Mac OS X development projects and source control is the fact that on all the operating systems, the bundle concept exists and complicates CVS considerably. The bundle is nothing more than a directory that's treated as a single entity. The app, the nib, the rtfd... all these are really directories on the filesystem and yet the OS thinks of them as single, indivisible entities. So when CVS goes to make a CVS directory in one of these bundles, and then the tool (such as InterfaceBuilder) saves a change, it overwrites the entire directory - destroying the CVS directory within the bundle.

With the recent editions of Xcode (from the old ProjectBuilder) the source code control has been built-in. However, I've been very gun-shy to use it for fear of not getting anything done because of these issues. So it was a very anxious time for me today when I tried to get my first Xcode 3.1 project put into my home repository.

Since the repository was already created, and running on Mac OS X 10.4 (Tiger), I thought if it were going to work at all it should work in this environment. So I did a little Googling on this and found a page that walks through this for Subversion, but not CVS. I figured that if I backed everything up, I'd be set. So I dove in.

Configuring Xcode for Your Repository

The first thing to do is to have a repository, since I had one, and it was already running on a pserver I'd set up ages ago, it seemed like I should be set to go. What confused me for a little bit was the fact that Xcode was going to allow me to configure the root of the repository and therefore view all the projects I've placed in that repository. Very nice. In order to get there, however, you need to tell Xcode how to connect to your repository.

Select SCM -> Configure SCM Repositories and then click the '+' button at the bottom of the list to add a new repository. Once you give it a name and a type, you'll get an appropriate dialog box where you can enter all the data for the repository Xcode will need. As a nice little bonus, as you fill in the components, Xcode builds up the CVSROOT (for those that are familiar with this, it's a real lifeline) and so you can see each component going into the CVSROOT as you enter it into the dialog. When it's all filled out, Xcode will automatically verify the connectivity (another amazing piece of work) and then you can click "OK" to save everything.

At this point you have the repository configured with Xcode. Now you can look at it by selecting SCM -> Repositories and then the one you just added will be on the left and it's contents in the finder-like view on the right. You can look at everything you've already put into your repository (for those of us that used existing repositories, this is really a place to give a wonderful sigh of relief.)

Import Your Project

Once you have a project ready to import, simply select where you want the root of your new project to sit, and click on the Import icon in the Repositories window. Xcode will then ask you what to import - navigate to the root of the project directory and give it a nice comment. Xcode will indicate in the bottom pane of the Repositories window what's happening and in a few minutes, you're all imported.

Check-Out Your Project

As with all other CVS imports, save the old project file by renaming it, or tar-ing it up and saving it off someplace, and then from the Repositories window select the project you just imported and click on the Check Out button. Xcode will ask you where to put this, and you can navigate to the directory you want it to be placed. Click 'OK' and Xcode will do the checkout. As a wonderful feature, it's going to ask you to open the project in Xcode when it's done. Say 'Yes', as we need to do one last thing before we're done.

Enable SCM on Your Project

Now you need to navigate in the 'Groups and Files' list to the name of the project (typically the first thing in the list) and then click on Info to pull up the main project info window. Navigate to the 'General' panel and then at the bottom choose the repository from the SCM list and you'll notice the GUI shifts a bit to place the CVS status next to each file. At this point, you're ready to go. The SCM menu is now greatly expanded and using SCM in Xcode is very simple.

You can do diffs... additions... moves... all the things you'd normally do, and with CVS, Xcode is smart enough not to blow away the directories of the bundles and so things get versioned properly. It was easy... but very nerve racking for my first time since the old days. In the future, I'll tell people to do it earlier. Much earlier.

Because the chat infrastructure team seems to be unable to get their issues resolved about the MindAlign bots, and because I can't really honestly say that the bots I need are business critical, I've started looking for IRC servers that I can start up on my own, point all my servers at, and then simply have my own IRC system. The load is trivial, and with this, I'd be able to get things done, but there's always the risk of being tagged as a rogue chatter.

Given that it's been two weeks, and there's no accommodations being made for important but not critical bots, I'm thinking that even if I get nailed (and that's not a guarantee by any means) that my excuse is exactly that - two weeks, no communication, and no alternatives. Sounds good to me, anyway.

So I've been looking at the IRC servers, and it seems like IRCD-Hybrid is a decent tool, and the history of it leads me to believe that it's really different only in the very 'far out' IRC issues - things I'm not liable to hit ever. I just need a basic communication hub, and for that, this should do just fine.

I'll get it, build it and see if I can get it going on a box of mine. If I can, then good enough and I'll try running it there for a while. I'll convert a few processes over and then see how that flies for a few days. If all looks OK, I'll convert over simply to await the day that the MindAlign guys get things worked out.

UPDATE: so I got the code and configured it by reading every single line. And boy, oh boy... you have to read every single line as there's a line in the config file that will crash the server intentionally if left in the config file. So, read and configure.

The big change was that the default IRCD-hybrid 7.2.3 does not allow for NICK names to start with an underscore (_). Since all the bots for MindAlign have to start with that, I needed to get into the code and fix it. The relavent code snippit was originally:

  int
  valid_username(const char *username)
  {
    int dots      = 0;
    const char *p = username;
 
    assert(p != NULL);
 
    if ('~' == *p)
      ++p;
 
    /* reject usernames that don't start with an alphanum
     * i.e. reject jokers who have '-@somehost' or '.@somehost'
     * or "-hi-@somehost", "h-----@somehost" would still be accepted.
     */
    if (!IsAlNum(*p))
      return 0;
 
    while (*++p)
    { 

and needed to be changed to:

  int
  valid_username(const char *username)
  {
    int dots      = 0;
    const char *p = username;
 
    assert(p != NULL);
 
    if ('~' == *p)
      ++p;
 
    /* reject usernames that don't start with an alphanum
     * i.e. reject jokers who have '-@somehost' or '.@somehost'
     * or "-hi-@somehost", "h-----@somehost" would still be accepted.
     */
    if (!(IsAlNum(*p) || ('_' == *p)))
      return 0;
 
    while (*++p)
    { 

After this change, a simple recompile, and the bots were working great. This would be a nice change to the codebase to put this in the config file, but I can see that this is probably pretty stable, and so they might not want to mess with it.

In any case, I'm set now with my own IRC server for my apps. If they come after me, I have a solid defense - it's for the best for the business.

This morning I was putting in a few changes to a project I was working on and one of the nice features of CVS over Subversion really hit me - CVS allows you to check-in part of your workarea by specifying the files you wish checked in. This is really the core of Subversion - that the entire workarea is a single check-in unit, but what happens if you're in the middle of changes for something and you need to checkin some changes to another totally unrelated part of the code.

Subversion would have to branch or checkout a new workarea, add in only those things that you can checkin at this time, check them in, and then go back to the original workarea and update that to merge in the changes.

That doesn't sound too bad, but it's nice to have several changes in CVS and check in just those that you know are safe for the repository, and then keep working on the others. I know it's part of the "problem" with CVS, but I see it as a genuine feature. I use it all the time.

In fact, the number of times that I checkin everything in CVS is so few as to be considered "rare" - less than 5% of the time. The vast majority of checkins are checking in a few files with a directed comment in the logs, and then repeating as necessary.

So while I like the Subversion feature of keeping a pristine copy of the workarea in the workarea, it's got an offsetting feature in the targeted checkin. At least for me.

[8/5/08] UPDATE: I was talking to a friend today and he mentioned that you could checkin individual files. But I looked at the PDF again that comes from the Subversion project and it doesn't mentioned this at all. I'm stunned that they didn't mentioned this in the docs from the project. It's one thing that really was turning me away from Subversion. Now I can relax and move back towards it as necessary. Wild.