What's it all about, Alfie?

I've been using MacVim for a while and it's really quite amazing. It's Vim as nice as Vim ever was, but it's also a complete Mac app with multiple windows, tabs, toolbar - everything you'd expect from a Mac app. It's really quite amazing.

That said, I've been fiddling with the .vimrc file to try to get it to do a few interesting things. I just thought I'd post them so that if I needed to look them up, I'd have them written down.

Switching Tabs

One of the nice things someone suggested was the ability for Cmd-1, Cmd-2, etc. to move to tab 1, 2, etc. like Firefox does. Cool! I thought, as this will keep my hands on the keyboard even more. The remapping of the key is really two parts: what key combination to use, and what to remap to it? The first answer is that <D-1>, <D-2>, etc. for Cmd-1, Cmd-2, etc. The second part os the command :tabn. Together, these fit into the .vimrc like:

  if (has("gui_macvim"))
    :nnoremap  :tabn 1
    :nnoremap  :tabn 2
    :nnoremap  :tabn 3
    :nnoremap  :tabn 4
    :nnoremap  :tabn 5
    :nnoremap  :tabn 6
  endif

Decent Colors

I know it seems silly, but I've really become used to the colors I've used in Vim over the years. They are called the 'torte' color scheme, and even that's not 100% - the cursor and search colors aren't really right, so I fix them up as well:

  if (has("gui_macvim"))
    :colorscheme torte
    :hi Cursor guibg=red
    :hi Search guibg=orange
  endif

Setting the Tab Name and Current Directory

I wanted just the filename in the tab and not the complete path - even truncated. Also, with a simple autocmd I can make it so that every time I change the buffer (file) I will make the current working directory the directory of the file in that buffer. It's an easy way to make sure that it's easy to add other files from the same directory.

  if (has("gui_macvim"))
    :autocmd BufEnter * :cd %:p:h
    :set guitablabel=%t
  endif

None of these are really earth-shaking, but they do make it a lot nicer to use Vim, and it's amazing that they are all very simple configuration file changes. Amazing tool Vim, and what they've done with it for MacVim.

I was talking to a guy yesterday and something hit me like a ton of bricks - the virtual keyword in C++ really is just about polymorphism (the ability to have a pointer know the right type to call) and has nothing in the order the destructors are called.

In my (weak) defense, I've always used virtual destructors in C++ - just to be safe, but safe from what is the issue? While I knew the reason for the keyword virtual in C++ class definition, I was thinking that there was something special in the use on destructors. Something where it told the compiler to 'unwind' the destruction properly. Not so.

If I have the simple class structure:

  #include <iostream>
 
  class GrandParent
  {
    public:
      GrandParent() {
        std::cout << "constructing GrandParent" << std::endl;
      }
      ~GrandParent() {
        std::cout << "destructing GrandParent" << std::endl;
      }
  };
 
  class Parent :
    public GrandParent
  {
    public:
      Parent() : GrandParent() {
        std::cout << "constructing Parent" << std::endl;
      }
      ~Parent() {
        std::cout << "destructing Parent" << std::endl;
      }
  };
 
 
  class Me :
    public Parent
  {
    public:
      Me() : Parent() {
        std::cout << "constructing Me" << std::endl;
      }
      ~Me() {
        std::cout << "destructing Me" << std::endl;
      }
  };
 
  int main(int argc, char *argv[]) {
    Me    a;
    return(0);
  }

we get the output:

  peabody{drbob}30: a.out
  constructing GrandParent
  constructing Parent
  constructing Me
  destructing Me
  destructing Parent
  destructing GrandParent

Which confirms the conversation I had with this C++ developer that the only reason that the keyword virtual would be used on the destructor is if you had a GrandParent pointer that was really pointing to an instance of Me and you wanted to make sure that the destructors for Me and Parent were called along with the destructor for GrandParent.

I made the silly assumption, and this guy was right. I'm glad we had the talk we did, as I'm going to question a lot of the assumptions I've been holding on to regarding C++ coding because of it. Interesting talk. Never too old to learn.

This morning I got news that one of my cron jobs was behaving badly - and by badly I mean very badly. The script was built to clear out the logs of some very verbose processes that are really not very necessary to monitor, but if they get too big and can't be written to, will crash the processes. And those processes are important.

The problem was that when running the script I made the assumption that I was in the directory I was in when I ran it manually. This was a big mistake and all the scripts I typically create have at the top:

  #
  # Now let's get right into it...
  #
  program=`basename $0`
  dir=`dirname $0`
 
  cd $dir
  echo 'Running '${program}' from '${dir}

so that we get into the directory of the script and then all the moves are relative to that directory. The problem was that this was clearing log files but the names of the log files weren't specific. This means that without this code, the directories I tried to clear weren't there, and then without sufficient checks to make sure they were there, I started clearing out all the files from the $HOME. Not good.

Thankfully, we have .snapshots of all the directories so that everything I cleared out was easily restored. But I needed to fix things so it didn't happen next week. In addition to the code (above) I also added in checking code for each directory so that I know the directory exists and don't get into the situation where I think I cd somewhere, and don't really move at all.

It was funny, in a way. It's been a very long time - about 15 years, the last time I remember making a script that wiped out data accidentally. Then, too, I was able to get it back from backups, but it was still a shock those first twenty seconds when it was gone and there was no 'undo' to easily bring it back. Everyone makes mistakes, me too. Live, learn, and code better next time.

Over the past two days I've been forced back into a project that I have been out of for a very long time. There's a need to get something done quickly, fixing a long lost (and wished forgotten) developer's code, and so I get pressed into this. Whatever... it's not what I wanted to do, and I stated that I was probably not the right guy for the task, but was forced into it anyway.

When I originally started this project - Oh so many years ago, my partner and I decided to make it cross-platform: Windows and Linux, IE and Firefox, because you never knew, and maybe someday we'd move to Firefox. So we made it that way. Things progressed, he left, another guy came, I gave the project to him, and I moved on.

Now I'm back, and I'm very surprised at what I'm seeing. Gone is the cross-platform capabilities - it's IE only now, Baby! It's not that hard to make it cross-platform, it just takes a little bit of effort. Why they didn't put this in, I can only guess - like most people, they all are looking for the easy way out. Since they don't have any users running Firefox now, they don't need to worry about it, so don't even spend a second thinking about it.

Problem is, it only takes a few minutes to get things so they work on both platforms. It's not that hard. I had to change a little Javascript to get the applet tags written out properly. I added a little compressed string code to the applet writer to make it faster to download and pretty much bing! it's done. Not that much to it. But you'd have thought it was weeks of work that just weren't worth it.

It's always worth it. Forget that someday you might actually need it, there's value in making something a little more generic than you need. Just in case you want to see if it's the platform or your stuff. Isolate the problem or variable. Easy. But I guess they are so wrapped up in IE they don't even care to look outside that box. I understand the logic, I'm just disappointed.

It's like the line out of Illusions by Richard Bach - Argue for your limitations and sure enough, they're yours. It's as if they wanted to make the system less capable. They aren't bad people, just not really making any effort to see beyond their immediate horizons. It's a waste.

I'll finish this page and then bow out and hopefully not have to deal with it for another several years. It's just not fun to come back to a system you had pleasant memories of to see it's all decayed and in need of repair. Sad.

Today, while doing a little work on a web page, I ran into a bug in the Sun Java plug-in for Firefox on Linux that I've known about for quite a while. Basically, if there is a very long PARAM tag on the applet, the plug-in gets stuck in the read loop parsing the data. The Firefox developers have confirmed this - saying it's not in their hands. It doesn't appear on Windows - only Linux. Sad, but true.

Anyway, I was thinking that maybe it was just the length of the tag, and a way to make it shorter would be to compress it and then decompress it in the applet. Since I've got a BKCompressedString from previous XML work (large XML files compress very nicely) I decided to add this to the web system to see if it would buy me enough headroom to get the job done I needed done.

The tricky part was that I needed to have something that could pass through HTML, so after compressing, I had to Base64 encode it, and then decode it, decompress it in the applet. Not a problem. In keeping with the other serialization methods I've put in BKit and CKit, it was easy to put in the serialization scheme on the BKCompressedString and it should work.

Almost.

OK, not even close.

The compression and decompression was lossless. The Base64 encoding and decoding was lossy depending on the data you were giving it. I spent a lot of time working through the bits trying to figure out what the problem was only to be bitten by this lovely little Java byte primitive feature.

If I have the code:

    // mask these into the four 6-bit chunks
    dest1 = (byte) (src1 >>> 2);
    dest2 = (byte) (((src1 & 0x3) << 4) | (src2 >>> 4));
    dest3 = (byte) (((src2 & 0xf) << 2) | (src3 >>> 6));
    dest4 = (byte) (src3 & 0x3f);

and the variables dest1, src1, etc. are byte values, then one would think that the right shifting in the middle lines would obey the logic that an 8-bit value (a byte) would. What I found out was that based on the data I was converting, the byte values either weren't actually only 8 bits, or the shifting was adding in a little something special - because there were ones getting put in the shifted bits where, logically, nothing should be.

When I changed the code to look like:

    // mask these into the four 6-bit chunks
    dest1 = (byte) ((src1 & 0xfc) >>> 2);
    dest2 = (byte) (((src1 & 0x3) << 4) | ((src2 & 0xf0) >>> 4));
    dest3 = (byte) (((src2 & 0xf) << 2) | ((src3 & 0xc0) >>> 6));
    dest4 = (byte) (src3 & 0x3f);

then everything worked fine and the encoding was lossless as well and then the compression followed by the encoding and decoding and decompression was lossless.

It took me the better part of 3 hours to figure this out. I was stunned when it finally presented itself, as I initially thought that the bit-wise operations on byte quantities were flawless. I know better now.

I've been working pretty feverishly the last several days on getting my market data server updated with a new API from one of my data vendors. It's been a significant update because of the fact that in this new API they have implemented (and forced us to enforce) login validation and data permissioning. I don't begrudge them the validation and permissioning because it's their bread-and-butter, and it's up to us to decide if it's too much to deal with, but it's the way in which they are doing it that's a real pain.

Based on experience, exchanges charge fees to see their data. Seems reasonable. But they don't really care if you see 1 instrument's price 1000 times in a day, or 1000 instrument's price 1 times a day - or even if you only see 1 instrument's price 1 time a day - it's still the same price. So the permissioning for standard exchange fee-liability is by exchange. Very reasonable.

Given this, we should be able to have a list of valid exchanges for an individual user and then check each instrument for it's exchange and be done. It would mean that we'd have on the order of a few dozen exchanges per user - a very manageable set. But this vendor isn't like that - they want to be able to know exactly what instruments you're asking for, so you have to permission the data at the instrument level. This, too, isn't horrible, but it does mean that instead of having a few dozen things to track for a user, you now have upwards of a few hundred or more. It's still possible, just a lot more work to do with each request.

Then there's the way they do the permissioining... a user has a list of entitlements, and each instrument has a list of entitlements and we need to find a common entitlement between them for the user to see the instrument's data. Why not have a single ID for a user - or an instrument? Why have lists? I'm sure you can have the degenerative case of one item in a list, but from the looks of things, that's not the way it's going. Just really odd. Not impossible to work with, but odd.

So it feels good to be out from under that weight. It's all working, and we'll be able to move to the verification phase with the vendor (certainly understandable) and then on to the roll-out to production.

Today was one of those days that seemed to drag on for a very long time. The progress was there, but it was very slow at times. There was several hours of refactoring on some requests to the server to make them fit the style of the other requests, and then the addition of another request type was similar to the last, but different enough to make it hard work.

It's just been one of those days that you feel drained at the end. But you can look back and say Yeah... I'm glad it's over, but I did make good progress. The details are everything, and I'm glad to say that before I leave today all the details of these new additions are taken care of. The clients have the code, and it's working. The logging in the server is up to snuff. The code looks good as well as running well. All the details that make this a good project are there.

But I've left it all on the keyboard, as it were. Gotta go home now and get some rest.

While I'm not a big JavaScript writer, I can see that there are a ton of sites that use this technology, and on that front, it's nice to see that WebKit has gotten a new JavaScript engine that is already significantly faster than the old one, but also has room for significant improvements as well.

This probably won't make it into Safari until a later release of 10.5.x, but it's nice once again to see that the Apple guys are pushing the envelope with this stuff and working hard to get the best and fastest tools to us. Bravo.

Today has been a very heads down coding day where I've spent almost all day getting the new features of the Vendor's new API working in the server and clients of my market data server. I was originally going to have the client-side code handle the authentication as I've done that in the past with pretty good success. But I know that this vendor is not really big into speed on his responses, so I was hoping to be able to build this all on the server-side and then cache the positive results for a day so that subsequent requests for the authentication would not hit the vendor's back-end systems all day long.

I was happy to see that they had what I needed - even if it took a little poking and prodding to get it to work, and know what data I needed to pass to it. In the end, I was able to incorporate it as a regular server-side feature which is going to make the cache a very welcome addition. Not the least of which is that we no longer are limited to their client-side code and response times.

With all this put into the market data server, I'll start tomorrow on the authorization - or the exchange-based filtering of the data. Again, I'm hoping to find it in the server, and I have a few leads to chase down before I'm sure, one way or the other, that I can or can't do it. It sure would be nice, though.

We have been forced into updating a Data Vendor's API because they never really envisioned that we'd use their desktop product in quite the way that we had - which is to say, built a server around it and made the content accessible to a large number of applications for the traders. So, when they saw what we did, they decided to make that unsupported, and brought out a product to be used in the same manner - for considerably more money. Surprise!

So, we had to upgrade, and in this upgrade, they biggest changes are in the authentication and permissioning of the data. It's understandable, they need to protect the data from unauthorized eyes, etc. The problem I'm running into over the last few days is the way they are having me (and all the other API users - all eight of us) do this authentication. First, everyone has to have a login to the service. Seems that's a good place to start and end. But no. They have an underlying 'ID' - that's guaranteed unique. Seems that's a good place to end then. But no. The minimal authentication is done in their Java client with an ID and the IP address of the machine they logged in on.

Now it seems to me that they already know this, and all they really need is the login. The login leads to the ID, and the login tells them the IP address of the machine they are logged in on. The way they have done it is doubly redundant. That's taking a bit of doing, but it gets even better in the C API.

In the C API - if I'm reading the documentation correctly, you need not only the ID, but four other pieces of information that are known by the service but I have to provide and have no way of knowing programmatically. I have to have the user go to a page on the service's application, read off four multi-digit numbers and then feed them to the C call.

That's an embarrassment. I can't believe they think this is a good scheme. I'm hoping that these four are redundant, or unnecessary, and the docs are simply out of date. But I have a feeling that they aren't and this is just the "way things are" with this vendor. I'm really shocked at this because I had a pretty decent way to have my market data server incorporate this into it's core so that data coming out of the server could be filtered based on the requested user. That would be wonderful as it would put all the permissioning behavior into one place which makes it far easier to control.

We'll see what I get back from one of their "API Experts". I'm really really hoping I'm wrong here.

UPDATE: I got a little help from one of the experts, but for the most part I have to say that getting bad information is worse than getting no information - which was the case today. I asked for the parameters I needed to pass in and got back a list that wasn't right. Once I figured out it wasn't right and what might the right set be, I was well into the afternoon and yet I did get it right. Then I had to realize that their idea of returning a negative was returning an error and not a negative at all. So I had to code for that as well. It's amazing... the people who created this additional functionality didn't even study the old style of doing things and it makes for a very clunky interface to the system. It's working, but as I had expected going in, it's not easy and it's a pain to make it look pretty.

Tomorrow I need to wrap this authentication into a request class and then integrate it into the market data server and finally add caching on the results so that multiple requests in a day don't need to each go to the service to be resolved. Once a day is going to be enough.