What's it all about, Alfie?

This morning I spent the entire morning messing with getting a new machine at The Shop working for me. I'd spent a little bit of last Friday making sure all the software I needed on it was installed, but that didn't prepare me for what I was to experience first hand with Windows 7.

Normally, I'm a live and let live kind of guy for work machines. It's what the company wanted to get, it's what tools the company has supplied, and so I make due with it because that's really the only possible outcome. If I dislike it that much, then it's my choice to leave and look for a place that provides the tools I like. I've turned down a lot of positions in the last decade for Windows development jobs - I just don't want to do it.

So today when I got the new machine - and by 'new' I mean 'different' with Windows 7 installed, I set about making it workable with the software and set-up I needed. A few things I couldn't have done remotely - like the virtual windows and such, but no matter, I had penciled in the morning for just such activities.

So I got the box, and while I'm not a great fan of the look and feel, I can see it's positives, and understand why they did it. But on the hardware that was provided, it was quite literally, a joke.

I am required to use NX Machine to get to my linux servers, and when I did that, I was stunned to see an 80-line Vim session take 2 seconds to redraw. The full screen was on the order of 10 sec. It was amazing... and totally unusable.

I tried all the things I could find on their web site - nothing to address the horrible slowness of the redrawing. Then someone pointed out a setting to skip DirectDraw on the refreshing. It seems it's the "budget" video card and Windows 7 that are at odds here. Had my machine had an accelerated video card, I'd have been fine. But with the hardware I had, I was forced to change the refresh method.

Thankfully, that worked. A little later the head of desktop support stopped by to ask how it was going. I had shared my problems with him earlier, and I was glad to see he was following up. We talked, and he pointed out that he turns off all the "new" Windows 7 features to get his box working quickly again.

Amazing... I go back to a Windows NT look-n-feel to get Windows 7 working OK. I'd have been better off sticking with XP! Really. The cost, and the time, and the now degraded UI... it's a joke. An honest to goodness joke. I don't expect the company to spring for all new hardware because of Windows 7, but they should have had a far more graceful degradation of performance so I could look like XP, and have the Windows 7 engine, as opposed to dropping all the way to NT.

But hey... I'm sure there's tons of people saying "Buy a $100 video card!", and they'd be right. For me. But for the hundreds of other machines? Now we're talking real money. Not so easy to say "Buy $30,000 in video cards so Windows 7 is nice!" But that's really what Microsoft is expecting you to do - invest in more hardware to get the user experience they ship out of the box.

This morning I finally finished up the client library I've been working on for a while. I finished up yesterday with it functional, but not the way I wanted it. I spent a little time this morning cleaning up the code - adding comments here and there, and clearing out the logging using only during the debugging phase.

Nothing very exciting so far...

Then I was beginning to think that I might be able to get the best of both worlds with the existing design. One of the problems with my one request - one socket is that we can possibly end up with a ton of sockets in use. We can also be spending a lot of time creating them, only to tear them down right away. Not as efficient as it could be.

Yet I didn't want to have the one bad apple problem, so I think I've got a nice idea that I'm going to be putting into the codebase this morning - pooling of sockets. It's a simple thing - have a list of "unused" socket connections, and when a request comes in, grab one from the list, or if there are non there, create one. Then, as you use it, great. Everything is isolated. Then, when you're done with it, return it to the pool, and it'll get re-used the next time someone needs one.

Very simple, but the socket stays open, all the resources stay allocated, and in general, we have the best of both worlds. Very nice. That's going to be my morning - pretty excited about it.

I've got an interesting problem that I sure wish I'd checking into the linux C libraries a lot more before getting too far into this, but so it goes... It's the old UUID problem. It's a 16-byte, unsigned number, but it's far too big to hold in one atomic data element, so it's often a union:

  union UUID {
    uint8_t   bytes[16];
    uint16_t  words[8];
    uint64_t  blocks[2];
  };

but the problem with this union is that you can't do something like this:

  UUID   a;
  UUID   b;
  a = b;

or:

  UUID   a;
  UUID   b;
  // ...
  if (a = b) {
    // do something...
  }

C++ just doesn't allow for the default operator=() or operator==() overloading like it does the default constructor, the default (shallow) copy constructor, etc. You have to build these things into a class.

However, there is one nice thing that these unions allow: it's the un-biased use of the union as the start of it's member data. For example, if we have:

  UUID   a;
 
  // ...
  memcpy(&a, src, 16);

we can write the 16-bytes of data into the union's elements by simply referencing the UUID itself. This is not the case with a class. There's the vtable, and that ends up throwing off all the offsets for the ivar data.

But what can a guy do to get these methods on a union?

The Answer is really the anonymous union:

  struct UUID {
    union {
      uint8_t   bytes[16];
      uint16_t  words[8];
      uint64_t  blocks[2];
    };
 
    // the constructor/destructor set...
    UUID();
    UUID( const UUID & anOther );
    UUID & operator=()( UUID & anOther );
    UUID & operator=()( const UUID & anOther );
    // the equality/inequality operators...
    bool operator==( const UUID & anOther ) const;
    bool operator!=( const UUID & anOther ) const;
    bool operator<( const UUID & anOther ) const;
    bool operator>( const UUID & anOther ) const;
    bool operator<=( const UUID & anOther ) const;
    bool operator>=( const UUID & anOther ) const;

where I added in the inequality operators so the UUID could be used in a std::map as a key. The problem now is that I'd really love to have the following still work:

  UUID   a;
 
  // ...
  memcpy(&a, src, 16);

but it won't. I have to reference the first part of the data to get past the vtable, etc.

  UUID   a;
 
  // ...
  memcpy(&a.blocks[0], src, 16);

It's not horrible, but it's no better than really building a class. I got the syntax I wanted, but the consequences weren't without their costs. Kind of bumming to get it all going and then remember the vtable, but that's life.

Well... the 'media event' was interesting to me primarily because it was a wonderful example of streaming done right. The stream was very fast to start, and while there were a few hiccups in the middle there, it was, by and large, a fantastic display of Apple technology, and I loved it.

Am I going to buy a new iPod? Nope. But I did like the new Apple TV, and at $99, am honestly considering getting one for the movie and TV rentals. It also supports Netflix streaming, so I can see that they are getting to the point that it's more of a display device as opposed to a gateway to the iTunes content. Nicely done. Add Hulu and you're in business.

I'm interested in the addition of 'Ping' to iTunes, but I'm not sure how many "friends" I'll have, or who in the world would be interested in what I've got in my library, or what I'm listening to. But there are certainly aspects of the public that are all about this - I know of three - all teenagers. So that's nice.

In all, an interesting keynote, and I'm looking forward to the new iTunes 10.

Today I spent the entire day banging away hard at a new client library to a data system that another group in The Shop has created. They built a Java client, but I needed to have something in C++, and have it interface nicely to the variant data type that I've already created. Someone working with me started the work on this client, but then was called away on another project because it was having lots of problems in production. So it goes.

So I took the start he had on the client library and took a good, long look at the code to see what needed to be done, and what was really looking really pretty good. As is so often the case, the start he'd gotten was OK, but it wasn't really good enough for the kinds of conditions that I'm used to seeing, and maybe that explains the problems in production a bit as well. So it was a total do-over.

The key points of the service, insofar as the design is impacted, is that there are socket connections, then on a single socket connection, there can be multiple simultaneous calls and subscriptions. The call is a one-time subscription, which returns a single result, whereas the subscription will send updates to the client with every change to the source data that generated the result.

It's a nice model, and the only problem I see with it is that boost asio is set up to have a single buffer of the incoming data, and so it's possible to really have that one buffer, or socket, be a bottleneck. Additionally, if there's a problem in my request, it's possible that the service will not know how to respond, and it's only solution is to kill the socket. That's a bit drastic, but I can understand why - the service doesn't know how to respond to me, so it's only solution is to tell me "that's VERY bad" by killing the socket.

With that, all my pending calls and subscriptions are dead, and so it's possible for one bad apple to spoil the whole bunch. Not my idea of a great way to go.

So what I ended up with as a first cut on this is to have an "updater" object that has a socket and all the supporting data with it. Then, a single call is a single socket, a single buffer, and a target variant. This means that as data comes in, boost's asio can handle as many of these as necessary. Much cleaner, but there's a cost - multiple open sockets, and starting one up isn't fast.

Still... it is a good start, and I spent today building that. At the end of the day, I had something that was working, but needed to be cleaned up a little bit. Still... stayed late to get that going, and was very happy to see all the bytes line up and get back solid responses.

Today I spent a good bit of time working on extending the variant class I have in my ticker plant to handle byte arrays and then adding in the map key space encoding and decoding. The idea is that map keys really don't need to be just any string - they can be a limited subset of the ASCII space, and in making that a limited set, we can pack more characters into fewer bytes, and then unpack them once they are on the receiver's machine.

Say we limit the ASCII space to 64 characters - any 64, really - just so long as there's a mapping from the 255 ASCII values to the 64 acceptable values, and back again. At that point, we know that we'll be able to store the mapped key space in 6 bits - 2⁶ is 64. So if we look at a series of three bytes we can pack four of these characters into that 24-bit space:

The code for the conversion is pretty simple - look for now many even 'blocks' of four characters to encode, map them into the limited key space, mash their bits and pack them into three bytes. The remainder is a simple process of doing part of that.

The decode is simple as well - we just need to have some terminal condition - and that can be a simple 0x3f (all 1s) in the last (terminal) 'character'. So we look at the length of the byte stream, see how many "full sets" of three bytes there are - convert each into four characters, and then based on what's left, we have only a few options. It's pretty simple to decode, and you have the original string back.

I needed to get this all in the variant class as the next thing I needed was to add in the update stream that will be sent by the data service in response to updates at the source. This is similar to the kind of updating I did in Java back in InfoShop: there was a HashMap that understood a 'path' concept, and transactions. You could put a map into a transaction, do things to it, and then commit the transaction and get a transaction log that could be sent to remote copies of the same map, and be applied to bring it up to date.

It worked very well for me at the time, and it's something very similar this time. There's no transaction, and the updates are all streams with an action, a path, and an optional value. It's simpler, but for the data sets we're dealing with, it's sufficient.

I got a lot of the decoding of the update stream done, but I think it's time to put together a client and see how it functions to see what I'm really getting from these servers. There's nothing like the socket-level byte stream to remove all the questions and ambiguity in the specification. So that's going to have to wait for tomorrow.

I believe, though have no factual data to support this belief, that most developers these days are learning to code on Windows with some IDE - like Eclipse, NetBeans, etc. It's the easiest way to learn to code, I'll grant you, and it's the cheapest platform to use. Heck, you can get a decent Windows laptop at BestBuy for under $800. Peanuts.

But there's a good contingent that seems to have fallen in love with the Free Software movement, and picked up linux to put on their $800 laptop, and are using Eclipse on linux. This is not bad. But it's the clash of these two worlds that often leads to problems.

Then again... it could just be that the developers aren't all that good, and need to be whipped into shape before they're set loose on the codebase.

So here's what happened today: I'm trying to clone this git repository onto a windows box because the way I have the monitors hooked up, it's easiest to see the code from the Windows screen. My first problem was with the git that ships with Cygwin. Turns out, if there's a problem with the repo (as there was - sort-of), the Cygwin git gets all messed up. Maybe in a subsequent release they'll get it fixed, but the consensus at The Shop is that the better solution is to get msysgit from Google Code.

(As an aside, I've also decided to upgrade my Cygwin to 1.7.7 as it seems I was pretty out of date, and that could possibly have contributed to the problems.)

So I clone this repository and I find that I've already got a changed file. Hmmm... that's odd... so I check that file out again... still changed. Very odd.

Turns out, the developer had created two files - one a Java source file, the other a shell script - one named MyTest.java, the other myTest.java. OK... that's got all kinds of wrong written all over it. Case is not the way to distinguish files - not in the multi-platform world. And who makes a shell script have an extension of .java?

When I pointed this out to him, he cleaned it up and there wasn't a problem any longer. But it reminded me that what I take for granted is not exactly universally understood.

Today was a day I'll be glad is over... there were just so many avoidable problems and delays that it makes for a day that really is best forgotten as soon as possible. It started out reasonably well - I was finishing up the work I'd start on Friday. I had really wanted to get it done on Friday, but it took another three hours (roughly), so it would have been silly to really stay and see it through. Also, there was no way I could have tested it Friday evening - and this morning I was able to use the live data to verify that things were working as planned.

But at this point, things were still going pretty well. The code was done, everything tested out, I checked it all in and pushed it to the central repo - pretty nice. But then the avoidable stuff started to bite me.

I needed two machines to test the throughput of ZeroMQ as a reliable multicast distribution system for my tick data. Nothing fancy, but I needed to have some way of replacing 29West as we weren't really using it as a solid middleware - just a multi-channel reliable multicast system. Given it's limited usage, I looked at ZeroMQ and thought Hey, if this works, I'm in business! But in order to know if it'll work, I need to actually get the ticker feeds working, put the messages into ZeroMQ, and pick them up on another box. Hence the need for two boxes.

Well... they got a few boxes with 10Gb ethernet NICs in them to make sure that I didn't have to worry about the NIC being the bottleneck, and they were ready for me to check the boxes out. As per the way things are at The Shop, the standard mechanism for getting to these servers is NXMachine or SSH. Given that I'd be testing and building, I decided to go with NXMachine. It installs pretty easily, and with this simple fix, it should work just fine.

Silly me...

I spent a full morning trying to get the NX Server to work. I knew the client was working, and I knew the server could work, but it wasn't allowing me to get a complete connection. Well... I got the connection, but when I went to actually display the X session on my box, it disconnected me. Very odd.

I tried logging the NX Server - no luck even when following directions. I tried re-installing the software - no good, either. I tried different parameters for the client - no good. In the end, I went to my boss and asked him for help. He couldn't get in either, but then realized that maybe it was because GNOME wasn't installed. Specifically, the GNOME Desktop environment.

That was the problem.

When he did a simple:

  $ yum groupinstall "GNOME Desktop Environment"

he got some 124 packages that needed to be installed. It seems that they didn't put the full GNOME install on the box as it was a "server". My previous box had the GNOME Desktop installed prior to me using it - which is why it worked. Had I stuck with a bunch of SSH sessions into the box, it would have been fine. It was just the desktop login that was the problem.

After that was solved, I was able to install boost, log4cpp, ZeroMQ, and a few other things to get this new box to the point that I was able to verify that all the code worked, and that everything compiled and ran.

Lots of grief for something as simple as not having the login desktop stuff installed.

Today was the final push for the last two codecs I needed to write. I was really hoping to get both of them done, but the amount of code required in the first was just too much to get both done in a day. I had to send an email saying that I had slipped on this guy, and that I'd get back to it on Monday.

Not really hard, just a lot of code to get the messages decoded. Bummer...

The second nice update on the browser front this morning is that Google Chrome dev is now at 7.0.503.0. This represents the jump of the major version number after the release of Chrome 6.x to "beta". While I'm not unhappy with 6.x, I think the major updates the Chrome Team puts into the browser will be in the 7.x branch now, so I'd like to stay on the 'dev' branch as long as it's even reasonably stable. It's good to see them pushing forward.