What's it all about, Alfie?

I've been trying to get a handle on what's in the current version of The Magic Schoolbus and it's hard. I mean it's a lot harder than it has to be. There are complete non-template implementations in header files, there are classes grouped - some logically, some not, into the same files... it's a mess. Trying to see what's happening - a real important thing in OO design, is next to impossible.

Many people have criticized me for making my code too verbose. Maybe it is. But every one of those people were never trying to understand it. They were trying to shy away from it as a coding standard, and simply write less. Hey, I understand lazy. It's easy to understand: You want to do less than you have to. Easy. But it's always going to cost you in the end.

Take this codebase... if they had taken the time to make header and implementation files for each class, then it'd be a lot easier to see what's happening. I wouldn't have a 277,000 line header file that's really a header with all the implementation in it. I'd have a good set of headers for use with a pre-compiled binary library and people would be able to use it like they should.

But that wasn't the path chosen.

I want to choose the better path. So I'm writing the entire thing over from scratch. Using Boost every single time I can to make it portable while not sacrificing capability and speed. I'm going to make this project something to be proud of, and I hope, I really hope that it catches on.

If not, I'm still going to do it.

That's just the way I roll. Baby.

Today I started the process of trying to consolidate the 300+ messages in The Magic Schoolbus into a few reasonable categories: OPRA messages (tons of them, space is critical, data format very rigid), Price Messages (little looser, but still important and small), and everything else. The remainder of the messages are really suitable for fitting into self-describing message formats like JSON, or more likely BSON, as they are very flexible - have variable number of components, and don't need to get shot around the network all the time.

The Really Wasteful

Take for instance, the Holiday Calendar. This is just like every other Holiday Calendar I've ever seen: give it a date (or default to today, and it'll give you all the trading holidays for the next 'n' months. Very simple data structure. Even simpler when all you're talking about are US Equities and their options - you don't even need to tell it which exchange you're asking about as they are all the same.

But here's what The Magic Schoolbus does: Every minute it will publish a list of all holidays for the next ten years and those that are registered for this data will receive it. Over, and over again. Every minute. The format is pretty simple as well. There's the basic header of the message (far too verbose and general) but the payload of the message looks like this:

  struct {
    uint16_t       modifiedBy;    // trader ID
    char           today[9];      // YYYYMMDD
    uint8_t        numHolidays;   // # of holidays
    Holidays_NEST  holidays_nest[];
  } HolidayCalendar;

where Holidays_NEST looks like:

  struct {
    char      holidayDate[9];   // YYYYMMDD
    uint8_t   holidayType;      // 1=no trading; 2=half day
  } Holidays_NEST;

Now even if we put aside the problems with this content - like a date that's 9 bytes when 2 would do (as a uint16_t) - in fact, we could compress the entire message to look like this:

  struct {
    uint16_t    modifiedBy;    // trader ID
    uint16_t    today;         // YYYYMMDD
    uint8_t     numHolidays;   // # of holidays
    uint16_t    holidays[];    // tYYYYMMDD
  } HolidayCalendar;

where the 't' is the type of day and the date immediately follows. A simple mask gets us what you need and size comparison (assuming 64-bit pointers) is:

  old size = 12 + n * 10
  new size = 5 + n * 2

and for a typical year we have, say 7 holidays, and ten years, so n = 70:

  old size = 12 + 70 * 10 = 712
  new size = 5 + 70 * 2 = 145
  savings: 79%

It's just stunning how bad some of these messages are.

The Horrible Congestion

Look again at the Holiday Calendar - it's sending this data out every minute. Why? Because the designers believed that this was the only way the data was going to get delivered to the client. What about a data cache/data service? They even have a cache server in the architecture - but it holds all the messages sent and as such, it's not nearly as efficient as a more customized data service.

So I need to do something here - basically, stop the insanity of sending all this data all the time. I need to have the client get it when it requests it and when it fundamentally changes. This means something a lot more intelligent and flexible than read from the database, make a monster message, send it, repeat.

The Task

It's huge. I have to look at all the used messages and then try to see what can be combined into a nice, compact format for sending at high speed to a lot of clients, and what can be more free-form and possibly even skip the 29West sending in the first place.

It's a monster job. But it's gotta be done. The reason this is in such a horrible state is because no one has taken it upon themselves to do this until now. It's ugly, and it's painful, but it's got to be done.

I've been struggling to come up with a way to describe the codebase of The Magic Schoolbus - and it's not all that easy for me. The code has good parts - the guys who wrote it are not without understanding. They have atomic operations from boost, and in places, it's clear that they have been trying to get this codebase up to a very respectable level, and in some places, they have done a good job.

But the real problem is that they haven't been consistent. It's a single codebase with multiple projects - many of which are no longer used by anyone, and there's no consistency in the code. OK, that's not 100% accurate - there's plenty of copy-n-paste reuse where they have taken whole applications in the codebase - copied them, and then just replaced a few letters in the name to make a new class. It's the worst kind of consistency: to make any changes across the board, you have to change everything in the codebase.

There's precious little in the way of real re-use. There's even precious little in the consistent use of types. In many places they'll use their own unsigned integer types, and in others they'll use those in stdint.h. I'm all for using either as there are distinct advantages to both, but you really need to pick one, and stick to it. No matter what.

That's the thing that really gets to me: the lack of consistency.

In many projects I've been in, I didn't like the way the original developer started writing the code, but for the sake of the project's consistency, I stuck with it. The goal was to have my changes look no different from the original code. If I succeeded, then there's only one style to understand. This is far, far easier to grok.

But in this codebase it's like Rube Goldberg gone amok. There are some sub-projects where the includes are in the source directory, and others where they aren't. Some use custom types, others use system types. Some use header files and implementation files, and some use massive structs in header files with included implementations. There's just no consistency.

And it's all very, very big. Like the 277,000+ line message header file.

So what to do?

If we try to clean it up - to really use a well-designed object model, then we're gutting everything. And I mean everything. If we're going to do that, then we might as well start over and make something that's far better - with a client library in multiple languages, and forget having 100% backward compatibility. We'll do our best to make a transition plan, but it's a new era, with better information, better performance, etc.

If we do that, are they going to be willing to go with me? Hard to say. I think they have serious doubts about if it can be done. In that, I have serious doubts if they can do it. If they have no faith that it can be done, then there's no chance they'll actually be able to pull it off.

We need to have a more consistent codebase. It's essential to monitoring, stability, low maintenance, etc. But to do that I may have to Just Do It, and then hand it off to them. That's not really ideal, but it may be the only option.

It's a tough place to be. But I'm glad that I have a good handle on the code, and can vocalize the issues for those that have asked me to look into this. It's not an easy decision, but it's one that needs to be made.

I've been really pounding away at some code that I'm writing to replace big sections of code in the Magic School Bus, and it's getting pretty close. There's a lot to look at when you have to make a class library for a 277,000 line header file. I've built a few classes, but I want to have a really good approach before I bring it to the rest of the Team.

Not bad so far. Pretty decent.

[6/18] UPDATE: having talked to the group, it looks like I'd have more luck getting Microsoft to adopt OS X as it's core for Windows 8 than getting these guys to look at a massive re-write of the codebase. It's almost hopeless. Very bummed.

I'm trying to get a little start on the reformulation of the Magic School Bus project and I've run into the 800 lb. Gorilla in the project - the 277,000 line header file with implementations. It's big, and I need to really attack it because it's the client-facing component of the project, and needs to stay completely unchanged as I go through this redesign so that the several hundred applications that use this in The Shop don't have to be retrofitted with the changes.

This doesn't mean I can't make new and improved interfaces and methods, I just can't change the functional behavior or interface that exists now.

So I'm trying to come up with a way that this quarter million line file can be broken up into real headers and real implementations and once class per pair, at that. Then I'll throw it all into a directory in the project, make a shared library out of it, and it'll be far far easier for folks to use.

Doesn't mean that it's going to be easy to do, just that it's not conceptually hard. Just a lot of work.

Working on code has to be a continual job. Especially if it's an evolving product. You can't just sit back and assume that once it's written, it's "done". It's not. Certainly, there are shared libraries that get to the "done" state - for components. But for a product where features are being added, sources are being added and removed, it's just next to impossible to believe that something written years ago is still going to be relevant and useful years later. It needs to be looked at, checked, and where necessary, updated.

Case in point today was the codebase for an app I've been asked to work on at The Shop. The functional description of the project is pretty simple - get data from some network feeds - like multicast or direct tcp data feed, and distribute it to 29West in a manner such that it can be subscribed to easily, and then the 29West data feeds terminal processes that decode it and serve it up on tcp to the clients.

It's a pretty simple message router with a pub/sub content. Not trivial, as it needs to be fast, but not conceptually that hard. Certainly, I've worked with some guys that would say that it's a simple receiver that rebroadcasts on 29West in a set of channels and the receivers just get it off 29West. I happen to like that there's a terminal app and clients don't hit the 29West feed directly - it allows the replacement of 29West in the event that it comes to be a problem.

In any case, this codebase is C++, but it's written as a lot of C with structs and a few classes, but very little really good design. There are a lot of odd little things based on the source control method - and it's a horrible source control system. There are more little odd things they put into the project for NetBeans. For C++?

OK, I can see people wanting to use an IDE, but why put the IDE artifacts in the SCM? That doesn't make sense. If you want to see something slightly different than I do, then that's OK, just don't force me to look at it that way, too. I don't understand that at all. But that's not the worst of it.

I can see using C in C++ for speed. But it's really a fool's errand. Face it, once the method call is made form the vtable, the execution of any one method's code is really the "speed of C". The issue some may take is that the using of the vtable is more costly than the jump table of C - and in that, they are right. But the time of using the vtable is only on the call invocation, and not in the execution.

Time-critical sections should be within a method, and not calling methods. So in that, you're going to get the performance you want when you need it, and the convenience and design simplicity of C++ when you need it.

But that's still not the worst I've seen in this project which I'll call The Magic School Bus, no... there is even worse.

I was reading the code, trying to get an understanding of what goes where and how it all works, and I ran into biggie.h... it's currently got 277,695 lines of C/C++ code. Yeah... over a quarter million lines in one header file. It's really next to insane. Because it's got the implementation with it, every single inclusion of this file essentially statically links in all this code. It's crazy.

Now I will say that the code isn't horrible, it's a bunch of messages, and it's decent, but it's not using any subclassing, and I'd sure like to do that. It's not using any really good designs, other than it does have a nice suite of operators built into each of the messages. It needs a serious re-write - but there's the rub: if the comments are right, the code is generated by a perl script.

Amazing to me that they didn't make it a pair of files for each message - if they are generating the code from perl, it'd be easy. Make a header file, make an implementation file, and then make a directory where they all co-exist and build them into a shared library. That's what I'd like to do, but I can't make these kinds of changes to the codebase without really understanding the implications. And the implications of this could be severe.

If the clients to the Magic School Bus are expecting to get the complete functionality by just including the header file, and not linking anything, then they need it to be this grotesque blob. But if we can move them to something better, it'll make the maintenance and support so much nicer.

I'm just hoping that we can make this entire message library a lot cleaner and nicer. There's no reason for a 250,000+ line header file in decent designs.

It seems that the auditing of a new IRC server at The Shop is too much to think of adding at this point - given that they have an existing XMPP/Jabber system that's up and running and working just fine. So rather than mess with using the IRC stuff I've used in the past, it looks like I'm off into the world of XMPP and Jabber. Thankfully, there seems to be more client libraries than you can shake a stick at. Including C++.

The one I'm looking at initially is called gloox, and looks to be a reasonable well done library, but I haven't even tried to build it - let alone use it. Still, it looks reasonably well defined and designed, and I'll need to get it, read through the code to see how it's commented and built. If it looks good, I'll use it until it doesn't work. I'll be certain to isolate it nicely so that should I need to replace it, it'll be easy enough to do later.

Still... it looks promising and it means that I don't have to hassle with implementing a new chat protocol. But it might have been nice... I'll never know.

This morning at The Shop I was installing NoMachine on a CentOS 5 server and an XP Client. The client was fine, but I kept getting SSH errors about Authentication failed. I looked at the NoMachine web site, and their knowledge base recommendation was to make sure the permissions on the ~/.ssh/config file were set properly.

Did that, didn't work.

I then stumbled onto the key to the problem: the nxserver wasn't really running. When I found this article, I saw that I had a way to check and see if it was really running:

  $ sudo /usr/NX/bin/nxserver --status

and if all is going well, we'll see something like:

  $ sudo /usr/NX/bin/nxserver --status 
  NX> 900 Connecting to server ...
  NX> 110 NX Server is running.
  NX> 999 Bye.

but if there's a problem - like I had, it'll look like this:

  $ sudo /usr/NX/bin/nxserver --status 
  NX> 900 Connecting to server ... 
  NX> 204 Authentication to NX server failed. 
  NX> 110 NX Server is stopped. 
  NX> 999 Bye.

OK... we now have a way to prove it's the server without including the client. Progress. Unfortunately, the solution was very difficult to find. Forget the knowledge base... that didn't help. Turns out, several of the guys here did figure this out in the past, but have since forgotten the exact solution.

So they had to play around. So they played.

The solution was in the user nx that's installed with the RPMs. This guy doesn't have the proper groups, so we had to manually add him to the localssh group:

  localssh:x:500:dbususer,...,nx

where the key is that the nx user be added (with a comma separator) on the end of the line. Save that, restart sshd with:

  $ sudo /etc/init.d/sshd restart

and you're ready to go. You just need to restart the NX Server:

  $ sudo /etc/init.d/nxserver stop
  $ sudo /etc/init.d/nxserver start

then:

  $ sudo /usr/NX/bin/nxserver --status 
  NX> 900 Connecting to server ...
  NX> 110 NX Server is running.
  NX> 999 Bye.

works just fine.

Whew! That was a ton of work to find the three characters to type into the right fine to get it to work.

I've been doing a little searching today looking for some lockless data structures for C++ because I'm expecting to have to do quite a bit of coding for the new market data (ticker plants) work that should be coming up pretty soon, and I thought it'd be nice to see if there was something that was going to give me a nice leg up as opposed to sticking with traditional locks.

The problem is that the state of the C++ support for this is pretty limited. In fact, there's going to be support for std::atomic<> for elemental data types and even regular C structures. There's no support for the STL classes, but that's a reasonable limitation. The problem is that this is not really the same as having std::map<> and such as lockless structures, but there's no support in the standard (yet) planned, and even Boost doesn't have anything - yet.

It's kind of depressing, as I expected to see some kind of lockless data structures because the Java Concurrent package has been around for quite a while and I would have just assumed someone would have taken the code for that package and converted it to C++. But I was wrong. Shucks.

No matter, I can still make it work, I was just had hoped to see something a little more...

This morning I was trying to get AIM working with Adium 1.4b18 with SSL via the guest wireless access in The Shop. It's not a major deal, but I've had it set up this way for quite a while, and when I realized that it wasn't working yesterday using SSL, but was working without it, I thought I'd spend a few minutes trying to figure out why.

Well, it turns out that when libpurple, used in Adium, connects to AIM, via SSL, it tries the provided machine and port: login.oscar.aol.com:5190, but if it's SSL it falls back (or promotes?) to using slogin.oscar.aol.com:443. Now a typical technique within places that need to log all chat activity is to have the local DNS redirect the requests to such machines to local proxies and there, all inbound and outbound communication is logged for legal purposes.

Makes sense for the trusted network at The Shop, as well. But this is the 'guest' network, and I'd like to get to AIM with SSL. The trick is pretty easy: add the following lines to your /etc/hosts file and then you're in business!

64.12.202.116   login.oscar.aol.com
64.12.202.117   slogin.oscar.aol.com slogin.gslogin.oscar.aol.com

The first is the real address of the Oscar AIM login server, and the second is the name of the secure (SSL) login server. It actually has two names, so it's necessary to put both in the line to make sure that whatever libpurple is asking for, it gets.

With this, I was able to get to AIM with SSL no problem. Nice.