What's it all about, Alfie?

This morning I've come across a real possible good use for a C++ IDE - renaming a namespace in all the places it exists. I realized that when I had the namespace AppKit in another project and wanted to use it in the greek engine project, I couldn't have eng::AppKit because the compiler couldn't tell what to do with:

  AppKit::Message msg;

Is that AppKit on it's own, or in the context of the eng namespace? I knew which one it was, but there wasn't any way I could fine to make the compiler understand that fact.

Also, I knew that the namespace was badly named. I just didn't understand how widely these things would be used. My fault. So I tore into the code with Vim. There's lots to like there, but doing a massive change like this is just not all that easy. It's not hard, it just takes time.

If an IDE could do that, I might have reached for it. But I didn't know one, so I toughed it out.

This afternoon I got the combined pricing feed for the greek engine working. It's not that hard - just a combination of an existing UDP exchange feed and a receiver of the legacy data with a little NBBO engine thrown in to generate the NBBO quotes for these options. It's nothing that amazing, but it needed to be put together as a unit so that it's a lot easier to use than glueing all those pieces together in the greek engine.

Not glamorous, but it's got to be done.

Another late night trying to get what Joseph calls my Impossible Project done on time. I've finally been able to let the Broker sit for a while, and I'm on to getting the database abstraction layer for the static data we'll need in the greek engine working. I've got a good start on it, and SQLAPI++ really helps, but I wasn't able to really get it nailed down before I had to run for the train.

It's getting a little old - these sprints for the train. But hey... it's part of the job these days. I know it's a nearly impossible timeline, but it's what they've asked for, and I'll do my best to deliver it to them as they asked. That's the deal we all make in a business.

I've got about another hour left on this guy and then my first database table will be done and I can hand it off to someone else for them to work in all the other tables that need to be included in the interface. Once I have all this stabilized, I'll work with the static data vendor group and see about converting these into more generally usable Broker-based data services.

Just for right now, it's enough to get them loaded from the database. We can get the rest done later.

This morning my newly re-written Broker was not working. Hmmm... so I look at the logs, and it's not too hard to see that there are timeouts on the emongo driver. This was the driver that I used to get from erlang to (and from) the mongo database. Now all of a sudden, it's giving me timeouts. I had no idea.

When the guys that handle the mongo DB arrived today they said that they reconfigured the mongo DB I was using from a single server to a replica set. This is a fault-tolerant mongo DB setup, and in general, I'm a fan of things like this.

But that's not how today went.

Oh no... turns out the API into a mongo replica set is not the same as a single server, and moreover, the emongo driver doesn't support the replica sets. Fantastic!

So now all the work I just finished has to be re-examined in light of a new erlang-to-mongo driver: erlmongo. This guy is capable of dealing with the single server and replica sets modes of mongo DB. This is good, but the API is very different, and most importantly to me, there are no connection pools. If you want to talk to a database, you have to open a connection to the database and then hold onto it.

Given the realities of erlang and persistent state, this isn't really a possibility. We need to have a module that's a connection pooler, and then be able to set that guy up and let him deal with opening the connections, etc.

So here I am, fixing code I just wrote because some other group decided to change the database into an incompatible format. Lovely. It's unavoidable, as it's their group, and their database, but it's a real pain - right on the heels of just getting it done.

I finally finished the erlang services for The Broker. I was able to get my ticker plants hitting it and all was running. It's been a lot of work, and not a lot of "new stuff" - face it, it's a re-write, but in the end, we have something that's a lot better than the old, and it's going to be paying benefits long into the future.

Good work.

Today I made some real progress on The Broker. It's still the login service and the configuration service, but I'm getting things together. It's all skeletoned out, so I know what I need to do, and what I need to make it work - in a minimal sense.

Things are looking a lot better today.

It's been another long day of meetings and trying to get work done. For the most part, I've been working on the new Broker, and it's login and configuration services. The trick is currently the mongo-to-erlang interface, and it's proving to be powerful, but not entirely easy to use or well-documented. But it is workable, and that's enough.

Lots of work, but not a lot of code. But I'm getting better at erlang, and that's a big plus.

Most of today has been spent working on the latest rewrite of The Broker - but this guy is the distributed erlang version where the service registry, distribution, routing, etc. are all handled by erlang. This guy will then be a little node on each server that uses or provides services, and it'll connect into a web of other such nodes and share it's state to all so that we can load balance around the net, and provide some nice fault-tolerance.

Today I spent a lot of time working on the login service. There's a lot there because I have to get the LDAP authentication going and then deal with the mongo database - all from erlang. There's a good start, but I'm still very new to erlang, and this is just plain slow going.

This afternoon a co-worker pointed out a problem with a threaded app he was testing. It was using a data feed component I'd written, and it was causing seg faults when it was shutting down. He was having a hard time figuring out why it was crashing, and I was having a hard time figuring out what was causing the state to be reset.

The thread model I was using is a simple class that runs a process() method over and over catching exceptions, etc. until the process() method returns a "stop" flag. Pretty simple stuff. There's the ability for users of the thread object to tell it to stop:

  Thread::setTimeToDie(true);

and the next time it's ready to call process() it bails out and stops. Pretty simple. But it wasn't acting that way in the tests.

I subclassed this Thread for my data feed class, and when I detected that the parent thread was to stop, I stopped some processing sub-threads. The structure is pretty simple - the main thread was handling supervision, a boost ASIO thread was handling the incoming data, and the processing sub-threads were taking that raw data and converting it (two steps) to be used downstream. What was supposed to happen was that the sub-threads were to detect when the parent Thread was to die, and they themselves, would then die.

What appeared to be happening was that the sub-threads weren't getting the message. Or at least not getting it in time. Very odd. If I told the Thread to stop, the sub-threads didn't stop. If I told the Thread to stop, and then did a little shut-down processing, and finally told the Thread to stop again, then things shut down.

It appeared that there's a nasty timing problem here, and I didn't want to leave it at this, but there's no more time today. For now, it's working, but very oddly.

[3/31] UPDATE: Interesting point... this morning, on my walk to the train, I saw it. When the Thread stops, it resets the "time to die" flag to false. The sub-threads weren't seeing it because they weren't checking fast enough. The main thread died, reset it's "die" flag, and the sub-threads just didn't think there was any reason to stop. The fix was easy - don't reset the flag until the start of the next thread. Easy fix.

Today I finished up my part of The Broker - the service registry and the service load (for load balancing) and they were pretty simple modules. That's a good thing because I'm new to erlang, and the simpler the better. Even so, I was introduced to the erlang crash dumps, and picked my way threw a few. Not easy, but once you know what to look for, it's not too bad. Certainly no worse than gdb.

Then I started work on converting the C++ code I had to use the new Broker. It was mostly a gutting as the code I had added for the direct dial was now useless, and that simplified the code quite a bit. Then I needed to add in the ability to send in the load numbers for each service, and I was in business.

I updated my Broker tests - the client and the service, and they now work nicely as well. It's a really good day. Lots of nice progress.