What's it all about, Alfie?

I've been working on the conversion of my Broker code to the new replica set-compatible erlmongo driver. It's not horrible, but the worst part is what the worst part always was - getting the data in and out figured out so I can write the correct matches in the code.

They store the arrays differently, and they don't like the long binary-data strings that I've been using, but after I figured out all the gotchas, it's working again. It's just not easy.

Plus... this is more than half a day shot on something that I didn't need to have done had they not switched the database clustering... and I'm on a week-and-a-half deadline. Not ideal at all.

But it's done.

This morning my newly re-written Broker was not working. Hmmm... so I look at the logs, and it's not too hard to see that there are timeouts on the emongo driver. This was the driver that I used to get from erlang to (and from) the mongo database. Now all of a sudden, it's giving me timeouts. I had no idea.

When the guys that handle the mongo DB arrived today they said that they reconfigured the mongo DB I was using from a single server to a replica set. This is a fault-tolerant mongo DB setup, and in general, I'm a fan of things like this.

But that's not how today went.

Oh no... turns out the API into a mongo replica set is not the same as a single server, and moreover, the emongo driver doesn't support the replica sets. Fantastic!

So now all the work I just finished has to be re-examined in light of a new erlang-to-mongo driver: erlmongo. This guy is capable of dealing with the single server and replica sets modes of mongo DB. This is good, but the API is very different, and most importantly to me, there are no connection pools. If you want to talk to a database, you have to open a connection to the database and then hold onto it.

Given the realities of erlang and persistent state, this isn't really a possibility. We need to have a module that's a connection pooler, and then be able to set that guy up and let him deal with opening the connections, etc.

So here I am, fixing code I just wrote because some other group decided to change the database into an incompatible format. Lovely. It's unavoidable, as it's their group, and their database, but it's a real pain - right on the heels of just getting it done.

This morning I saw that Google Chrome dev 12.0.725.0 was out and the release notes say it all:

All

• Updated V8 - 3.2.6.0
• Spring cleaning in the code, lots of code cleanup and refactoring under the covers

It's nice to see the clean-out of old code. I'm sure there's a lot that's sitting there - like maybe the H.264 decoder... OK, enough of that. In any case, at least they got the icon back. Baby steps.

I finally finished the erlang services for The Broker. I was able to get my ticker plants hitting it and all was running. It's been a lot of work, and not a lot of "new stuff" - face it, it's a re-write, but in the end, we have something that's a lot better than the old, and it's going to be paying benefits long into the future.

Good work.

Today I made some real progress on The Broker. It's still the login service and the configuration service, but I'm getting things together. It's all skeletoned out, so I know what I need to do, and what I need to make it work - in a minimal sense.

Things are looking a lot better today.

It's been another long day of meetings and trying to get work done. For the most part, I've been working on the new Broker, and it's login and configuration services. The trick is currently the mongo-to-erlang interface, and it's proving to be powerful, but not entirely easy to use or well-documented. But it is workable, and that's enough.

Lots of work, but not a lot of code. But I'm getting better at erlang, and that's a big plus.

Most of today has been spent working on the latest rewrite of The Broker - but this guy is the distributed erlang version where the service registry, distribution, routing, etc. are all handled by erlang. This guy will then be a little node on each server that uses or provides services, and it'll connect into a web of other such nodes and share it's state to all so that we can load balance around the net, and provide some nice fault-tolerance.

Today I spent a lot of time working on the login service. There's a lot there because I have to get the LDAP authentication going and then deal with the mongo database - all from erlang. There's a good start, but I'm still very new to erlang, and this is just plain slow going.

This afternoon a co-worker pointed out a problem with a threaded app he was testing. It was using a data feed component I'd written, and it was causing seg faults when it was shutting down. He was having a hard time figuring out why it was crashing, and I was having a hard time figuring out what was causing the state to be reset.

The thread model I was using is a simple class that runs a process() method over and over catching exceptions, etc. until the process() method returns a "stop" flag. Pretty simple stuff. There's the ability for users of the thread object to tell it to stop:

  Thread::setTimeToDie(true);

and the next time it's ready to call process() it bails out and stops. Pretty simple. But it wasn't acting that way in the tests.

I subclassed this Thread for my data feed class, and when I detected that the parent thread was to stop, I stopped some processing sub-threads. The structure is pretty simple - the main thread was handling supervision, a boost ASIO thread was handling the incoming data, and the processing sub-threads were taking that raw data and converting it (two steps) to be used downstream. What was supposed to happen was that the sub-threads were to detect when the parent Thread was to die, and they themselves, would then die.

What appeared to be happening was that the sub-threads weren't getting the message. Or at least not getting it in time. Very odd. If I told the Thread to stop, the sub-threads didn't stop. If I told the Thread to stop, and then did a little shut-down processing, and finally told the Thread to stop again, then things shut down.

It appeared that there's a nasty timing problem here, and I didn't want to leave it at this, but there's no more time today. For now, it's working, but very oddly.

[3/31] UPDATE: Interesting point... this morning, on my walk to the train, I saw it. When the Thread stops, it resets the "time to die" flag to false. The sub-threads weren't seeing it because they weren't checking fast enough. The main thread died, reset it's "die" flag, and the sub-threads just didn't think there was any reason to stop. The fix was easy - don't reset the flag until the start of the next thread. Easy fix.

Today I finished up my part of The Broker - the service registry and the service load (for load balancing) and they were pretty simple modules. That's a good thing because I'm new to erlang, and the simpler the better. Even so, I was introduced to the erlang crash dumps, and picked my way threw a few. Not easy, but once you know what to look for, it's not too bad. Certainly no worse than gdb.

Then I started work on converting the C++ code I had to use the new Broker. It was mostly a gutting as the code I had added for the direct dial was now useless, and that simplified the code quite a bit. Then I needed to add in the ability to send in the load numbers for each service, and I was in business.

I updated my Broker tests - the client and the service, and they now work nicely as well. It's a really good day. Lots of nice progress.

There are a lot of things to like about being a lead developer - you get to have a significant hand in the design of the work you're doing. You get to pick what you want to do, and farm out the rest to others. But there's a dark, ugly side as well - people. Yes, those devices that can't be debugged no matter how long you use gdb - people. And I'm reminded of it today in a very unpleasant way.

The lead on a project is expected to help the junior guys along. Help them learn the craft as it were. They will pick up habits, the goal is that they pick up the good habits of the lead, and then they are able to be self-sufficient in the workplace. Kind of like kids. Being in the middle of that cycle at home gives me keen insight into what's happening to me here at work.

I've got a clingy 5-year old. At home we called it "the five-foot rule". The kids would never be more than five linear feet from us no matter what we were doing, where we were doing it, no matter how big the house was. They were right there. Always. In their early years we were everything - playmate, audience, judge, everything. The same is true at work, it seems.

He's telling me about the status of his work. He's telling me he'll be done soon. When he's done. That he needs to be busy. It's like a clingy 5-year old. I can appreciate that he's trying very hard to be productive and contribute, but he also doesn't realize - like a 5-year old doesn't realize, that there are completely different expectations for him than for me. He's supposed to learn, listen, and if he's really doing his job, be as light a load on me as possible.

But that's not how it's working out.

I'm going to have to slug through this for a while longer. I'm not sure how much longer, but a while at least. If he doesn't get better, then he's the learning disabled kid, and will continue to be a burden on me as long as he's here. That's not going to fly. But assuming he's going to get better, I just have to give him time. It's tough some times.