What's it all about, Alfie?

Today has been just a lot of code re-writing. It's all about not being hard enough on the more junior members of the team. I want to lead them down the path of doing the right thing, but along the way, it's painful when they keep missing the points I'm trying to make.

Documentation is easy to get right - it's what you're trying to do. Coding it up is a lot harder. But why then, is it that someone who isn't doing the latter, fails to do the former? I'm trying to teach them that rather than making two mistakes, get the documentation right and then you're only going to make one - max. I'm afraid it's just going to be a long and slow process.

But I'll get there, it's only a question of time.

I've been on a very accelerated timeline for a project that has the potential to make a very big impact to The Shop. But I've been amazed at the number of times that I've wanted to scream at the guys on my team about making sure they're doing the right things. I guess I take it for granted that developers know what it is they should be doing. In general, I've seen that not a lot of folks really do get it, and so it's clear why so many people fall back on process.

It's the way to ensure that things get done. But it's based on the idea that you don't really have skilled labor - you have a bunch of advanced beginners, and they need all the rules and regulations because that's how they code.

I understand why people push for this - it makes it appear that you can get good work out of marginally skilled workers. But you can't. Not really. The rules can't cover every possibility, and there will be things that fall in the cracks. There's no silver bullet - you have to have good, skilled people, and then you don't really need the rules.

Unfortunately, I'm not "there" yet. So I have to try to make the most with the people I've got.

Today has been a hoot. I've gotten to the point that I'm able to glue in a major piece of the puzzle into the framework I've got: I can now load up the slice of the universe I need to cover and the clients can hit it and get what they need while at the same time not crashing the system with nasty allocation/deallocation errors. OK... there was a little bug hunting I had to do for that last part, but still - I got it.

The only thing I'm missing now is the ability to do the calculations and generate the messages. I'm letting another guy finish that and hopefully, he'll be done very soon and tomorrow I'll be able to have a complete functioning system. That would be great.

Great progress.

This morning I saw that Google Chrome dev 13.0.767.1 was out with a few nice updates, and something I hadn't noticed in the past - significantly better restart rendering performance. I mean Wow! When I restarted 13.0.767.1 the first page was there. This is in start contrast to taking a few seconds to get that first visible page up. Now I don't know if it's just cached, and that's OK, but either way, the impression is that it's UP, and fast. Very nice UI choice.

Today I spent all day re-writing an instrument loader because over the course of the last two weeks the junior guy on the team that I handed this project to has made a complete and total mess of it. Each day, I asked him how things were going. Each day I got the same "looking good... almost done". Each day, I fear, it got a little worse. Finally, the project was waiting on him, and I had to step in.

I should have done so much earlier. My fault.

I like this guy, I really do, but this was a train wreck, and he should have asked for help after the first few days. There was shared state in the loader methods that would have failed had even two clients tried to load anything. In the tests he wrote and used, this never came up, but it took me less than 5 minutes to see this problem, and realize the depth of the problem.

This is the second guy on the team that I needed to put on test writing duty for a while. I should not have allowed it to get this bad, but I did, and that's my fault. I don't believe I shoulder all the blame, as I asked how it was going and he didn't speak up - but that's inexperience for you: overestimating their own skills, and underestimating the problems they face.

So today I was coding like a madman trying to get something that worked for the problem at hand. Thankfully, I got it working, but it took a full 13 hour day to do it. From here on out, I'm going to keep his work to finishing the skeletons that I write for him. That way, I can limit the damage.

Late today I had to start the process of re-writing two very important parts of the greek engine that I had given to another guy on the team. I don't like doing this for a lot of reasons - not the least of which is the fact that we lost a ton of time. I blame myself far more than I blame him - he was doing his best, and trying to get it all done to impress me. I was hoping that his little "updates" were correct, and I failed us both. Badly.

So I had to take it back, and I had to start looking at what was and quickly think about what I wanted it to be. The job I set before myself this evening was to get the "map" of Instruments in order. This is basically a trie with a thread-safe list that are going to hold the families of instruments - the stock and all it's futures, their options, and the options on the stock. It's a bit of a mess, but that's the nature of the beast. The idea was that this structure was going to make it easier to deal with these families.

There were a lot of things that were very broken. It took a couple of hours, but I was able to clean this up and cut out a lot of mess. Getting it to a point that it can do the job for me starting tomorrow when I have to handle the really nasty problem - the InstrumentFactory.

That's going to be nasty.

I spent the first part of today working on getting the serialization scheme of the Instruments going so that I can save them to a persistent store of some kind. In the past, this has been a mongoDB, but that can easily change. There are a lot of issues to deal with, but the first pass was just getting the easy things serialized - the data ivars. The more complex problem was the links to other objects. There I took a hint from a lot of O-R mappers : faulting placeholders.

I can serialize out the identifier for the link, and then when everything is deserialized, scan back through the data and "re-link" the links as needed. It's really very simple.

I haven't worked out all the issues and something else has come up, but I'll get back to it.

This afternoon I started work on the serialization scheme for the Instruments in the greek engine I am working on. It's going to be interesting because we have come rather tricky multiple inheritance, so we're going to have to figure out exactly how to handle each object, but I think that's a little more bookkeeping and a little less genius.

The tricky part is the fact that we have references to other Instruments - say the Options on the Stocks, etc. Those are going to be a little tricky for while we can save the securityID of each instrument, we can't be sure everything is going to be serialized out in time for every link to be satisfied. In fact, it's almost a guarantee that we won't. So how to handle these type of links?

I think I have a solution using a subclass 0f Instrument that's simply a placeholder for the instrument - containing the securityID. Then, when I need to get at it, I can easily fault the data back in and carry on. It's not completely done in my head - maybe there's a fault class for every instrument type - maybe there's only one for all. Can't be sure until I get in there and find out what the limitations are and what I really need.

But it's a great little idea. Fun stuff.

This morning I got an email from a co-worker saying that the Java client to The Broker was not getting the date/time values properly in their code. Seems odd, because I had this working for the JSON/websockets interface to The Broker, but OK... I'll go with it.

When I looked at what he wrote, he was dead on - the natural erlang way of specifying a time is to have a tuple of the megaseconds, seconds, and microseconds:

  Time = {M,S,U};

but internally to The Broker we hold it as microseconds since epoch but in order to keep this separate from other integers, we tag it in a tuple:

  Time = {time, USec};

The problem was that I wasn't stripping out the time atom before sending it to the client. Also, for persisting it to mongoDB we need to convert it back into the erlang megaseconds, seconds, and microseconds tuple. This meant that I needed to add a few functions to out time module, but that's not too bad.

Again, it's all about making The Broker more full-featured.

This morning I ran the entire US Option market (all 48 OPRA channels) on one 24-core (hyper-threaded) box. This was a big test for my ticker plants as we have never had it all on one box, and not dropped a thing. Pretty sweet. Sure, it's a decent box, but it's not a ridiculous box - a Dell for under $10k is not crazy, considering you only need one of them.

So good news. I'm glad. Great shot in the arm going into the weekend.