What's it all about, Alfie?

Late today I had to start the process of re-writing two very important parts of the greek engine that I had given to another guy on the team. I don't like doing this for a lot of reasons - not the least of which is the fact that we lost a ton of time. I blame myself far more than I blame him - he was doing his best, and trying to get it all done to impress me. I was hoping that his little "updates" were correct, and I failed us both. Badly.

So I had to take it back, and I had to start looking at what was and quickly think about what I wanted it to be. The job I set before myself this evening was to get the "map" of Instruments in order. This is basically a trie with a thread-safe list that are going to hold the families of instruments - the stock and all it's futures, their options, and the options on the stock. It's a bit of a mess, but that's the nature of the beast. The idea was that this structure was going to make it easier to deal with these families.

There were a lot of things that were very broken. It took a couple of hours, but I was able to clean this up and cut out a lot of mess. Getting it to a point that it can do the job for me starting tomorrow when I have to handle the really nasty problem - the InstrumentFactory.

That's going to be nasty.

I spent the first part of today working on getting the serialization scheme of the Instruments going so that I can save them to a persistent store of some kind. In the past, this has been a mongoDB, but that can easily change. There are a lot of issues to deal with, but the first pass was just getting the easy things serialized - the data ivars. The more complex problem was the links to other objects. There I took a hint from a lot of O-R mappers : faulting placeholders.

I can serialize out the identifier for the link, and then when everything is deserialized, scan back through the data and "re-link" the links as needed. It's really very simple.

I haven't worked out all the issues and something else has come up, but I'll get back to it.

This afternoon I started work on the serialization scheme for the Instruments in the greek engine I am working on. It's going to be interesting because we have come rather tricky multiple inheritance, so we're going to have to figure out exactly how to handle each object, but I think that's a little more bookkeeping and a little less genius.

The tricky part is the fact that we have references to other Instruments - say the Options on the Stocks, etc. Those are going to be a little tricky for while we can save the securityID of each instrument, we can't be sure everything is going to be serialized out in time for every link to be satisfied. In fact, it's almost a guarantee that we won't. So how to handle these type of links?

I think I have a solution using a subclass 0f Instrument that's simply a placeholder for the instrument - containing the securityID. Then, when I need to get at it, I can easily fault the data back in and carry on. It's not completely done in my head - maybe there's a fault class for every instrument type - maybe there's only one for all. Can't be sure until I get in there and find out what the limitations are and what I really need.

But it's a great little idea. Fun stuff.

This morning I got an email from a co-worker saying that the Java client to The Broker was not getting the date/time values properly in their code. Seems odd, because I had this working for the JSON/websockets interface to The Broker, but OK... I'll go with it.

When I looked at what he wrote, he was dead on - the natural erlang way of specifying a time is to have a tuple of the megaseconds, seconds, and microseconds:

  Time = {M,S,U};

but internally to The Broker we hold it as microseconds since epoch but in order to keep this separate from other integers, we tag it in a tuple:

  Time = {time, USec};

The problem was that I wasn't stripping out the time atom before sending it to the client. Also, for persisting it to mongoDB we need to convert it back into the erlang megaseconds, seconds, and microseconds tuple. This meant that I needed to add a few functions to out time module, but that's not too bad.

Again, it's all about making The Broker more full-featured.

This morning I ran the entire US Option market (all 48 OPRA channels) on one 24-core (hyper-threaded) box. This was a big test for my ticker plants as we have never had it all on one box, and not dropped a thing. Pretty sweet. Sure, it's a decent box, but it's not a ridiculous box - a Dell for under $10k is not crazy, considering you only need one of them.

So good news. I'm glad. Great shot in the arm going into the weekend.

Since it's post-Google I/O days for Chrome, the boys at Google Chrome have released an update to the dev channel - 13.0.761.0, and this morning I picked it up. It seems to handle remembering page passwords better than the previous cut - maybe that's a fix, or just a different fork of the code. There's no release notes up (yet), but I'm guessing this is just the next in the series. Good enough for me.

We have made some amazing progress in the past few days with the greek engine we're building. For the first time in this project, I feel like it's really coming together. So much so that I was willing to do a little extra for a group that's trying to use our erlang-based Broker to replace the previous version written in Java. The problem revolve around the fact that they needed to be able to look up the user's authentication data based on the 128-bit auth token that the login service generated. Kind of like a "reverse-login".

The problem was that they were going directly to the mongoDB database where this data was stored, and when they tried to look things up by the 128-bit token, it was byte-ordered differently than expected. This was a problem, but not a horrible one. Still... if I had the time, and it seemed I did, it sure would be a nice thing to write them a service that did this for them so they didn't have to hit the database directly, and didn't have to worry about the byte ordering.

So I did.

Interestingly, it was only about 10 lines of erlang to add to the login service. Where the problems arose was in the encoding/decoding of the data to make it "fit" in with the rest of the serialized data. Times, it seems, weren't really being handled well, and that's understandable, so I had to fix that up before I was getting useful values out.

Then I had to mess with the offset. Originally, we used milliseconds since epoch, as it's pretty universal. But when the high-frequency guys wanted to get in on the act, they wanted microseconds since epoch as their stuff happened a little faster than that. So I had to make sure we had the offset right. Once fixed, we were ready to go.

Almost.

Seems the decoded data on the Java side is not giving us the right date. It's a date, but it's not the right date. I've handed this off to a guy in the group, as I'm convinced that we're sending the correct data - it's just not being delt with my the java client. We'll see what he comes back with.

But it was nice to do a little extra, and help out the adoption of the Broker. It's pretty nice stuff.

I thought I had this particular problem with The Broker solved - saving and loading lists to and from mongoDB using the emongo client - but I was wrong. Turns out, I can do it with lists of strings, but not with lists of integers. Something to do with the pattern matching in the erlang code.

What I did was to put in the more overt conversions and then both types of saves and loads worked. I'm just hoping that as my test cases get more complete, it really starts to equate to fewer bugs... but I have my doubts. This is just so frustrating. But it has to be done.

I finally have a successful test of the system - including the JSON interface over websockets. It's a long time in coming, but it's here. I can hardly believe it. There are still a lot of things to work out, but we have something that the GUI guys can play with, and that's very important to move forward. I can push the guys to get the rest of the pieces done - or take them back and finish them myself.

Either way... I feel I can rest for 15 min and catch my breath. It's certainly put a smile on my face!

Today has been a real kick in the pants, and not necessarily in a good way. I was getting ever closer to have something to test with the GUI team, and it seems that at every turn I was finding things that didn't quite fit. I know some of this is expected, but it's really a little more than I expected. A few of the components were really just "done enough" for the testing at hand. There's a world of difference between "good enough for limited testing" and "ready for applications", and I'm paying for it today.

Still, the end is in sight, I just have to slug through a ton of erlang code and JSON conversions. It's all there, I can see it, but getting the bloody matches to work is brutal.

Another long day... Man, I'm tired.