What's it all about, Alfie?

There are a lot of things to like about being a lead developer - you get to have a significant hand in the design of the work you're doing. You get to pick what you want to do, and farm out the rest to others. But there's a dark, ugly side as well - people. Yes, those devices that can't be debugged no matter how long you use gdb - people. And I'm reminded of it today in a very unpleasant way.

The lead on a project is expected to help the junior guys along. Help them learn the craft as it were. They will pick up habits, the goal is that they pick up the good habits of the lead, and then they are able to be self-sufficient in the workplace. Kind of like kids. Being in the middle of that cycle at home gives me keen insight into what's happening to me here at work.

I've got a clingy 5-year old. At home we called it "the five-foot rule". The kids would never be more than five linear feet from us no matter what we were doing, where we were doing it, no matter how big the house was. They were right there. Always. In their early years we were everything - playmate, audience, judge, everything. The same is true at work, it seems.

He's telling me about the status of his work. He's telling me he'll be done soon. When he's done. That he needs to be busy. It's like a clingy 5-year old. I can appreciate that he's trying very hard to be productive and contribute, but he also doesn't realize - like a 5-year old doesn't realize, that there are completely different expectations for him than for me. He's supposed to learn, listen, and if he's really doing his job, be as light a load on me as possible.

But that's not how it's working out.

I'm going to have to slug through this for a while longer. I'm not sure how much longer, but a while at least. If he doesn't get better, then he's the learning disabled kid, and will continue to be a burden on me as long as he's here. That's not going to fly. But assuming he's going to get better, I just have to give him time. It's tough some times.

I've been talking about The Broker for a while, and it's been through at least five iterations/revisions/re-writes that I can remember, but we're in our sixth now. What I've seen, as I hit it the hardest of almost anyone, is that the java version built on Netty is just too complex. The design is reasonable, but even that's complex because of the direct dial interface we put on The Broker to remove the massive load some clients can generate. It just wasn't working. So we tried a new approach.

I mentioned distributed erlang, and asked why we didn't build the original erlang Broker using that? It seemed to handle all the registration/location service issues. It also handled the problem of a centralized service in general. When one client makes a request to one service, it's only those two nodes that will see the load - not anyone else. It's a really nice idea - extremely simple, and therefore, probably the best.

So my boss started on this next version. He and I worked on it today and we're down to just a few things that need to be finished: the service registry and the service load (for load balancing). I'll do those tomorrow and we'll be feature complete.

The beauty of this design is that we can add as many native services to the Broker as erlang modules in the distribution package. Each one is very simple, and is loaded on demand from the elang runtime. This places us in the wonderful position of being able to add in the login service, the configuration service, and any other service that we want in the native erlang. Because this new design places a Broker node on each box running a service, this means that for the most used services, like login and configuration, we won't even go off-box to get the response.

This is going to have a huge impact to the reliability of the Broker. It's going to be able to be stopped/started on each node, as needed, it'll connect into the grid and share it's state, and will be able to handle all the routine services on it's own. This is a major win for me. I think it's going to make it great.

Well... the first week of the four week "delivery cycle" of the new Greek Engine has come and gone, and I have to say, if we get done with it in the next three weeks, I'll be surprised. Maybe not off by much, but there's so much to do, and with two new guys on the team, one very junior, it's a lot of work on my part just to keep them for imploding.

On the other hand, I have to say that things are coming together. They are taking shape in the code, and things are getting written - just not fast enough, and the hardware was always going to be an issue. There's just not enough time to get all the hardware purchased, installed, built and running. It'll be close, if they hurry, but I've already found out that I'm going to be delayed at least a week on some hardware coming from another data center. Doesn't sound like much, but that's 25% slippage.

But progress has been made. We have better integration between the two projects - my ticker plant and the greek engine, and we at least know the sections of code we need to build in order to make this thing work. That's a big help. It's just a matter of time.

Today starts the push for the next four weeks to get out a new greek engine at The Shop. I'm not at all sure it's even possible, but I'll give it everything I've got for the four weeks to see if it can be done. There are a lot of things that have to happen in the right order, and pretty quickly. I've been told they'll all get done, but I have my doubts. It's just a lot of things that have to happen just so.

This morning already I'm spending too much time getting the next release of ZeroMQ checked out. It should be a pretty simple thing to check out, but it's taking far longer than the previous versions. Still, I have to get this done, and then I have to get SQLAPI++ working for the FreeTDS drivers we have on the boxes. All this in order to load up the data from the MS SQL Server databases that holds it all.

Yup, it's going to be tough to get it all done, but I'm willing to give it a try.

Today has been a mixed bag... I did get a little development done on the ticker plants - just a few little things to polish up some things, but I also had hour-long design meetings (not bad), and then multi-hour-long debugging meetings where there was no hope of actually finding a problem because the developers had so little experience developing on linux that there was no hope of a successful test, and they really just needed to understand the proper way to edit/compile/test code.

The design meeting was really kind of interesting. We have The Broker, and it's been an erlang process, and a Java process, but it's always been centralized. I mentioned today that I wondered why it wasn't using Distributed Erlang, and let the entire brokerage system sit on all the machines and handle everything in a more distributed manner. Let erlang handle the registration, and the message passing. We can use J interface (Java library) and el (C library) to make our server applications appear as distributed erlang nodes. This makes it much easier to do all the things the Broker does.

Sure, the clients still can use sockets to connect to the Broker - or should I say A Broker, where we can have one running on each server in the server room. The client (defined as processes that aren't nodes in the distributed erlang system) can then use the socket interface to connect to a Broker and work as it always has, but the wrinkle is that it's only really brokering his traffic. This gets rid of a lot of the problems we had faced in the past - and went to great lengths to try and solve.

So much gets easier if the main components of the system are all distributed erlang nodes. Very nice solution to the problem.

The debugging session was only slightly productive, and ultimately disappointing. The code was compiled on CentOS5 and run on Ubuntu 10.04.1 - I'm not at all surprised that things broke. Far too different a version of libc, gcc - everything, really. We couldn't even get the code to build on Ubuntu. They need to step back, build the code on the box they are going to run it on, and then move forward.

It's slow going sometimes.

I was having a bit of a problem with The Broker today, it seemed. It appeared that when I saved my state to the Broker's configuration service, I was hanging, and the monitoring thread that fired off this save was hung. I got the guys to restart the Broker and things seemed OK, but I decided to take advantage of one of the really neat things of boost threads, and fire off the call in a separate thread so if it stalls, the monitoring thread doesn't.

The old code looks like this:

  if (secs - mLastMessageSaved >= 300) {
    saveMessagesToConfigSvc();
    mLastMessageSaved = secs;
  }

becomes:

  if (secs - mLastMessageSaved >= 300) {
    using namespace boost;
    thread   go = thread(&TickerPlant::saveMessagesToConfigSvc(), this);
    mLastMessageSaved = secs;
  }

and now the call to saveMessagesToConfigSvc() is now called by the separate thread, and as soon as the method returns, the thread is killed and cleaned up. Exceedingly sweet!

OK... this is what boost threads are all about, but in comparison to Java threads, or something that takes a little more scaffolding, this is elegant to the extreme. Just add a few constructs to the line and it's done. You can't get much simpler than that. Very nice.

The past couple of days have been about speeding up the processing of the exchange data through my feed system. Because the code for the decoding of the messages is fixed, specifically for the big OPRA feeds (using OPRA's FAST decoder), most of this is accomplished by re-organizing the data flow and data structures. One of the things I had done a while back was to have the two channels of an exchange feed put their datagrams in separate queues, and then have one thread empty both so as to remove the need for locking on the sequence number arbitration code.

The problem was that this required twice the time to process the data through the decoder because both A and B sides went through the same thread. This can get to be really quite nasty. For example, on an OPRA feed, it takes about 40 usec to decode an average datagram, and at 50,000 datagrams/sec we're looking at 2 sec to process one side, but this design would have to do double that work. Nasty. Lots of buffering.

The solution is to have one thread per datagram stream. That immediately cuts the processing time in half. The problem is that we then need to lock for the sequence number arbitration. Nasty. Then I had a flash - merge the data!

First, tag one of the channels as primary, and have it control the arbitration. Every other channel decodes it's datagrams but then instead of trying to have the thread send it out, have that thread put the decoded messages into a queue that the primary will process as soon as it's done with it's datagram. The arbitration is very fast because it's as simple as checking the sequence number and a few flags. It's the decoding that takes the time. With one of the FIFO queues, we can have multiple non-primary channels, and have the primary take the results off and send them out.

Even more importantly, the primary can be the primary feed line of the exchange, and that makes things even better as the secondary feed is really only needed when there's a failure of the primary. What we've done then, is to make it more like the "normal" feed with a "backup" just in case.

Very neat.

Today was a day where I spent about half my time tweaking for speed, and the other half recovering from the problems of making it too fast. I'm in the middle of a push to make the core of my ticker plants as fast as possible for a group that's going to start testing it next week. There have been a lot of little tweaks - each getting just a touch faster, but together, it's a non-trivial improvement. Unfortunately, these little changes pointed out threading issues as well. Those then needed to be fixed.

It's a game I've played for a while now, so today isn't unlike a lot of other days lately, but it's just "more of the same". It's all part of the improvement process.

This afternoon I found my speed problem, and it's not an easy solution. It should be, but it's not, and in retrospect, I really wish OPRA made a better FAST codec. But I'm getting ahead of myself.

I started adding a lot of timing snapshots to the code. It's annoying if there's a lot of it, but in the short-run, it's really the only way to see how long this section of code takes. Sure, there are profilers, but they can only look at modules or functions - not sections of code, as I needed. So I put these in and started the runs. Much to my surprise, the FAST codec is not too bad, averaging about 20 to 50 usec per message. Sure, it depends on the size of the message from OPRA, but it's not bad. This also includes the sending of the messages out to my test harness and calculating the minimum, maximum, and average latency for the messages it receives.

So it's not in the FAST codec. Where, then, is it?

Certainly, one of the nastier delays is the queue restart. By boost ASIO thread reads off the UDP datagram, and then tags it with a timestamp, and places it on a FIFO queue. The queue is lockless, and so there's no conditional to wait() on, and that's a good thing, as the delay time would be horrible if it were (already been there, tried that). So when the queue is empty, my servicing thread can choose one of two options: sleep and let it queue up, or spin and burn up the CPU. I've tried both, and I can get the time to a reasonable level with the spinning, but then the CPU usage is eating a complete core - not ideal.

Sleeping isn't any better because it allows the minimum "maximum" latency to be the sleep interval and that is currently set at 1 ms. However, it's much more considerate on the CPU usage - averaging only about 12% of one core.

But still, the delays remained. Then it hit me. I was looking at the problem all the while and just not "seeing" it. If I have one thread decoding these datagrams from both A and B channels of the exchange feed, then I'm doing twice the work I need. This isn't bad, but it's taking twice the time. I needed to get two decoding threads active again. I say "again" because I had this in the past, and removed it in favor of a lockless arbitration on what messages to send. At this point, I need the speed and am willing to have the lock (for now), to get the processing time in half.

But that's not all. If I think about the load from a single OPRA channel I'm looking at upwards of 100,000 msgs/sec. If I take, on average, 40 us per message to decode it, I can get only about 25,000 msgs/sec - not even close to the speed they are arriving. So this is my fundamental problem: the OPRA FAST codec is just not as quick at decoding the messages as OPRA is at sending them. Sure... with faster CPUs I might be able to get it a little faster, but in the end, it's not going to be more than maybe 100% faster. That still leaves me with a factor of 2 to make up.

So we're going to queue things up. No way around it. Given that we're queueing, the delays I've been seeing are completely explainable. Let's assume we get a burst of 10,000 messages - the last message in the burst is going to get decoded some 400 ms after it arrives and is timestamped. No two ways about it.

Some of my competitors feeds say they have much smaller latencies - and the way they measure it, they do - because they allow the OS-level socket buffers to hold the 10,000 messages as they process one completely before pulling the next off the socket. If they timestamp it, decode it and send it, their latencies should be no more than about 50-100 usec. That's what mine would be. But you're not measuring the real latency of the server. For that, you have to keep the OS-level socket buffer empty and timestamp them within the process space and then measure the overall latency.

There's no free lunch. You can buffer the datagrams in your process or you can have the OS buffer them for you. The only difference is with the former, you'll actually know when the datagram arrived at your NIC, and with the latter you won't.

So... what's the solution? Well... we can throw more cores at it. If we somehow parallel processed the datagrams into messages, then the effective decoding time could be much less. But then we'd have to worry about getting the messages out of order.

We could re-write the FAST codec - and this I've thought about, but the code isn't that bad, and OPRA is going to 48 channels soon, and that should cut the rate to a max of 50,000 msgs/sec, and give the decoder a little breathing room.

I'll work on it, but I'm confident now that there's no horrible problem in my code. And no one is doing it really any better than I am. I just need to see if I can come up with some clever way to get around these limitations.

This morning I went through a section of the code I'll be focusing on today and realized that when I originally wrote it I had the idea that every UDP datagram from the exchange should generate a message to send. If it didn't then it was an error. As a point of fact, there are a lot of datagrams that don't generate messages - Admin messages, Control messages, all kinds of messages.

What I had been doing was to create a simple base message and place it into the list so that an empty list signaled an error and I then filtered out the simple messages from being sent. Now I don't think I was spending a lot of time checking the message type, but the new and delete of the message could not have helped. So I decided to clean that up and get rid of the error checks on the empty generated list. This shouldn't be a huge difference, but every little bit helps when you're looking for fractions of a millisecond.

The next point I found was the skipping of sequence numbers by the OPRA FAST codec. In general, OPRA sends datagrams encoded in this pseudo-FAST protocol where the first message is specified, but then only "deltas" are enclosed in the packet. This makes it very fast to encode multiple messages - which is, of course, why they used it. The upshot of that is the starting sequence number is provided for the datagram, and I was using that to advance the "last sequence number" field in the decoder. The problem is that if there are more than one message in a datagram, this is going to make it appear that we're skipping (and therefore missing) messages from the exchange.

So I went in there and have a few interesting ideas about how we might be able to make that work. Of course, the simplest scheme is to check all the messages and take the largest number as the new "last sequence number" and use that. But I might also get lucky if OPRA only sends contiguous messages. If that's the case, I might be able to take the starting sequence number, add the count of messages, and then run with it.

I also might be able to just look at the last message in the list and use that. It's really full of a lot of possibilities to clean this up.

The next thing was to get an idea of the speed of the datagram processing. To that end, I skipped all the message processing and put in checks for the timeliness of the datagrams. I'll need to wait for the open to test this, but then I need to get busy really hammering on this code. I want to make some real progress on this today.