What's it all about, Alfie?

Well... it's been a while getting here, but I honestly think that this morning was a break-through for my ticker plants. The maximum delay was considerably less than in the past (thanks to finding the silly second consumer thread) and even far better than I remembered from the other group's tests. It's a big day for the good guys, and it's been a long-time in coming.

To be sure, a lot of the problems I came across were because of the changes in the target market. But it took me quite a while to get all the details worked out. But Holy Cow! It's a great feeling.

Well... with the last threading bug figured out for my new UDP transport, I have to wonder if ZeroMQ is really just fine. There are several reasons to use it - OK, one: reliability, but that's a biggie. Still, it might be nice to try it again. The UDP transport, if I can add in the necessary reliability, will be better, for sure, because it's less "black box" and more transparent to all involved. It's certainly already suppressed a few nagging naysayers because they feel that PGM is not reliable at all, and they'd rather have straight UDP. Giving that to them allows them to feel like they have a victory in this "battle", and so adoption will go a little easier.

In short: Politics.

Every time I come to a crossroads like this where a technical decision is given a back-seat to a political decision, I've always felt the wrong decision was made. Always. I'm hoping to give them their UDP transport, and make it technically better than what they have, and so not have to come to this decision.

Alternatively, since they have been so negative in this adoption, it would be nice to simply do all the work myself and then point out to management how we can "re-task" these "resources" to "different efforts". Problem with not being part of the solution is that after the solution is done, you've proven yourself to be unnecessary.

Right out of a job.

Anyway... for now, I have one good transport, and I think I have two. But I'll have to run with these changes for a bit to convince people it's as good as what they have. Gotta prove they aren't needed any more. Sad but true.

Today I've been fighting a lot of issues in my new UDP-based transport for my ticker plants, and I've tried changing queues - multiple times, more logging, structural changes - all kinds of things that seemed to be the issue, but the real issue seems to be that I was not being careful enough starting my threads. Yup... threading is hard, and it's easy to make mistakes.

The set up is that I have a lot of "single-consumer" queues in my system. If you try to hit them with multiple consuming threads, you're going to be very sorry. Couple that with the fact that I was trying to be clever in my thread initialization and processing code and "covering all my bases" by putting redundant code in the initialization and processing blocks. The problem is, if you get the initialization complete, the next thing is the first trip through the processing code, and if you duplicate the code there, but have an off-cache flag set, it might appear as though you haven't properly initialized the threads.

So you start more.

Ouch.

The obvious answer is don't try to be cute. Have code in one place and one place only and then make sure it's successful. When I did that, all the other problems "magically" disappeared. Amazing.

Not really. It's obvious if you have multiple consumers on a single-consumer queue you're going to get into trouble. Also, it'll appear as though you are loosing data, when you really aren't. It's a mess.

Well... it seems to have cleared up and I'm glad. It's been a major pain.

Today I've been at it all day - writing more code for my new UDP-based transport. There's a lot that I need to get going, and it's not at all trivial to get these guys working properly when you aren't locking anything. It's been a nasty day of a lot of tests and tweaks, but in the end, I have something that delivers bursts of 320,000 msgs and it does it all pretty efficiently.

Tomorrow, I'll start the live testing.

This morning I noticed that a significant new upgrade had been made to WordPress to bring it up to version 3.1 - so I had to go to HostMonster and upgrade my installs. Since my last upgrade, SimpleScripts has adopted the "update all at once" idea for multiple installs of the same piece of software. I haven't had any problems with SimpleScripts to date, but I didn't want to start this morning. Still... I knew they made backups, so I went ahead and upgraded them all at once.

Worked out just fine.

The new WordPress has a menu bar at the top of the page when you're logged into the site. It's not bad, but it's not the same style as the blog, and I'm sure it's only a matter of time before the artists get things going and make that, too, in-sync with the theme of the site.

I spent a few minutes looking over the new features - some are nice, others have a few bugs in them, but I'm sure will get worked out with the release. In all, it's a nice improvement, and while I really just use this for my journal, it's nice to know it's got the room and capabilities to be much more.

This afternoon I started work on a new UDP-based delivery system to replace ZeroMQ in my ticker plants. It's going to be as simple as I can make it while still giving me the level of service that I need for my users. Basically, I'm sitting in nice data centers, and the switches are nice, so I shouldn't have a lot of drop. But there will possibly be some, so I need to be able to plan for that. I'm not exactly sure how to implement the reliability in this system, I've got a few ideas that all could work, but I'll for sure need something.

What I want to start off with is a simple UDP broadcaster and antenna. These will take the messages, serialize them, put them in the UPD datagrams, and send them out on the different multicast channels I'd been using for ZeroMQ. I'll use boost ASIO for all this, so it shouldn't have too much overhead - it works for the incoming data from the exchanges.

On the antenna, I'll have one boost udp socket for each multicast channel, and then have a single io_service thread reading them off the socket and into "byte buffers" - one per socket. Then we'll have a thread pulling the datagrams off the queues and deserializing them to place them into the conflation queue. Pretty simple model.

Lots to write. Better get at it.

It really breaks my heart to see problems with ZeroMQ that aren't being addressed. For instance, I've been trying to work with the guys on IRC about the fact that the GitHub master doesn't receive messages when using OpenPGM. I've talked to the guys on IRC about this, and it seems that some of the more recent changes really messed this up, and there were no checks on the code to make sure it was still working. Sad, but true.

I've tried several times to jump-start this several times, but I didn't get very far on any occasion. This makes it hard to use, as I can't use it without OpenPGM, but I'd like to get some of the newer features they are talking about.

But that's not the worst.

I think ZeroMQ is either delaying, re-ordering, or retrying with excessive delay, some of my messages - and it's only when I'm really hammering the data rate. For example, I know that the ZeroMQ send() method is asynchronous. So it buffers up the data and then sends it. But what if it gets messed up?

The delivery seems to get worse and worse as the day goes on, and it seems to be based on a zmq::socket_r getting into a "bad state" and never getting itself out. I believe that it's in the receiver, because two apps on the same box have different reception profiles after a time.

In any case, I can't trust it and I can't build the latest code. It's just no longer a really workable solution. I have to find some kind of solution that doesn't include ZeroMQ.

This morning I noticed that when I had subscribed to a large portion of the option market data feed, I got into a position where the conflation queue could exceed 131,072 (128k) messages. It's not impossible for a slow client to have this problem, and the safest thing to do is to increase the size of the FIFO queue in the conflation queue to cover all possible cases. So I bumped it up by a factor of 4 to 512k (2¹⁹), and then all of a sudden I started getting seg faults. What?!

I started digging into this and was just plain stunned to see that it really didn't matter what the use case was for the conflation queue - if I had a conflation queue as a stack variable, I was going to seg fault. Right at the beginning of the app.

When I changed it to allocate one on the heap, everything was fine. OK... looks like a limitation or limit on the shell - but nothing I could find pointed out what to change. Also, this was something that everyone would have to know, and that's not a very user-friendly experience.

So I decided to make all the uses of the conflation queue in my code to be heap-based. It wasn't too hard - change the variable to a pointer, all the 'dotted' references to 'arrows' and then you're almost there. The last big thing was to make sure that they were initialized an cleaned up properly.

Thankfully, I didn't have all that many places that I had used the conflation queue - three classes in total. It was pretty easy to look at each and come up with a plan for creating these guys and cleaning them up properly. The whole thing probably took me 45 mins - and of that, more than half was spent trying to come up with a way around the seeming limitation.

Important thing to note when creating large items - you make have to go to the heap just because.

Today I had an amazing debugging session with some code I'd written and thought was in pretty good shape. It took most of the day, but there were really several things I needed to figure out in the test case I was given. Basically, it was an extreme edge case, and yet, it was possible, so it needed to be handled properly.

But the final bug was a doozie. The original code looked like this:

  const uint128_t  & ckey = aMessage->getConflationKey();

and in that simple line there was the final bug.

If we had multiple threads hitting the same code, it was possible to have these messages cleaned out. But what happens when you delete the aMessage and you have a reference to a value in it? Well... it goes to poo, that's what. The fix was simple:

  uint128_t  ckey = aMessage->getConflationKey();

One character - the ampersand. That's it. Make a copy as opposed to holding onto a reference. One character. Amazing.

Some days I love this job.

This morning I noticed that Google Chrome dev 11.0.672.2 was out and with it quite a few changes. While the major version number doesn't mean a great deal, in my experience, the additions of the new V8 engine as well as a few nice Mac specific features makes this nice to have. I'm sure there's more to it than just this, but this is enough.