What's it all about, Alfie?

When I got the 128-bit integer done, and the string-based security key put into my ticker plant I was not totally surprised to see that the performance was bad. I mean really bad. It's not something to hack a finger off for, but it was bad. So I knew I had to come up with something to get rid of the need for the 128-bit integers and keep the speed of the processing up.

Along those lines, there were several other things I could do, and still need to do to the ticker plant and how it parses the incoming data from the exchange. I need to handle parsing of the data a lot faster, as it's getting to the point that I've got it right, now I need to get it fast.

Still... the biggie is the map of messages for each instrument. It was the reason for the 128-bit integer, and it needs to be handled. So I'm trying a trie - but only to a point. I looked at the data, and if we encode A-Z, a-z, 0-9, and a little punctuation, we can do it all in 47 values. So if I map the ASCII symbol name with this scheme, I can have 47 bins for the first character, 47 for the second, and 47 for the third - 47³ = 103,823 entries. At a minimum of a pointer, that's still under 1MB for everything in three characters. If I add a map at the 'leaf node' for those symbols in excess of three characters, it'll slow down a bit, but the locks will be almost non-conflicting, and things should be pretty speedy.

The core of the map will be a Family where we'll hold the message for the underlying along with a map of the call options and another for the put options. These maps will be keyed on the expiration date and strike, and will be protected by spinlocks. But since they are only for these maps, the likelihood that we'll encounter a lock contention is very small. It'll still be very fast.

  struct Family {
    Message                  *stock;
    option_map_t             calls;
    boost::detail::spinlock  callsMutex;
    option_map_t             puts;
    boost::detail::spinlock  putsMutex;
  };
  typedef struct Family family_t;

The leaf node of the component will have one of these for those symbols that are exactly three characters long, and a map of these for those that are longer.

One level up, we'll have an array of these leaf nodes - one for each possible encoded third character in the symbol name, and a Family for those symbols that end in two characters.

We keep going like this and at the top level we have an array of these "trees" - one per message type. In this way, we make it almost impossible to have a lock contention as we're forking the data very early, and once past a certain level, you have the pointer for the current level, and that's not going to change on you.

Tomorrow morning I'll finish off the last bits and start the testing. I hope this thing is going to give me the kind of speed I need.

Today, after I built the 128-bit integer, I started a massive refactoring of the ticker plant code to swap out the uint64_t "security ID" from the instrument master database with a std::string that is the name of the instrument. The reason really boils down to dependencies, and while it's not perfect, it's better than the uint64_t.

If we have a database that is the sole dispenser of the security ID, then every time I need to make a new one, I have to go to that database. In theory, this is a good idea, but in practice - for a ticker plant, it's a horrible idea. With thousands of ticks a second, not being able to get one means a several second delay in the processing of that one tick in order to get the security ID.

This isn't rocket science, but it's also unnecessary. Every downstream system can get the security ID from the database - or have it make one for them, if they need it. What the ticker plant provides should be a nice, concise, simple, fast data set that is a source of ticks.

The changes are really quite profound. Now, instead of a nice 64-bit integer to use to uniquely identify the security, I have a string. With this string, and the 128-bit integer class I made, I can still use it in most of the same ways, but the speed is taking a hit for certain. I'm just not sure how much until I get everything in and start testing.

Shouldn't take too much longer... I hope.

In the past, I've built UUID classes in Obj-C, Java, and C++, and in the recent project of my ticker plant, I did it again. It's not hard - you put 16 bytes in a union and allow them to be accessed as bytes, or words, and add a few methods and you're done. But today I realized that those 16 bytes could really be much more if I created a uint128_t. Basically, I created a general 128-bit unsigned integer and then subclassed the UUID off that. It wasn't that hard, but I was trying to encode a generic instrument name, and realized that a uint64_t wasn't going to fit, and it made sense to see if I could make it work with the new 128-bit version.

It didn't take me long, and the code was really pretty primed for it's use. The only thing that remains to be seen is the issue of performance. In general, there's going to be a performance edge to the built-in uint64_t because it's an internal CPU type. But exactly how much performance I give up for that doubling I don't know. We'll have to see as I put it into play.

Today was spent dealing with this external data service that's delivering massive data sets to my ticker plant for it's use. It's basically a bunch of look-up tables - 400,000 rows in all. It's massive, and so there's going to be lots of issues with loading and accessing, but also the time required to get it all in-memory and running.

Details. It's all in the details.

I've got it to the point that it all loads, but there are a few things I'm not really happy about. First and foremost, is the asynchronous loading of the data. I had to do a few tricks to make sure that we didn't immediately reload the data after loading it once. When you place a simple lock on the loader, that's exactly what you can get: a back-up of loads. Not ideal.

Then we're left with the idea that the second call to the loader thinks the load is done because someone else has taken care of it. But it's not really done, it's just being done. So data might not be there. It's a non-trivial problem, and I'm going to have to deal with it sooner or later.

Like I said... it's all in the details.

Today I spent quite a bit of time tweaking the codebase to make sure that there wasn't something I was doing that could be done more efficiently with a little work. I've been working very hard on getting things done in the ticker plant, and it's client, that I haven't had a lot of time for the make it faster part of Make it work then make it faster mantra, so today I wanted to take some time to try and see what I could do to get anything more out of the system.

I also got a few requests from the new users, and I added those changes into the system, and I even actually like one of them. Based on the person that asked for it, I was surprised that I liked anything he had to say, but that's personal. Professionally, he's a dud too, but that's a different story, no?

After I was pretty much done with that, he came back to ask about something else, and I told him the API I had implemented for him - but hadn't finished to this point. It's some of the details that I knew I'd have to come back to, and it just hadn't been the right time. Clearly now, was that right time.

What I found in digging into the source of the data I needed was that there's no way for me to handle it by conventional means. It's data stream is about 55 MB, and that needs to be deserialized into a map of maps representation of about 800,000 elements. Way too big. All I need is the name-value pairs in the data, and I don't need to blow it all into a map-of-maps and then walk the maps to get the pairs. I can do that in a more streaming manner.

So I started to think about that and came up with an absolutely wonderful idea - I'll make an iterator for the data. It'll look and act like a standard STL/boost iterator where you'll initialize it with the data stream, and then 'increment' your way to the end. Each 'step' will have all the data you need for that step, and then the trick will be that the processing of the data will be up to me, the caller.

This will work wonderfully, as I won't have to deserialize all the data at once and I won't even have to spend the time to deserialize all the data at once. I can hold onto the data stream and deserialize what I need. If the user breaks out of the iterating loop I stop decoding. Simple.

I spent the time to get most of the header file done, and tomorrow I'll need to implement it. But I'm very excited about this as it'll fit very nicely into the scheme of things and really help what I'm trying to do.

Cool stuff.

Yesterday I had the real pleasure of working with the lead developer of ZeroMQ on a problem I was having with my TickerPlant. Specifically, I was sending one message and seemingly receiving more than 10. Very odd. When I started off with a note to the mailing list, it then transitioned into a nice chat on IRC, and when he needed to see code, I simply posted the programs I had been using on github as gists. It was a completely amazing experience.

I haven't worked remotely for a long time - about a decade, really. But when I did, the experience was a wonderful one. I got a ton of work done, I didn't feel out of the loop, and everyone was happy. This experience was even better, and it's because of the tools that are now available: github, the Mac and iSight cameras, universal development tools on most platforms - it's really made it possible for someone like me to do my job completely physically removed from the traditional workplace.

Once again, I long for the days of working at home. There's just so much to like about it.

I'm trying to narrow down my problem with the excessive messages I've receiving in my ZeroMQ receiver based on the messages I know I'm transmitting. Basically, the difference is about a factor of ten. It's bad.

So I made a very simple ZeroMQ-only receiver application with everything set up the same to give to the same results and tested it. Yup, same problem. Now, how to share it to the mailing list?

Gist!

I remembered that you can throw up code samples, etc. up there and then share them. So I pulled it up on my work box, sent up the app, and then saved it as a public gist. Bingo! I have a super simple URL to post on the mailing list and they'll be able to see what I'm trying to do.

I sure hope someone has some ideas.

UPDATE: I've been able to see that the transmitter is using just two of the 27 URLs and when I limit the receiver to those two URLs, the numbers match. It's as if the broadcasting is duplicating on all 27 of the channels and the receiver is picking them all up. I've sent something else to the mailing list and we'll have to wait and see.

[10/20] UPDATE: On the mailing list, I got a response from Steve-o that indicates that I really need to be using unique ports as well as unique addresses. The switch will filter on the address, and the OS on the port, but the listening on the port is by the port number, and it appears that the suggested use from Tibco is to have unique ports. This matches with the exchange multicast channel mappings, and I can do that.

When I put that into my receiver and re-ran the tests, the results were rock-solid and accurate. Martin still thinks there's a better way to allow for this in ZeroMQ, and I'm all for that, but until then, I'm very happy with the unique ports as it seems to make the OS/NIC as happy as the switch is with the unique addresses.

Sweet!

Today's problem is a stumper - aren't they all?

I'm sending about 3000 msgs/sec out of a ZeroMQ transmitter using the OpenPGM encapsulated PGM transport, and on the client side I'm receiving about 50,000 msgs/sec! Where are these messages coming from? I know they are duplicates of the messages I'm receiving as I'm conflating them in my receiver's queue, but still... what process is generating these guys?

I look at the stats for the transmitter and receiver, and I don't believe they are miscounting. I don't believe that it's possible for the same code to under count on the transmitter and over count on the receiver. And it's the same code.

The only speculation I'm left with is that it's in the OpenPGM part of ZeroMQ. Maybe this is how they are handling the "reliable multicast", but if that's the case, then why isn't the client filtering out the duplicates?

I'm going to have to figure this out tomorrow. Time to go listen to a talk on RabbitMQ.

Today I was doing more testing, more polishing, on my ticker plant and I realized that there's a bug in my message serialization. It wasn't obvious, as it effected the application the long it ran, it seemed. I was looking into the serialization and deserialization of the variants which forms the backbone of the serialization and deserialization of the ivars in the messages, but that all seemed to be working.

Yet I seemed to be receiving several thousand messages, but only a fraction of them were valid. Something was very wrong.

I started do a lot of logging, because there was no way to easily catch this in a debugger - it got more pronounced as time went along. So I had to put in some logging, then let it run a while, hoping that the logging was going to point out the problem, and cross my fingers.

The one "silver lining" of this is that when these kinds of things happen, I end up going back into the code and placing a lot of DEBUG level logging that clearly illustrates what's moving through the system at that point. I'm not talking about dumping byte streams, but there are times for that in the process, I'm talking about the nice, human-readable logging. Like how many messages are being passed in, the byte counts, and even the break-down of how many messages of each kind.

I end up making these logging helper methods that do a lot of this for me so the code looks clean and it's easy to add this kind of behavior in several places - like the server and the client. It makes life a lot easier. And when I make it look nice, leaving it in at the DEBUG level is really no "cost" to the project, and can help debug things later.

The problem today was that just when I was getting to the point that I was close on this issue, I had to leave for the weekend. I know there's not a lot I can do after-hours for a ticker plant - there's no source of ticks anymore, but I wanted to get this solved.

I'll have to live with disappointment.

On the up-side, I think it's in the Summary messages as those are having the problem right at the end of the day. Maybe it's the packing/unpacking of values not matching. I'll have to verify that on Monday.

Well... I won't lie to you... I'm disappointed in the boost unordered_map. I really am. In the STL std::map an iterator is not invalidated unless you're on the element in the std::map that's being deleted. That's nice. You can get an iterator on a map, and as long as your application isn't deleting elements, there's no way you're going to get into trouble. That's nice for a multi-threaded situation where there's one writer and one reader. The reader gets the iterator and the writer just shoves stuff into the map. No problems.

But that's not how the boost unordered_map works. It's not written up in any of the boost docs, but a careful search of the online message boards yields the fact that no, in fact, the iterator on the unordered_map is not valid across any write whatsoever. This is really bad news, but not totally surprising.

I've been battling a serious problem in my ticker plant, and the core dumps I was getting were in the unordered_map which made no sense to me. But now it makes perfect sense. My reader thread was expecting it's iterator to be valid for the scan, and any write was able to blow it up. It didn't happen all the time, but often enough to make it clear what was happening.

Bummer.

I read that the SGI STL hash_map was supposed to preserve iterators like the std::map, but when I tried it, I ran into the same problems as the boost unordered_map. In the end, I had to settle for simple (but fast) spinlocks on the maps as the majority of the time there will be no contention, and I don't want to slow it down that much.