Archive for the ‘Cube Life’ Category

Tricking a C++ Union into Having Methods

Thursday, September 2nd, 2010

cplusplus.jpg

I've got an interesting problem that I sure wish I'd checking into the linux C libraries a lot more before getting too far into this, but so it goes... It's the old UUID problem. It's a 16-byte, unsigned number, but it's far too big to hold in one atomic data element, so it's often a union:

  union UUID {
    uint8_t   bytes[16];
    uint16_t  words[8];
    uint64_t  blocks[2];
  };

but the problem with this union is that you can't do something like this:

  UUID   a;
  UUID   b;
  a = b;

or:

  UUID   a;
  UUID   b;
  // ...
  if (a = b) {
    // do something...
  }

C++ just doesn't allow for the default operator=() or operator==() overloading like it does the default constructor, the default (shallow) copy constructor, etc. You have to build these things into a class.

However, there is one nice thing that these unions allow: it's the un-biased use of the union as the start of it's member data. For example, if we have:

  UUID   a;
 
  // ...
  memcpy(&a, src, 16);

we can write the 16-bytes of data into the union's elements by simply referencing the UUID itself. This is not the case with a class. There's the vtable, and that ends up throwing off all the offsets for the ivar data.

But what can a guy do to get these methods on a union?

The Answer is really the anonymous union:

  struct UUID {
    union {
      uint8_t   bytes[16];
      uint16_t  words[8];
      uint64_t  blocks[2];
    };
 
    // the constructor/destructor set...
    UUID();
    UUID( const UUID & anOther );
    UUID & operator=()( UUID & anOther );
    UUID & operator=()( const UUID & anOther );
    // the equality/inequality operators...
    bool operator==( const UUID & anOther ) const;
    bool operator!=( const UUID & anOther ) const;
    bool operator<( const UUID & anOther ) const;
    bool operator>( const UUID & anOther ) const;
    bool operator<=( const UUID & anOther ) const;
    bool operator>=( const UUID & anOther ) const;

where I added in the inequality operators so the UUID could be used in a std::map as a key. The problem now is that I'd really love to have the following still work:

  UUID   a;
 
  // ...
  memcpy(&a, src, 16);

but it won't. I have to reference the first part of the data to get past the vtable, etc.

  UUID   a;
 
  // ...
  memcpy(&a.blocks[0], src, 16);

It's not horrible, but it's no better than really building a class. I got the syntax I wanted, but the consequences weren't without their costs. Kind of bumming to get it all going and then remember the vtable, but that's life.

Banging Away at a New Client Library

Wednesday, September 1st, 2010

GeneralDev.jpg

Today I spent the entire day banging away hard at a new client library to a data system that another group in The Shop has created. They built a Java client, but I needed to have something in C++, and have it interface nicely to the variant data type that I've already created. Someone working with me started the work on this client, but then was called away on another project because it was having lots of problems in production. So it goes.

So I took the start he had on the client library and took a good, long look at the code to see what needed to be done, and what was really looking really pretty good. As is so often the case, the start he'd gotten was OK, but it wasn't really good enough for the kinds of conditions that I'm used to seeing, and maybe that explains the problems in production a bit as well. So it was a total do-over.

The key points of the service, insofar as the design is impacted, is that there are socket connections, then on a single socket connection, there can be multiple simultaneous calls and subscriptions. The call is a one-time subscription, which returns a single result, whereas the subscription will send updates to the client with every change to the source data that generated the result.

It's a nice model, and the only problem I see with it is that boost asio is set up to have a single buffer of the incoming data, and so it's possible to really have that one buffer, or socket, be a bottleneck. Additionally, if there's a problem in my request, it's possible that the service will not know how to respond, and it's only solution is to kill the socket. That's a bit drastic, but I can understand why - the service doesn't know how to respond to me, so it's only solution is to tell me "that's VERY bad" by killing the socket.

With that, all my pending calls and subscriptions are dead, and so it's possible for one bad apple to spoil the whole bunch. Not my idea of a great way to go.

So what I ended up with as a first cut on this is to have an "updater" object that has a socket and all the supporting data with it. Then, a single call is a single socket, a single buffer, and a target variant. This means that as data comes in, boost's asio can handle as many of these as necessary. Much cleaner, but there's a cost - multiple open sockets, and starting one up isn't fast.

Still... it is a good start, and I spent today building that. At the end of the day, I had something that was working, but needed to be cleaned up a little bit. Still... stayed late to get that going, and was very happy to see all the bytes line up and get back solid responses.

Working on Magic Map Implementation in C++

Tuesday, August 31st, 2010

Today I spent a good bit of time working on extending the variant class I have in my ticker plant to handle byte arrays and then adding in the map key space encoding and decoding. The idea is that map keys really don't need to be just any string - they can be a limited subset of the ASCII space, and in making that a limited set, we can pack more characters into fewer bytes, and then unpack them once they are on the receiver's machine.

Say we limit the ASCII space to 64 characters - any 64, really - just so long as there's a mapping from the 255 ASCII values to the 64 acceptable values, and back again. At that point, we know that we'll be able to store the mapped key space in 6 bits - 26 is 64. So if we look at a series of three bytes we can pack four of these characters into that 24-bit space:

Byte 1 Byte 2 Byte 3
1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
Char 1 Char 2 Char 3 Char 4

The code for the conversion is pretty simple - look for now many even 'blocks' of four characters to encode, map them into the limited key space, mash their bits and pack them into three bytes. The remainder is a simple process of doing part of that.

The decode is simple as well - we just need to have some terminal condition - and that can be a simple 0x3f (all 1s) in the last (terminal) 'character'. So we look at the length of the byte stream, see how many "full sets" of three bytes there are - convert each into four characters, and then based on what's left, we have only a few options. It's pretty simple to decode, and you have the original string back.

I needed to get this all in the variant class as the next thing I needed was to add in the update stream that will be sent by the data service in response to updates at the source. This is similar to the kind of updating I did in Java back in InfoShop: there was a HashMap that understood a 'path' concept, and transactions. You could put a map into a transaction, do things to it, and then commit the transaction and get a transaction log that could be sent to remote copies of the same map, and be applied to bring it up to date.

It worked very well for me at the time, and it's something very similar this time. There's no transaction, and the updates are all streams with an action, a path, and an optional value. It's simpler, but for the data sets we're dealing with, it's sufficient.

I got a lot of the decoding of the update stream done, but I think it's time to put together a client and see how it functions to see what I'm really getting from these servers. There's nothing like the socket-level byte stream to remove all the questions and ambiguity in the specification. So that's going to have to wait for tomorrow.

Case Sensitivity and New Developers

Tuesday, August 31st, 2010

Crazy Lemon the Coder

I believe, though have no factual data to support this belief, that most developers these days are learning to code on Windows with some IDE - like Eclipse, NetBeans, etc. It's the easiest way to learn to code, I'll grant you, and it's the cheapest platform to use. Heck, you can get a decent Windows laptop at BestBuy for under $800. Peanuts.

But there's a good contingent that seems to have fallen in love with the Free Software movement, and picked up linux to put on their $800 laptop, and are using Eclipse on linux. This is not bad. But it's the clash of these two worlds that often leads to problems.

Then again... it could just be that the developers aren't all that good, and need to be whipped into shape before they're set loose on the codebase.

So here's what happened today: I'm trying to clone this git repository onto a windows box because the way I have the monitors hooked up, it's easiest to see the code from the Windows screen. My first problem was with the git that ships with Cygwin. Turns out, if there's a problem with the repo (as there was - sort-of), the Cygwin git gets all messed up. Maybe in a subsequent release they'll get it fixed, but the consensus at The Shop is that the better solution is to get msysgit from Google Code.

(As an aside, I've also decided to upgrade my Cygwin to 1.7.7 as it seems I was pretty out of date, and that could possibly have contributed to the problems.)

So I clone this repository and I find that I've already got a changed file. Hmmm... that's odd... so I check that file out again... still changed. Very odd.

Turns out, the developer had created two files - one a Java source file, the other a shell script - one named MyTest.java, the other myTest.java. OK... that's got all kinds of wrong written all over it. Case is not the way to distinguish files - not in the multi-platform world. And who makes a shell script have an extension of .java?

When I pointed this out to him, he cleaned it up and there wasn't a problem any longer. But it reminded me that what I take for granted is not exactly universally understood.

Tough Day Full of Avoidable Problems

Monday, August 30th, 2010

Today was a day I'll be glad is over... there were just so many avoidable problems and delays that it makes for a day that really is best forgotten as soon as possible. It started out reasonably well - I was finishing up the work I'd start on Friday. I had really wanted to get it done on Friday, but it took another three hours (roughly), so it would have been silly to really stay and see it through. Also, there was no way I could have tested it Friday evening - and this morning I was able to use the live data to verify that things were working as planned.

But at this point, things were still going pretty well. The code was done, everything tested out, I checked it all in and pushed it to the central repo - pretty nice. But then the avoidable stuff started to bite me.

I needed two machines to test the throughput of ZeroMQ as a reliable multicast distribution system for my tick data. Nothing fancy, but I needed to have some way of replacing 29West as we weren't really using it as a solid middleware - just a multi-channel reliable multicast system. Given it's limited usage, I looked at ZeroMQ and thought Hey, if this works, I'm in business! But in order to know if it'll work, I need to actually get the ticker feeds working, put the messages into ZeroMQ, and pick them up on another box. Hence the need for two boxes.

Well... they got a few boxes with 10Gb ethernet NICs in them to make sure that I didn't have to worry about the NIC being the bottleneck, and they were ready for me to check the boxes out. As per the way things are at The Shop, the standard mechanism for getting to these servers is NXMachine or SSH. Given that I'd be testing and building, I decided to go with NXMachine. It installs pretty easily, and with this simple fix, it should work just fine.

Silly me...

I spent a full morning trying to get the NX Server to work. I knew the client was working, and I knew the server could work, but it wasn't allowing me to get a complete connection. Well... I got the connection, but when I went to actually display the X session on my box, it disconnected me. Very odd.

I tried logging the NX Server - no luck even when following directions. I tried re-installing the software - no good, either. I tried different parameters for the client - no good. In the end, I went to my boss and asked him for help. He couldn't get in either, but then realized that maybe it was because GNOME wasn't installed. Specifically, the GNOME Desktop environment.

That was the problem.

When he did a simple:

  $ yum groupinstall "GNOME Desktop Environment"

he got some 124 packages that needed to be installed. It seems that they didn't put the full GNOME install on the box as it was a "server". My previous box had the GNOME Desktop installed prior to me using it - which is why it worked. Had I stuck with a bunch of SSH sessions into the box, it would have been fine. It was just the desktop login that was the problem.

After that was solved, I was able to install boost, log4cpp, ZeroMQ, and a few other things to get this new box to the point that I was able to verify that all the code worked, and that everything compiled and ran.

Lots of grief for something as simple as not having the login desktop stuff installed.

Pushing Forward with More Codecs

Friday, August 27th, 2010

Today was the final push for the last two codecs I needed to write. I was really hoping to get both of them done, but the amount of code required in the first was just too much to get both done in a day. I had to send an email saying that I had slipped on this guy, and that I'd get back to it on Monday.

Not really hard, just a lot of code to get the messages decoded. Bummer...

Handling Fast Market Data Efficiently – Hint: Go Lockless

Thursday, August 26th, 2010

Today I was doing some testing on my latest data codec in my new ticker plant, and I ran across some performance issues that I didn't really like. Specifically, the processing of the data from the UDP feed was not nearly fast enough for me. As time went on, we were queueing up more and more data. Not good. So let's see what we had in the mix that we needed to change...

First, the buffer I was using was assuming that the messages from the exchange were not completely within a UDP datagram. This was a nice "luxury", but it's not true, and it was costing us time in the processing. It's better to assume that each UDP datagram is complete, and queue them up as complete units to process, than to have the logic in the buffer to "squish" them together into one byte stream, and then tokenize them by the ending data tags.

That was really quite helpful because at the same time I decided that it was a bad idea to use the mutex/conditional I had set up to allow the one producing thread and one consuming thread to efficiently access the data. Instead, I grabbed a very simple lockless circular FIFO queue off the web and cleaned it up to use for this UDP datagram buffering. It's easy enough to use - there's one thread that moved the head, and another that moves the tail. Simple. As long as the head and tail aren't cached on the CPUs, it'll work without locking. Simple enough.

But when I get rid of the locking/waiting, then I have to handle the case where the queue is empty and we need to try again. My solution there is to start simple and put a simple 250 msec wait. When I started testing this, I saw that there were significant pulses in the incoming data because a lot of datagrams arrived while we were waiting. So I got a little smarter.

I added an expanding delay - starting small, and building, so that we can hit it quickly if it's a short delay, but when the close comes, we'll only do a few checks before it goes to only a few times a second. That's very reasonable.

I did more tests and finally ended up with a variable scheme that had no delay for a few hits and then started stretching it out. Very nice.

In the end, I had something that emptied far faster than the UDP data source, and that's critical for a ticker plant. There's enough to slow it down later in the processing, so it's essential to start out as fast as possible.

Finally Finished Major Addition to Ticker Plant

Wednesday, August 25th, 2010

MarketData.jpg

Well, it's taken me a few days, but I've finally finished the code in my ticker plant to handle the options data feed. It's a biggie because instead of doing the same ASCII encoding that the other exchanges do, they switched some time ago to a FAST (FIX Adapted for STreaming) encoded stream to reduce the bandwidth needed to move the data from them to us. This just added a new wrinkle as we had to incorporate their FAST decoder implementation (initially), just to get the data into a binary format that we could do something with.

Then we had to adapt the code to allow for the fact that some messages from the exchanges, specifically OPRA right now, generate multiple messages to flow downstream. This wasn't hard, but it was in all the codecs, so it took a little time to get it all right and working properly.

I got it all finished, compiled correctly, and looking like it's ready to test. Time to commit it all to git and then get to the business of testing.

Fun with Exchange Codecs – FIX Adapted for Streaming

Tuesday, August 24th, 2010

MarketData.jpg

Well, it turns out that the ASCII-based exchange protocols NASDAQ, and some of the other lower-volume exchange feeds use is fine as far as that goes, but OPRA decided that it had pushed the limits of the ASCII protocol, and decided to make/adopt this FIX Adapted for Streaming - or FAST, protocol. In a sense, I can see why they'd adopt it - as opposed to writing their own, but I've read enough on the net to know that they really didn't adopt it 100% - just the compression of data part.

Basically, the FAST protocol is based on a few ideas:

  • Very Little to no ASCII to decode - no longer will there be numbers represented as ASCII digits. Most numbers are now simply integers. In fact, they only allow for three data types: 32-bit integer, unsigned 32-bit integer, and a string. WIth those, and a few decoder tables, you can handle anything an exchange needs.
  • Delta Encoding - there will be fields that are required in each message, but for some fields, the value present will be a simple increment, and in fact, it's possible to have nothing in the message, and have the assumption be that the value is simply incremented. This helps a lot. There are also values that are simple changes from the last value in the field, so duplicates can be removed. It's small, efficient, and makes for a compact encoded data stream.

The problem is, of course, that there is now state in the decoder. In general, this isn't bad, but what it requires me to do is to completely decode all the messages that I get, and the shortcuts I had that would extract just the sequence number, or just the flags for skipping the message - those are tossed out the window. I need to get all the data, and then deal with it.

This took a little while to work into my application, but in the end, I had the concept of a decoded message, and that message included the elements I had originally extracted, as well as the actual message. Thankfully, this is still pretty fast as OPRA isn't messing around with a lame decoder as it knows the point of this is to get more through the system.

I still need to do a lot of tests, and even finish writing my codec for the OPRA data, but at least I've got all the essentials of the FAST decoding working, and should be able to get moving forward again tomorrow with the messages.

Indiana Jones and The Legend of the Lost Codebase

Wednesday, August 18th, 2010

Detective.jpg

Well... I'm donning the old fedora again, and off in search of the Lost Codebase. It's really quite amazing the skill that some people have to hide code. I'm sure they don't think of it that way - they probably consider it to be exactly where they want it to be - the right spot. But if I can't find it after working in the repository for nearly two months, then it's time to call it "hidden". Yup... hidden. And that means I need to get out the fedora and get exploring.

The first thing I check is of course, the most obvious - the name of the directory. Clearly, this is a trap, for who in their right mind would put the code in a clearly labeled directory. No, that's the location for some of the code. Maybe. Hard to tell as the class files are nearly completely empty, and one would wonder if the code even compiles. I'm not fool enough to fall for that trick - typing 'make' could end of wiping out my entire machine's drive. I'm no fool.

Next, I check the similarly named directories. No luck there, but not nearly as complex, and some of the traps aren't even well constructed. In one there's no Makefile - a dead giveaway, if ever there was one. In another, they foolishly only include a handful of files. This is too easily scanned and I can see what I'm looking for isn't there. In all, a minor detour, but I have no idea where to go next.

Next I have to go with the big guns - I grep for a keyword in the entire source tree. As expected, this yields far too many hits, and I need to filter it down. Doggedly, I wrestle the filter on the grep to give me something I can work with. I struggle weeding out the false hits. I finally think I may be onto something only to have my hopes dashed when it's a simple comment and not the real code I'm looking for.

It's frustrating, and in the end, I realize I've met my match. I have to back off, regroup, and hope that when the author(s) decide to come in for the day, they have some answers to where the hid the secret directory to the code.

Oh yeah... I even checked for the hidden directories... no luck.