What's it all about, Alfie?

Today I needed to run some tests on my ticker plant code to get a decent scale for ordering hardware and network taps. The new machines are going to need double 10Gb ethernet cards - one for the incoming feeds from the exchanges and the other for the outgoing packets. Right now I don't split it up like that, but I know I'll need to in the real UAT testing. But today I just wanted to get as close to "real" as possible - given that I don't have a lot of the supporting data sources I'm going to need before UAT.

The big missing data source is the mapping of the exchange symbol to "security ID" - an internal unsigned integer that is generated in the database and used for all references to an instrument. I'm expecting that it'll be a simple data service where I'll open up a subscription channel to the data service and issue calls to map the symbols to security IDs. I have the code to map these (in both directions) so it's only necessary to get this data once, but I need a source of this data.

Well... not really. For these tests all I need is a unique ID for these guys. So let's make them up. Easy. I'll implement the "lookup" method on the class to generate a uuid_t, which is a random 128-bit number, and use the first 64-bits as the "security ID". I'll pass this back to the mapping method and it'll think it's real. For the sake of these tests, it's good enough as we just need to have these in order to check on the conflation of the data stream.

When I fired it up running on 1/24^th of the OPRA feed it used under 10% of one CPU. A full-tilt feed using less than 10%! You gotta be kidding! I watched it for a while and if you toss out the CPU usage of the terminal that is streaming the log data, it's well under 10%. From time to time it spikes to 40% - not quite sure what that's all about, but it's not even 1% of the time.

If we assume we can get 4 channels on one CPU, and factor up the memory, it looks like we can get the complete OPRA feed on one 8 CPU box with 32 GB RAM. If we add another for all the remaining feeds, which is a good estimate, we're fitting the complete ticker plant into two small boxes. Pretty wild.

Past wild... that's better than I'd have ever believed. Sweet.

This entry was posted on Monday, September 27th, 2010 at 1:44 pm and is filed under Coding, Cube Life. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.