Getting Ever Closer to the Goal
This morning has been better than most, but still challenging. Today I was able to handle the open with my ticker plants, and the client had a latency of less than 368 msec. This is great news. I had expected the upper limit at around 400 msec, and this was based on the tests I've been running with the individual components. At about 100,000 msg/sec from the exchange, a processing time of 0.4 usec equates to 400 msec. So we're probably a little shy of the 100,000 msg/sec, and that was what we were seeing.
Great news.
But it was not without sadness. The multiple-producer/single-consumer linked FIFO was not being a nice citizen, and corrupting the malloc/free heap. It was causing a lot of problems. It was easy to track down as it was a big user of new and delete, and it was newly added to the code. When I added a simple spinlock to the code, everything cleared up, and slowed down. So I went on a little fishing expedition for some other implementations to look at.
What I found was that by starting with a Node, and pointing the head and tail at it, you can really simplify the code. Also, you can segregate the head and tail pointers to their single, respective, methods so that there's no chance of a cross-contamination. Much nicer. I put that in and things are looking much better.
I'll still need to test for several days to be sure, but things are looking a lot better. Next up is to rebuild the two other development machines to be Ubuntu 10.04.1 as I've had some real issues with this code on CentOS 5. I believe it's due to the memory allocation scheme in the kernel, and Ubuntu is a lot newer than CentOS 5. Hard to say. What isn't hard to see is that on CentOS the code won't run for 2 mins, and on Ubuntu, it seems to run forever.
I'm hoping to get these boxes rebuilt today and then all my stuff moved back over to them so I can be working off the set of three for tomorrow's open. That would be a really nice goal.