Optimization Brings It’s Own Problems
Today has been an up and down day... on the one hand, the changes I made yesterday have increased the speed of the ZeroMQ transmitter to new heights, but with that has come other problems - most notably in CentOS 5. Interestingly enough, I'm now seeing real problems with the ticker plants on CentOS 5 when the message rates get to even moderate levels. It's always in the ZeroMQ code, but I've dug into that code, and I can't see what it's doing wrong other than calling malloc() quite a bit.
Interestingly enough, when I run it on Ubuntu 10.04.1, it's fine - so I'm currently inclined to think that it's something int e linux kernel to do with memory allocation/deallocation that was fixed in the later kernel, and simply not available for CentOS 5. We'll have to run more tests on Monday, but for now, I'm at a bit of a loss.
Thankfully, we're moving to Ubuntu, and so long as it runs there, I'm good to go. The only wrinkle is the client - does this run on a CentOS 5 client? Don't know, and we'll have to see next week. It could go either way.
I did a lot of other little optimizations today - things that weren't an issue before, have now become an issue because of the speed of the overall process. Locks on maps that aren't really needed... things like this. I spent the entire day finding these little issues and fixing/updating them, and checking CentOS for a possible fix.
They all worked on Ubuntu, and it seemed that nothing I could do was going to get it running on CentOS again. We'll have to start back at it on Monday.