So Amazingly Easy to Make Threading Mistakes
Today I've been fighting a lot of issues in my new UDP-based transport for my ticker plants, and I've tried changing queues - multiple times, more logging, structural changes - all kinds of things that seemed to be the issue, but the real issue seems to be that I was not being careful enough starting my threads. Yup... threading is hard, and it's easy to make mistakes.
The set up is that I have a lot of "single-consumer" queues in my system. If you try to hit them with multiple consuming threads, you're going to be very sorry. Couple that with the fact that I was trying to be clever in my thread initialization and processing code and "covering all my bases" by putting redundant code in the initialization and processing blocks. The problem is, if you get the initialization complete, the next thing is the first trip through the processing code, and if you duplicate the code there, but have an off-cache flag set, it might appear as though you haven't properly initialized the threads.
So you start more.
Ouch.
The obvious answer is don't try to be cute. Have code in one place and one place only and then make sure it's successful. When I did that, all the other problems "magically" disappeared. Amazing.
Not really. It's obvious if you have multiple consumers on a single-consumer queue you're going to get into trouble. Also, it'll appear as though you are loosing data, when you really aren't. It's a mess.
Well... it seems to have cleared up and I'm glad. It's been a major pain.