Tracking Down Nasty Memory Issue – Patience is a Virtue
I've been trying to track down what I believed to be a nasty memory leak in my code today. The short-cut to the answer is that it wasn't a leak, and it wasn't in my code. But I'm getting ahead of myself.
The problem was manifesting itself as steadily growing memory on some of my ticker plants. In truth, it was probably all of them, but it wasn't effecting all of them equally. I have spent a lot of time on this over the past weeks, and today I was going to get to the bottom of this for sure.
So I started digging into the problem by shutting things off. What I found was that if I was listening to anything on the UDP socket and doing anything with it I was getting about an 8-byte increase every two seconds. Very odd. I had turned off ZeroMQ at the time, so the messages were just getting dropped in the trash, but they were being processed completely up to that point.
I was trying everything, and then I had to run to a meeting. I left the test running because I needed to hurry. It wasn't going to consume the box in half an hour, anyway.
When I came back I noticed that the memory had stabilized!
Now it was getting interesting. Very interesting. I started tracking things down and it turns out that the ZMQ_RATE parameter was a major factor in the terminal memory value. I then wrote up a simple test - something that I knew the ZeroMQ guys would appreciate, and started running it.
Again - major dependency on the value of ZMQ_RATE. I'll have to do more work on this tomorrow.