Shutting Down Multi-Threaded Apps Takes Special Care
Today I spent a lot of time working on getting rid of core dumps when I shut down the ticker plant. This seems very obvious, but after I just did all the work to fold in the configuration and then even more work today to get the authentication token into all the configuration code, it required that I change a lot of little things around. The upshot of that was that the subtle interactions on the shutdown were quite broken and I had to fix them all up again - but not revert to the old system due to the configuration and authentication changes.
Little things I ran into made this a lot harder than it sounds. We have boost's asio to contend with, and the fact that we need to cancel any pending I/O with them on the shutdown. However, that's going to generate an error in the socket communications, and from that we'll try to shutdown the socket. This is a nasty loop that makes it very difficult to unravel.
What I had to do was to be very careful about the two different conditions, and if I'm being interrupted, then I need to assume I'm being told to shutdown and not assume there's something wrong I need to attempt to recover from. Thankfully, the error messages are clear enough to make this possible, but it's still a lot of detail work about when things are happening and who's responsible for doing what.
All this took me the better part of the day - with the initial part being trying to get the timing of the initialization of everything right as well. It's like I spent two days getting nowhere as I didn't advance the codebase one feature - I just made it possible to use the configuration and authentication tools we have to use. That's important, do be sure, but it didn't feel like I got a lot done.