Making Code Deadlock-Proof
I had another lock-up in my tick injector that uses the Vendor library that I've had so much trouble with in the past. It appeared that it was in my code this time, and that surprised me quite a bit because I was convinced that there was no way I could have a deadlock because there was no section of code that had both locks in place at one time. If you don't have a thread that holds two locks then there's no way you can have a deadlock. Delay, sure, but no deadlock.
So I started going through all the code and checking to see if I missed anything. Well... sure enough, I had a little section where I had a stack locker active and then called a method that grabbed the other lock for a bit. I was able to clean this up, and I'm hoping this had to be it, but in truth, I'd have to have another section with two locks to have a deadlock. And I didn't find that section section.
I did, however, find a few places where I thought that encapsulation was a better plan, and so I made methods that were, themselves, thread-safe, and then put those into the code where I had originally had the locks. These should be essentially "no-ops", but in fact the locking will be slightly different and in that I may have helped myself even though I hadn't seen a section of code with two active locks.
I've also added in some more logging into my addInstrument() call - where it appears the lock-up was, just to see what happens next time. I'll run with this and we'll see if and when it fails what the logs tell us.