Never a Dull Moment
This morning one of my servers started sending out prices of nan, which is clearly not the right thing to do. This caused my downstream systems to stop ticking, and that caused a lot of production issues. First off, I had to find the problem, and patch the downstream systems so that they would be effected by nan values anymore. For the most part, they were protected, but from this one server it was far far too optimistic. So I had to batten that hatch down and get those changes into production as soon as possible. Then I could turn my attention to why?
The why? turned out to be a volatility curve that was primarily short-term, and rising significantly at the end - coupled with a far-term warrant that ended up with a very large volatility at it's expiration due to the interpolation of the curve. The fix was to note what the problem was and then to put into the code a configurable parameter that would 'cap' the volatility at expiration and not allow the options to be valued if the volatility exceeded a given cap. This works great, but I tell you... there's nothing like coding a fix for a system with 50 users breathing down your neck. It makes you make sure you're right - and fast.