A Very Busy Day, Indeed
This morning was a lot of work that turned out to be something almost funny in retrospect. I had just completed the conversion of the data source from the old MS SQL Server databases being accessed through SQLAPI++/iODBC/FreeTDS to using PostgreSQL using just the SQLAPI++/PostgreSQL libraries. Things were running, but the greek values were all wrong. I mean not even close to right. So clearly, I had messed up on one of the data import statements or values - somewhere. So I had to track it down.
The problem was that there were a lot of values coming out of the database tables, and I needed to very each one to make sure there wasn't an extra factor of 100 or 10000. So I started doing a lot of checking. Pretty soon, it appeared that there was nothing wrong with the data. Not a single thing. Very odd.
So I backed off and started looking a little wider. Maybe it was the set-up outside the calculations themselves. So I looked at the Holiday Calendar. In the previous holiday calendar SQL call, I received the list of holidays. When I looked at what I was putting in now, I realized that I was looking at a calendar - not a Holiday Calendar. Basically, I was telling the FE Model that every day was a holiday. Very bad. So I changed the SQL on the calendar to only pick out the market holidays and BINGO! The numbers fell right back into place. Very nice to see.
But after that I had to giggle… Every day a holiday. Kinda funny.
The next problem that was sitting on my "TODO" list was a core dump that we got from having a web client hit us as hard as possible - pretty hard, in fact. The problem was pretty simple, and as soon as I saw the back trace, I knew what the problem was: concurrent modification of an iterating container.
Basically, I have been using the TBB concurrent_hash_map to hold things, and in this case, it was really back-firing on me. The TBB containers are thread-safe, which is great, but when you're iterating them, it's possible to have one thread think it's got something, and another thread to delete it out from under them. No fun, and almost no way to clear it up with these data structures. What you really need, and what I went to, was a nice read/write spinlock that we can then ensure isn't getting modified during an iterating pass.
With this in place, we don't have to worry about the excessive load of the exclusive lock, we can typically use the read level on the mutex, saving the write for those times we add and remove pairs from the hash map. Then we can back off and use something like a boost::unordered_map for the container and be better off all the way around.
At the end of the day I started working on the next feature I was adding - the ability to pull out of a running Engine a provided list of values for a group (or all) instruments in the engine. For example, a way to say Give me all the last trade prices and sizes for everything in this profile. I'm going to need it, so I got started on it knowing I wouldn't be able to come close to finishing it today.