Working on Serialization – Big Payoff Potential
Today it was all about serialization. The problem I've been having is that my tests hit the database for some 400,000+ instruments, and because of that, each test takes about 5 min to run. The database is not fast, but it's the fact that it might only take a minute to get the data, it's the fits and starts with iODBC that's hurtful, and the database is really in bad shape. So I decided this morning that if I could get serialization of my objects going, then I could use persistence, and that would make my tests a lot faster.
So today it's all about serialization.
It's not all that hard - just a lot of details and making sure things work properly. The big problem is that when working with instruments, there's a lot of references going on, and getting all that right in the serialization is non-trivial. Still, having seen it work for me time and again, it's definitely worth it.
At the end of the day, I had something that worked, but it took far too long to persist everything. I was serializing each instrument family (a stock and it's options) independently, and saving that to it's own mongoDB document. This was just too slow, as it was taking 30 sec to write out all the data. In the morning I'm going to pack things a little differently and put as much in a document as I can. This way, it's fewer, bigger documents, and not so many small ones.