Historical MarketData Source and CKString Fun
Yesterday afternoon one of the developers came by to ask me about finding historical prices for delisted symbols. "That's hard", I said - because it is. Bloomberg has historical data back some 20 years, but you have to know their symbology to get it out of Bloomberg, and one of the things my MarketData server does is to allow us to use our own symbology and it converts this to the symbology of the data provider (in this case, Bloomberg) and then requests the data. This is great so long as you can do the symbology mappings. The problem comes in with delisted symbols - they aren't in our mappings table so I can't convert from our symbology to Bloomberg's. This means that eventhough you might be able to get at the data, you don't know what to ask for.
I told him he'd have to investigate another data provider and get the data from them. This other provider needs to have the historical data and some sense of the re-use of tickers so that given a date and a ticker it knows that there was one - and only one instrument for that ticker on that date. This kind of historical data is not common, but it's available. You just have to know where to look.
So we started looking.
There's a project in the Bank that handles a ton of historical data from a lot of sources - one is Bloomberg. So we did a few tests to see if this source - which was in my MarketData server as a provider, had the data for a delisted company. Turns out it did. But at the same time, I found that there were problems with how I was calling and using this data source. So I dug into fixing them.
The first was pretty easy - when I requested data of this source, if I received nothing, then I assumed it was an error. In reality, if there's no data, they returned nothing. So if you asked for a delisted symbol after it was delisted, you'd receive nothing. I fixed those tests up to allow for nothing to be returned. Easy enough.
The second one was more interesting. The data fields you request should stay the way you request them - case-wise, but I wasn't preserving the case of some requests and mismatching the case of others. So I added several nice little methods to CKString to allow the user to copy a string and uppercase it (the only existing method uppercased the string in-place) and then I added equalsIgnoreCase() like Java's String to allow the code to not have to worry as much about case in the tests. When I put these in and realized that I needed an uppercase copy of the fields for getting data from the source, caching it, and then using the original field list to pull it out of the cache, things worked beautifully.
In the end, it was a lot of fun to add these things and see how they simplified the code I was trying to write to fix the problem. Lesson learned: add power to the underlying libraries and that will make any higher-level fixes much easier.