Digging into Noisy Data
I've been digging into this problem of 'noisy' data coming out of one of the systems I've inherited here, and at first I thought it was a problem of loss of significance due to subtraction, but that turned out to be part, but not all of the problem.
Today I was digging again, trying to find out what was possibly causing this. I was looking at trade flow... and greeks being sent in... nothing seemed to explain it. I talked to the original developer of the system for a while and together we didn't some up with anything.
I left thinking that the best I was going to do was to shotgun the problem and hope I hit something that might be close enough to the problem to point out the solution.
I sat down and started to look at the code again, and then it hit me. At least I think it hit me. I have a very plausible reason in mind, but in order to know for certain, I'm going to have to reconfigure some machines and run them tomorrow.
Basically, I think we're getting into a situation where the multitude of portfolios are feeding data into a single collector for P/L calculations. The problem might be that because several of these portfolios are normalized to the same contract, we're get an increased feed rate on that normalizing contract, and that is upsetting the calculations.
My limited understanding of the messaging of data might be off. That's why I need the isolation test. If the data from my test is smooth, and the data in Test and Production is noisy, then we'll know that we have a solution to the problem. The wrinkle here is that this solution is going to mean that we need to have a lot more hardware to run these portfolios than originally thought.
We'll have to wait and see tomorrow.