Complex Systems can be… Well… Complex
Today I have spent most of the day working on a problem we ran into this morning about the FX conversion of the dividend curves in my server. I had to dig into this problem for quite a while to figure out what the problem was. At first, it seemed pretty clear - the FX rate for USD/CNY was 1.0 and it should have been a tenth that. The problem was Why?
The first thing I did was to be sure that the FX rate was now not 1.0. It wasn't, but this is where the complexity comes into play - I forgot to realize that the server is a multi-machine, multi-process entity, and the FX rate I was looking at was, indeed, not 1.0, but that's not the FX rate that was being used to convert the dividend curves. The FX rate used for that was still 1.0.
After looking at this for about 10 mins., I realized what the problem was, and realized that by restarting one of the components I could have it reload it's FX rates, and this guy was the one doing the dividend curve FX mapping. In the end, it was an easy fix, but it made me realize that I needed to have a better way to have these dividend curves mapped, and so that's what I set out to do.
Interestingly enough, one of the strengths of the server is that the components are very loosely coupled. This means that they can be independently restarted and the 'whole' will not suffer. Things are re-tried, re-sent, and life goes on. Very resilient. Problem is, this means that you need to have an exceptionally good communications system to make sure that what you want done to one component will be done to all effected components. Case in point, telling the system to reload an individual FX rate.
Clearly, restarting the components is an option, but that's not very user-friendly. What I was looking for was a way to tell the components when something was changed and tell them of that change. Problem was, this would represent a significant addition to the protocol that was already in place between some of the components. Not something to do lightly. Especially, if there's an easier way to accomplish the same thing.
So I kept digging. It turned out that the only reason this one component had the FX rates was for this dividend curve mapping, and if the data coming back from the database didn't need to be mapped, then this would no longer be an issue. Idea: Make the FX conversions in the database calls. Problem: can't slow things down, so I can't add a lot of processing and I have to be careful about the FX rates I use.
Turns out, the first worry wasn't too bad. I simply looked at the original data as a 'rough cut' of the data. If any FX adjustments needed to be done, I did them en masse. This meant that most things didn't experience any slowdown, and those that did, most times the curve would be converted all at once and not a point at a time. Nice.
The last problem was a little more difficult. I was able to work towards it, and I only had to make two assumptions - that the 'latest data' for both parts of the FX conversion had to occur on the same date (good idea anyway), and that it's always the 'posted source' so that we had the best marks in use. I thought these two facts should be true, but I wanted to run it by someone who's had a few more years at the data-side of things than I. After I explained this to him, he was convinced that they were valid assumptions/rules and I then knew I could complete the FX conversion within the database.
What's the point? Well... complex systems are complex. It's in the name. Even when I've worked with this guy for years, there are parts and interactions that you might not realize and it plays tricks on what you think should happen. Then you try it and you get a different result and start to think about it, and then in a flash it comes to you that you were wrong and it was right.
Kind of like those simulations where you don't program in certain behavior, but the higher-order behavior is a direct result of the low-level rules, and so the complex behavior is displayed on your simulation. Wild stuff. Cool, but when you're trying to make a simple change and the complexity of the system is staring you right in the face, it's giggling at you. You are the one that needs to adapt.