The Effects of Being Marginalized
Part of the New Direction that we are embarking on is to re-examine the topology that runs the data feed. Now I had built this, and done experiments covering months to get this to the point it was. Yet one of the managers talked to another Storm developer in another division, and that developer convinced him - without any knowledge of the specifics of the topology - that what I was doing was "All wrong". So my manager told me to work with this guy to fix it.
So I followed his advice and made the changes.
It didn't help, but it didn't make things worse.
Now today I'm being asked to revisit the topology and "Scale it back" because my manager is convinced there is something else in play here. OK, so I do as I'm told - because they are clearly not interesting in my opinions or we'd still be at the topology structure we had before all this.
So I start the experiments with a baseline, and then I start halving things: half the workers, half the bolts - all to get numbers to see if I'm going in the right direction. I don't fine-tune things until I'm really close, and half is easy to work with because we're still doing coarse tuning.
After about three experiments, we're far better, far faster, and have higher peak capacities during time of load. All is looking very good. Then I look at the Storm UI to the Capacity numbers, and I fine tune a little. This guy is a little high, so add some. This guy is very low, he doesn't need as much. Not a lot, just a little tweaking.
In the end we're looking a lot better from all the metrics. Good.
And the values?
Yup... just the ones I had before all this started.