Coding on Thanksgiving – Trying to Get Performance Up
Well… the addition of the new divisions (added just to meet the crazy deadline) didn't go as well as I'd hoped. Thankfully, I had good New Relic data to look at and see what was happening in the process(es). What it looks like is that there are large sections of code that aren't really taking advantage of the machine, and doing too many things serially. So I set about attacking them.
On Thanksgiving.
First, there was one process that was doing a lot of writing to Couch serially. That was easy enough to fix with a simple java Executor and a couple of threads. I also moved all the single document writes to Couch to bulk stores so that we got much better performance when we had all the data to write up-front.
The next thing was to try adding timeouts to the CouchRest API just to see how that would go. I'm hoping that these REST calls that simply don't return can be trapped in a simple "total timeout" and then retried. As it is now, some of them simply never return, and that's no good at all.
In the end, I had to get the speed up. I'll see how these changes work tonight and make any needed adjustments in the morning.