The Hidden Gotchas of Dodgey Design
I've been working on this new deployment of this app I inherited and today I was nailed with a design decision that really shouldn't have been in the code in the first place, but was, and it had an ripple-effect that was really quite spectacular.
The design decision was to save every generated report for the lifetime of the web app. This was done so that anytime during the day a difference report could be made (on the server) between any two points in time. The reason for these difference reports was to enable the client to essentially freeze the report and then track the changes to the frozen state as time progresses.
One might say that this was the job of the client code - face it, it's already getting the data - it just needs to hold on to one dataset and then difference the incoming set to the saved set. It'd be minimal coding, easy, but it wasn't done. Nope.
So... when we added a lot of fields to this release, each report is now much larger, and we've added the roll-ups by product which only adds to the data per report. What happened was that by 2:20 pm the memory usage of the Tomcat instance was at 12 GB! It was slugging through Garbage Collection and I had to restart it. That helped, but I had to get a solution, and fast.
I talked to the original author and he suggested an LRU Cache on the data and not hold all the results - only the last n that have been accessed. It turns out, the implementation of an LRU Cache was pretty simple. I added that in place of the HashMap data structures for the retention of the reports and it appeared to work just fine. I checked that things worked, and that was what I put into production for tomorrow.
This should help a lot.