When it Rains, it Pours
This morning I was hoping to see a nice, stable, web app that I had to patch together last night. Instead, I had a crashing app that wasn't working at all. Crud.
It seems that the report updating thread - that worker thread that updates all the reports for new data arriving in the system, was dead. And after a restart, it died in only a few minutes. Not good.
So the first thing I did was to put the HashMap back into the code for the LRUCache which gave us the stability, but it was a ticking time-bomb. Around 2:00 this afternoon it was going to run into that same 12 GB limit and we were going to be in a mess of trouble. Not good, but I had no choice.
Once the HashMap was back in the code, I started looking for the fall-out of using the LRUCache. Because, in theory, there's no reason that the LRUCache should not have worked just like the HashMap - it's a Map, after all. It's just a question of removing some aged entries.
The code was telling me that once again, we had optimistic coding and the problems were not as easily removed as switching out one Map for another. The problems came about because the code assumed that everything was going to exist - no checks whatsoever. With the LRUCache not having some data that was once there, we had an easy NullPointerException. With no try/catch block on the thread, it simply died. Wonderful.
The changes were that horrible - basically just putting in the checks that should have been in the code in the first place, but the idea of not having something like a try/catch block on the Thread's run() method is just amazing.
Why?
Oh... I know. It's never going to fail.
I finally got something that stays up, looks like it's all OK, and most importantly, the memory footprint is stable. It's been running in NYC for a few hours and the memory is flattening out nicely. One issue was that the LRUCache was really only needed on the archive storage for the reports. The other HashMap was really just a mapping of the report description to the last report instance. That makes sense to leave as a HashMap as it won't be growing without bound like the archive will.
The only way to know is to find out what tomorrow will bring.