Updated Configs to Work in Two Datacenters
The current plan was to have Production running on the old EC2 boxes and have UAT run out of our data center until we were sure things were OK, and then switch Production over as well. This seemed like a good plan, but there were issues with this and management wanted to run the essential production data, and UAT data, in EC2 and then run the non-essential production data and UAT data in SNC1. This means that there would be multiple boxes running the same code hitting the same data sources and sinks, but covering different regions.
Sounds reasonable, and even safe. So let's do that.
The issues with getting a different config for UAT in one datacenter is that all we really have is hostname versus hostname -f, and I had to use that every place I could. The wrinkle came in when looking at the configuration, as that isn't looking at the machine name - just the environment setting - 'hat' or 'production'. Not so easy.
But I worked on this all day. It was not easy. And then I was ready to test.
It wasn't pretty.
The problem comes in that we aren't synchronizing the work between data centers, and this makes the later processing steps (prioritization) fail because you only have part of the picture. There's no easy way around it.
The next problem was continued Couch errors. Yes, using my cacheing endpoint helped, but we'd still get problems now and then. No easy solution in our code.
So in the end, almost a wasted day. Almost.
Not a great feeling.