Refactored Code for Sharding Redis
Today has been a very interesting day due to a speedy refactoring of a topology and analysis library to shard redis across four instances - up from one. This was brought about by a change we tried to deploy to the cluster, and then rolling it back only to see that we were so close to capacity on the single redis server that it wasn't ever going to work.
Redis is great - it's really quite good, but it's single-threaded, and that means that you need to have some scheme of sharding the data across instances if you want to keep pushing the throughput of the system up. I've found myself doing this a lot in the last month - we're adding more messages, and the single redis instances just aren't enough. So we have sharded several of the systems. It's turned out to be exceptionally efficient as well.
So now it was the experiment analysis that had to be sharded. This was different than the others in that it wasn't a single place that redis was accessed - it was all over the one namespace that did the data collecting. It meant that operations that used to be a single function had to be broken up because the keys might not shard to the same instance, and it was important to make sure that everything ended up in the redis instance that it was expected to be in.
Interestingly enough, the complete refactoring was a complete success - first try. No need for edits or changes. It Just Worked. Very nice experience.