MongoDB Replica Sets Issues
This morning I started to see some disturbing problems with the configuration service written in erlang for The Broker. It's all backed by a mongoDB that's currently configured as a replica set, and after a few apps were up, the speed took such a hit as to start to time out my requests. I wasn't sure what it was, so I took the advice I was given, downloaded the latest pre-built binaries and ran a stand-alone install on one of my boxes.
It was really pretty amazingly easy. You unzip the tarball and just run it. I made a simple directory to put all the data in, and away it went. Very nice. I was able to reconfigure my Broker code to hit this database for the three Brokers I had in my little dev cluster.
Then I ran the app.
Very nice response times. Very. I let it run for an hour or so, accumulating data and saving it to the new mongoDB. Then I stopped everything, and restarted it. Rather than hanging, as the replica set did, it started up with a little slowness, but everything worked. It's not mongoDB, and it's not the way I was using it. At least not in a single server mode.
Someone did a little digging and found that the 1.8.1 release might have released a bug in the replica set negotiation. So we're going to get the "final release" source and put it on the boxes and see if that doesn't help. But we need something. As it is now, replica sets are really not going to scale like we need.