What's it all about, Alfie?

This morning I was working on some code and it was getting a little frustrating working with the mongoDB replica sets - after all, they are meant to be able to switch the PRIMARY as will. Fair enough, but that makes it hard if you have to target the primary in your mongo command-line shell:

  $ mongo --host xchi43mongo:32018

where we might have a replica set of xchi43mongo and xchi44mongo. Seems they should make the third box in the set be the one to talk to - but I don't write it, so I have to try one and then another if I'm wrong. Kinda frustrating.

So this morning I decided to figure this out, and sure enough, mongoDB allows for referencing it's replica sets:

  $ mongo --host dapool/xchi43mongo:32018,xchi44mongo:32018

and it picks up the name of the replica set from your command as well as the machines in the set. Nice. I can see this - it's an alias now, and I can pretty easily deal with this.

Not bad, mongoDB.

This morning, once again, I'm getting timeouts hitting the mongoDB replica set in staging at The Shop. I know mongoDB is supposed to be this amazing piece of software, but I'm seeing it right now as a major pain in the rear. I'm looking at a database whose size right now is less than 10GB, and it's a major source of grief. We are having replication issues, and the erlang drivers had to be changed, and then retrofitted. The Java drivers had to be updated to get document sizes greater than 16MB.

But those are the known issues. It's these gotchas that are really annoying. I'm trying to get a single document from it and it's timing out. I get no idea why. One box it's fine, another box it times out. Same mongoDB, but different boxes. This worked yesterday, so what's changed overnight?

Arrggghh... I'm really beginning to hate this software.

UPDATE: it turned out to be the socket buffer size on the default socket for erlang. It was at something like 1kB and I was moving MB through. Very inefficient. When we got the socket options set right, the speed wasn't too bad. Not fantastic, but not bad.

This morning the Google Chrome team released 12.0.742.16 with what it calls "UI and performance issues" - which appears to be the theme in recent weeks. It does appear that they are taking the time now to really clean things up and not focus on new features as much as clean up the codebase and polish the app as much as possible. Good enough. Sounds like a plan.

After my experience with the utility of indexes for mongoDB, I was not surprised to find that this extended to pretty small doc sets as well. Specifically, I had a lot of configuration data stored in a mongoDB collection as some 300-ish documents. But they were binary, and when I added an index on the appropriate key field of the documents, the access speed dropped like a rock. Once again, mongoDB wasn't storing the starting points of the docs, it was doing a complete table scan, and in this case, it was very expensive because the table it was scanning had a few very large docs.

Amazing.

This has become such an issue, we put code into The Broker so that the first time someone accesses one of these document stores, it ensures there's an index on the key field. It's going to make it impossible to have bad performance because you forgot the index. Sure, it may be calling ensureIndex() more than needed, but it's only once a runtime startup, and the benefit far outweighs the cost.

Just amazing difference.

I haven't had a lot of issues with Adium, but when they start another beta cycle, it's something I want to be a part of. They usually have significant features or fixes in the betas - so it's something to be a part of. So this morning when I read in chat that they were starting another with fixes for the MSN buddy list maintenance, I downloaded 1.4.2b1.

The release notes are already impressive. I did have one thing I needed to deal with - the security setting on AOL was causing me grief, but it was an easy fix.

Today I spent a very long time on a problem that was really quite simple, but very hard to find. The problem was that many of my configuration service calls were timing out - but only after other calls had successfully been sent and received. This degradation of performance was very repeatable, but equally puzzling. It was clear early on that the problem was in the mongoDB - specifically, getting data out of Mongo. This wasn't exactly clear in all cases, but the hints were very much there.

For instance, if my configuration service hit a new single-server configuration mongoDB, everything was acceptable. But if it hit the staging replica set, it timed out. All this was with reads, so there's no chance of the writing coming into play. Very odd, then that a replica set was slower.

We kept digging, and went so far as to turn off the replica set and turn it into a single server. This yielded the same times as the replica sets - which is to say "slow". Maybe it was the hardware? Nope, a new single-server instance on that hardware was fine.

Finally, after several hours, we got to the heart of the matter: my configuration service was hitting the authorization mongoDB for the auth token to make sure the user was allowed to hit the configuration data. Bingo! We had a 266,000 entry mongoDB table without an index!

All that was needed was to type in the mongo shell:

  db.token.ensureIndex({token:1});

and the times sped up dramatically. This was the key - we didn't look at the data - just the hardware and the software. It was a long day, and while I'm glad we got this one out of the way, it didn't solve my problems 100% as my larger queries are still timing out. David says he's going to look at the the emongo driver this weekend for possible causes. He added the replica sets support to it today as we needed to move away from erlmongo as it inly uses one socket connection to the database. emongo allows for connection pools, which is going to help me a lot.

Well, the Google Chrome guys are still putting the UI polish on 12.x, as they just released 12.0.742.12 with the release notes saying it's just UI issues and a few sync issues. It really appears that they are going for a stabilized release of 12.x for beta as I've read that they believe 11.x is headed for release. Time marches on...

This morning I started to see some disturbing problems with the configuration service written in erlang for The Broker. It's all backed by a mongoDB that's currently configured as a replica set, and after a few apps were up, the speed took such a hit as to start to time out my requests. I wasn't sure what it was, so I took the advice I was given, downloaded the latest pre-built binaries and ran a stand-alone install on one of my boxes.

It was really pretty amazingly easy. You unzip the tarball and just run it. I made a simple directory to put all the data in, and away it went. Very nice. I was able to reconfigure my Broker code to hit this database for the three Brokers I had in my little dev cluster.

Then I ran the app.

Very nice response times. Very. I let it run for an hour or so, accumulating data and saving it to the new mongoDB. Then I stopped everything, and restarted it. Rather than hanging, as the replica set did, it started up with a little slowness, but everything worked. It's not mongoDB, and it's not the way I was using it. At least not in a single server mode.

Someone did a little digging and found that the 1.8.1 release might have released a bug in the replica set negotiation. So we're going to get the "final release" source and put it on the boxes and see if that doesn't help. But we need something. As it is now, replica sets are really not going to scale like we need.

This morning I noticed that there was an update to WordPress to 3.1.2, and so I took the few minutes to update all my installs at HostMonster. It's a single security issue that would never effect my sites as I don't allow for contributors, but still, it makes sense to patch any and all security holes just in case.

Seems the Google Chrome guys are busy - today it's 12.0.742.9 with the release notes calling it a release candidate. Interesting. If they are planning on moving the 12.x branch to 'beta', then they'll soon be bumping the 'dev' channel to 13.x - which is an interesting number to say the least. I wonder if anyone is superstitious? In any case, they are trying to polish up this version for the release, and that's always good news as it means a better experience for everyone.