What's it all about, Alfie?

This morning the Google Chrome Team upped the major version number to 23.0.1243.2 with a pretty decent set of release notes. The inclusion of the V8 3.13.1.0 javascript engine, and updating WebKit to 537.6 are both really nice.

I was wondering if the least-significant updates of late meant that we'd be seeing this, but I really thought they'd just hit a plateau. Guess not, and that's great.

I can say that the refresh of a page is amazing. Very nice, and this release just keeps moving the bar up. It's nice to see forward progress on Chrome.

Because we have several data centers - including boxes at EC2, the standard set-up at The Shop is to have several proxies - forwarding all traffic through a port to a datacenter. So all traffic to the eastern EC2 center leaves my laptop on port 1234 (for instance). This means that when I want to use the CouchRest ruby client for CouchDB, I need to do something like this:

  require 'couchrest'
 
  CouchRest.proxy('http://localhost:1234/') if use_proxy?
  @db = CouchRest.database('http://whackamole.east:5984/hammer')

to connect to the hammer database on the whackamole.east server in the east EC2 datacenter. Not hard, but there seems to be a catch in there somewhere.

For when I try this, I get about 2000 of 4000 documents saved and then I get this problem. A nasty stack trace that appears to be about some timeout, and then an attempted reconnection. What's very interesting is that it has to be in the proxy handling code of CouchRest.

Why?

Because when I do a simple port forwarding on my box to the database server on the other box, and use that port - thus bypassing the need for the proxy setting, everything works. Also, if I run it on a box in the east EC2 datacenter so that I don't need a proxy, then everything works.

I need to dig into the proxy code. I'm not exactly sure what I'll find, but it's gotta be there.

I've used mongoDB for a while, and it's a decent document database, but I've now been using CouchDB for a day or so, and I have to say that CouchDB beats mongoDB hands down! There just isn't any comparison - even though there may be slightly different targets, I'm not convinced that you can't do everything in CouchDB that is a "strength" of mongoDB.

The one thing that mongoDB has, that I wish CouchDB had was a nice CLI. It would be really nice to be able to do ad-hoc queries against CouchDB on the command line, but as a point of fact, you can do all that with a Temporary View… and get what you need. It's a close point, and certainly a mater of taste.

But for everything else, CouchDB is a far better implementation of a document database. The views are a fantastic feature. Their speed and power is really very impressive. I can't imagine that this is all running on my laptop. It's really pretty neat.

I think I've found my new document database - CouchDB. Fantastic!

This afternoon I've been doing a lot with CouchDB's map/reduce framework for querying data out of CouchDB. The terminology is pretty simple: a Document can hold multiple Views where each view has a Map component that looks at each document in the database and returns something based on the inspection of it's data, and an optional Reduce function that takes all the results of the Map function calls and reduces it to a smaller dataset.

It's pretty standard in a lot of languages: first you operate on the individual elements in a collection, and then you summarize those values. In CouchDB it's all in Javascript. That's not bad, I've done a lot of that in my day, so it's pretty easy to get back into the swing of things.

One interesting issue is that CouchDB is written in erlang, and while I don't see myself digging into the guts of this thing, it's interesting to know where it all comes from, as it makes it a lot easier to understand why they chose Javascript, for instance.

Anyway, let's say I want to see all the merchants that have no OTCs assigned to them. I'd create a Temporary View in the CouchDB web page, and then in the View Code I'd have something like this:

  function(doc) {
    if (doc.meta.label == "QuantumLead.results" &&
        doc.otcs.length == 0) {
      var key = [doc.division,
                 doc.meta.created];
      var blob = { name: doc.merchant.name,
                   sf_id: doc.merchant.sf_id };
      emit(key, blob);
    }
  }

The interesting parts here are that the emit() method is really the action item in this function. When we want to add something to the output for this Map function, we have to call emit() with the first argument being the key, and the second the value. The key, as shown here, can be a multi-part key, and the value can be any Javascript object.

The thing I like about the use of Javascript here is that the attributes look like "dotted methods" and not hash members. This makes it so much easier to reference the data within a doc by just using the key names and dots. Very nice use of Javascript.

So now that I have my first few Views and Documents in the system, I need to work on getting things out of these calls, and into some nicely formatted output for the important demo that's coming up.

This morning Jeff is still having problems getting CouchDB deployed to our Amazon EC2 machines, and it's almost certainly due to the deployment system that's in place in The Shop. It's something I completely understand, but it's also based on the idea that you can't trust anyone. That, and it's an old RedHat-based distro that I know from experience is not as easy to deal with as something like a more recent Ubuntu.

Still, it's just the way it has to be, as that's the only way Prod Ops can deal with things, so there's no real way around it. The problem is that you need to be able to build the code on one box, and package it up - similar to an RPM or a deb package, and then deploy it across a lot of machines. All well and good, but Jeff is having a horrible time getting CouchDB 1.2.0 compiled on his build box.

There are some things he's trying, and even seeing if the other folks around here have any ideas. But the latest attempts have left something that looks like CouchDB running on the server, but when I go to add things to it, I get a nasty stack trace about 'Connection refused' after some kind of timeout. I've inserted about 1500 documents of the 2500 I need to, and it stops.

At the same time, I was able to use Homebrew to simply:

  $ brew install couchdb

and then follow a few instructions about getting it to run on my login startup, and that's it. It Just Works.

I would say that this would also be the case if we were looking at standard Ubuntu boxes in EC2 or Rackspace, and using yum or apt get. The real question is why do we need to do these custom packages for Open Source software when they are so easy to just install?

Again… no way to know… no way to answer. It just is and that's it.

Jeff, the guy recommending CouchDB as a document database for our app, suggested that I look at CouchRest as a nice ruby client for CouchDB. And the docs look impressive - so far as they go. It's pretty easy to open up and use a database:

  require 'couchrest'
 
  @db = CouchRest.database('http://localhost:5984/megafun')

and then saving a document is pretty easy as well:

  @db.save_doc({ one: 1, two: 2 })

even doing a bulk store of multiple documents is easy:

  @db.bulk_save([{ one: 1, two: 2 },
                 { one: 1, three: 3 },
                 { one: 1, four: 4 }])

But the main docs don't really say anything about using a proxy, and in The Shop, with lots of hosts in Amazon's EC2, there's a lot of proxy work that we have to do.

Specifically, to get to Amazon's East datacenter, we have to use a re-directing proxy on our laptops and there was just nothing in the docs about using a proxy, so I had to dig into the code for CouchRest, and thankfully, I've learned a bit of ruby in the last few weeks, and the support was already there!

Because we have servers in EC2 east, I couldn't hard code the proxy usage, but using the pattern we have used for other proxy-based access, I was able to very quickly set up the config files for the CouchDB databases, and then in the code say:

  class Database
    def self.database
      CouchRest.proxy(AppConfig.database.proxy_uri) if AppConfig.database.use_proxy?
      @db ||= CouchRest.database(AppConfig.database.uri)
    end
  end

and then in the rest of the Database class I could just reference this database method.

The things I'm doing have a lot more to do with how we want to organize the data, and not the logistics of the CouchDB itself. We'll have to come up with some standards on the document format to enable the selection, aggregation, etc. that we're going to need in this project. Not a bad start, and it's looking pretty good right now.

I just need Jeff to get the server spun up on one of our EC2 boxes.

This morning I thought I'd spend a few minutes getting my fixes to the existing WordPress plugin - CodeHighlighter, up and into the WordPress site so that I could easily update it, etc. After all, there might be several folks that are looking for something like I wanted, and not finding it in the existing tools. I downloaded the version I'd hacked up on my site, and then placed it into a git repo, added a README.md, and then pushed it up to a new GitHub repo. I was then hoping to simply publish it on the WordPress site, and be done with it.

Silly me… Why should it be that easy?

Turns out, the WordPress plugin site is an SVN repo where you have to give them your code and then they give you access to an SVN repo (Sourceforge?) where you can put your code. A few years ago, I wouldn't have minded, but now… SVN… really?!? Nah… I think I can do just fine without that cruft.

I can simply use GitHub and clone the repo in any WordPress install that I have. There's no need to have anything fancier than that. In any event, I'm guessing that in a little while, the WordPress team will switch to GitHub anyway as the number of SVN users are going to dwindle like Perforce, Visual SourceSafe, PVCS, etc. all have. There's just no way to keep the project looking up to date with SVN.

So it's there, and it's easy to use, you just have to be a little smarter than the average WordPress blogger, but that's OK. I am, and that's all that really matters.

UPDATE: by simply getting into the wp-content/plugins/ directory and doing:

  $ git clone git@github.com:drbobbeaty/CodeHighlighterPlus.git

and then using the WordPress Plugins page, I can disable the old version, enable this new clone, and then delete the old one. After this, everything is OK, and it's all controlled by the GitHub repo.

To be true, this isn't going to auto-update from he WordPress Plugins page, but I didn't have to mess with the SVN repo either - and that's a win for me.

The decision was made late last week that we really should try to use some document database - like CouchDB for saving our run state for metrics extraction and historical analysis. We had initially planned on using MySQL, as it's the dominant database in The Shop, but that was going to require that we flatten the ruby objects to some set of key/value pairs, or come up with some ORM that would properly store and load things. Neither was really super attractive, and so we had Jeff in the group take a look at CouchDB. I knew MongoDB wasn't the answer, because I'd used it previously, but there were a few nice sounding things CouchDB had that could really tip the scales in it's favor.

Most notably were the views. These are basically the same things that you'd expect in a SQL database, but in the document sense they can be arbitrary map/reduce schemes implemented in javascript, and stored on the CouchDB. This means that we can make some interesting views for the metrics that gather data across different documents and make it presentable in a very simple and efficient way.

I'm thinking that generating the majority of the metrics are possible in this way, and then the stuffing these values into a visualization system shouldn't be too bad. We'll have to see as I'm still in the very early stages of using this, but it certainly has some interesting potential for what we're trying to do.

Today I'm going to be spinning up a MySQL instance in Amazon's EC2 for holding the metrics of our application. The goal is to have historical records for detecting trends and A/B testing to see the effects of the changes. It's all about "save it - report it", and I'm looking forward to using MySQL in this case. I've not done a ton with MySQL, and it's a huge deal here at The Shop, so it'll be nice to see it all done right, with good tools, and monitoring as well as good backups.

I've found an interesting OS X client for MySQL - and I'll be digging into it as soon as I get things set up. Or maybe I'll just stick t the command line as I like that on PostgreSQL best. Who knows… it's exciting to be moving in this direction this morning. I just hope I can make some real progress today.

This morning I noticed that Google Chrome dev 22.0.1229.6 was out, but without any release notes. None. Zip. I did notice that the "Stable" channel of Google Chrome is on 21.x, so it's possible that the 'dev' channel is about to just to 23.x, and this is a last little patch before the big jump, but that's just a guess.

Something changed, and it's nothing I can visually see, so I'm guessing it's not a major deal, but it's interesting that they've gone from "sparse" to "nonexistent" release notes. Maybe they're cutting back at Google? Can't afford the digital ink? Crazy.