What's it all about, Alfie?

Jeff, the guy recommending CouchDB as a document database for our app, suggested that I look at CouchRest as a nice ruby client for CouchDB. And the docs look impressive - so far as they go. It's pretty easy to open up and use a database:

  require 'couchrest'
 
  @db = CouchRest.database('http://localhost:5984/megafun')

and then saving a document is pretty easy as well:

  @db.save_doc({ one: 1, two: 2 })

even doing a bulk store of multiple documents is easy:

  @db.bulk_save([{ one: 1, two: 2 },
                 { one: 1, three: 3 },
                 { one: 1, four: 4 }])

But the main docs don't really say anything about using a proxy, and in The Shop, with lots of hosts in Amazon's EC2, there's a lot of proxy work that we have to do.

Specifically, to get to Amazon's East datacenter, we have to use a re-directing proxy on our laptops and there was just nothing in the docs about using a proxy, so I had to dig into the code for CouchRest, and thankfully, I've learned a bit of ruby in the last few weeks, and the support was already there!

Because we have servers in EC2 east, I couldn't hard code the proxy usage, but using the pattern we have used for other proxy-based access, I was able to very quickly set up the config files for the CouchDB databases, and then in the code say:

  class Database
    def self.database
      CouchRest.proxy(AppConfig.database.proxy_uri) if AppConfig.database.use_proxy?
      @db ||= CouchRest.database(AppConfig.database.uri)
    end
  end

and then in the rest of the Database class I could just reference this database method.

The things I'm doing have a lot more to do with how we want to organize the data, and not the logistics of the CouchDB itself. We'll have to come up with some standards on the document format to enable the selection, aggregation, etc. that we're going to need in this project. Not a bad start, and it's looking pretty good right now.

I just need Jeff to get the server spun up on one of our EC2 boxes.

This morning I thought I'd spend a few minutes getting my fixes to the existing WordPress plugin - CodeHighlighter, up and into the WordPress site so that I could easily update it, etc. After all, there might be several folks that are looking for something like I wanted, and not finding it in the existing tools. I downloaded the version I'd hacked up on my site, and then placed it into a git repo, added a README.md, and then pushed it up to a new GitHub repo. I was then hoping to simply publish it on the WordPress site, and be done with it.

Silly me… Why should it be that easy?

Turns out, the WordPress plugin site is an SVN repo where you have to give them your code and then they give you access to an SVN repo (Sourceforge?) where you can put your code. A few years ago, I wouldn't have minded, but now… SVN… really?!? Nah… I think I can do just fine without that cruft.

I can simply use GitHub and clone the repo in any WordPress install that I have. There's no need to have anything fancier than that. In any event, I'm guessing that in a little while, the WordPress team will switch to GitHub anyway as the number of SVN users are going to dwindle like Perforce, Visual SourceSafe, PVCS, etc. all have. There's just no way to keep the project looking up to date with SVN.

So it's there, and it's easy to use, you just have to be a little smarter than the average WordPress blogger, but that's OK. I am, and that's all that really matters.

UPDATE: by simply getting into the wp-content/plugins/ directory and doing:

  $ git clone git@github.com:drbobbeaty/CodeHighlighterPlus.git

and then using the WordPress Plugins page, I can disable the old version, enable this new clone, and then delete the old one. After this, everything is OK, and it's all controlled by the GitHub repo.

To be true, this isn't going to auto-update from he WordPress Plugins page, but I didn't have to mess with the SVN repo either - and that's a win for me.

The decision was made late last week that we really should try to use some document database - like CouchDB for saving our run state for metrics extraction and historical analysis. We had initially planned on using MySQL, as it's the dominant database in The Shop, but that was going to require that we flatten the ruby objects to some set of key/value pairs, or come up with some ORM that would properly store and load things. Neither was really super attractive, and so we had Jeff in the group take a look at CouchDB. I knew MongoDB wasn't the answer, because I'd used it previously, but there were a few nice sounding things CouchDB had that could really tip the scales in it's favor.

Most notably were the views. These are basically the same things that you'd expect in a SQL database, but in the document sense they can be arbitrary map/reduce schemes implemented in javascript, and stored on the CouchDB. This means that we can make some interesting views for the metrics that gather data across different documents and make it presentable in a very simple and efficient way.

I'm thinking that generating the majority of the metrics are possible in this way, and then the stuffing these values into a visualization system shouldn't be too bad. We'll have to see as I'm still in the very early stages of using this, but it certainly has some interesting potential for what we're trying to do.

I don't know… maybe positive thinking really does work. This week I had a run-in with someone that was amazingly uninterested in being flexible. The next day, they changed the API on their system. I was starting to believe that this was just going to be status quo when dealing with them, but then today, I received an email and it amazed me.

Their leader totally reversed his position, and was asking me how I wanted the data. This was a real shock as I had never expected it, and was settling myself in for a series of constant changes to the API and fixes to the ETL to keep things working. Very nice to see.

I spent about 5 min thinking about it, and decided that what we originally had was a good plan, and we just needed to keep going. The original plan was to have an array of maps (this is all JSON) where each map represented a possible match and the array was a logical OR. This allows them to change the nature of the individual maps and the logical OR can include a region and a zip code… or a series of zip codes… or regions… all this was vey well thought-out for the geo-tagging.

I wanted that back, and then asked that for the taxonomy of the demand, we do something very similar - an array of maps where each is a tuple of the classification of the demand based on the default taxonomy. This will also be logically OR-ed to get the possible classifications that this demand can fulfill.

In short, I think it's clean and clear, and makes a lot of sense. I hope they accept it.

In any case, I'm shocked that I might have had an effect on the change of heart, but who knows? Maybe I'll step it up with the people on the train at night and see if they get a little nicer!

Today I'm going to be spinning up a MySQL instance in Amazon's EC2 for holding the metrics of our application. The goal is to have historical records for detecting trends and A/B testing to see the effects of the changes. It's all about "save it - report it", and I'm looking forward to using MySQL in this case. I've not done a ton with MySQL, and it's a huge deal here at The Shop, so it'll be nice to see it all done right, with good tools, and monitoring as well as good backups.

I've found an interesting OS X client for MySQL - and I'll be digging into it as soon as I get things set up. Or maybe I'll just stick t the command line as I like that on PostgreSQL best. Who knows… it's exciting to be moving in this direction this morning. I just hope I can make some real progress today.

Today, while working on a bunch of other stuff, I got a chat from the data scientist in Palo Alto asking me why the service tag changed to primary_service, and what else changed. Turns out that the folks I'm getting the data from has decided to change the data format I was getting from them, without telling me about it. Nice.

The problem wasn't that it only took me an hour or so to change the ETL on the data to get it back to where I needed it, it was that they decided to change it in ways that weren't required without telling us anything about it! I know this is often the case, but we're supposed to be on the same team, and they just didn't bother telling us.

I know the point of the ETL code is to fix that up, and allow us to insulate ourselves from changes on their part, but it's also the timing. These guys could not possibly have been in my position yesterday or they'd never have done it. It's one of those cardinal rules of making an API - once you have it, you stick to it! If you have to change it, it's because it doesn't work, and then you open a discussion - even a simple "heads-up" email, to make sure that people are aware of the change that's coming, and why you had to make it.

It's not horrible, and I liked that I got new data that made the results better, but it was the way they did it that was annoying. I fear that this isn't the last such occurrence from these guys.

I just got out of a meeting, and I have to say that I'm really quite shocked to see someone at The Shop so incredibly inflexible as Jim. This is the first time I've met Jim, and while I haven't been impressed by his emails, and specifically his responsiveness to my requests, I always wrote it off as him being exceptionally busy. Busy people have a lot going on - I get it. But today was the first face-to-face meeting, and during that, Jim really blew my socks off.

Jim is vending some demand data, and it makes perfect sense to everyone but Jim to have him include the current inventory in his service. After all, if there's some change in inventory, it's going to really effect what we need to get because excess inventory will shrink demand, and a big run on something will push it up. But not in Jim's mind.

No, Jim was making a theoretical demand calculation, and in that, I can see a value. But he's also including the inventory on hand at the time of his run - which is once a week, so it's going to make it harder for us to know what current inventory-effecting events to include in the demand we get from Jim. We have to look at the time of his run, and then look at all events to see how they might effect the demand.

It'd be far simpler to have Jim vend the raw demand and then we can always correct it by the current inventory. Much simpler. No problems in potentially lost transactions. Better.

But not to Jim. Holy Cow! It's been a long time since I've seen a professional programmer push back so hard on a feature. It was speculated that it was all about the level-of-service issue… after all, hitting Jim's service once a week and saving the results is far easier on Jim than having to make sure it's up all the time and vending the right data. I just can't get over the sheer laziness of this guy.

He's a developer. What's he got to do but develop? How hard is it to make it work 24x7? Not too hard, I've done it in a bunch of technologies over the years. Stop being so bloody lazy, Jim!

Holy Cow!

This morning I noticed that Google Chrome dev 22.0.1229.6 was out, but without any release notes. None. Zip. I did notice that the "Stable" channel of Google Chrome is on 21.x, so it's possible that the 'dev' channel is about to just to 23.x, and this is a last little patch before the big jump, but that's just a guess.

Something changed, and it's nothing I can visually see, so I'm guessing it's not a major deal, but it's interesting that they've gone from "sparse" to "nonexistent" release notes. Maybe they're cutting back at Google? Can't afford the digital ink? Crazy.

Today we've had a lot of really great work done. Things are starting to accelerate, in part because we're down a few guys due to a clojure-rewrite of the main app, but also because we're focused on making real and significant progress on the project. It's been all-(remaining)-hands on deck, and it's been really nice.

I've done a lot of email to folks about things we need their assistance with. In some cases it's just knowledge transfer, in others it's liable to be a little more complex. Still, we need to get all these issues figured out before we release this to the users, and since someone has to do it, it might as well be me.

At the same time I've made some nice progress on closing some stories for the calculations done and logged in the app. Nothing amazing, but due to the optimistic nature of 90% of the ruby code I've seen, it really means tracking down the potential errors and making the code more robust as well as implementing the actual features. It's nice to see things really start to take the shape of a completed app - especially when there are so many people working on it, as it more often resembles swiss cheese than a working app.

I still would like to get some answers from a few folks, but that's what I'm waiting on now - external dependencies. I've never been very patient, but I'm going to try hard to last this out with a nice, and gentle attitude. Personal growth time.

I've been talking to a few friends about editors on Mac OS X, and one editor that's getting a lot of press lately is Sublime Text 2. It's supposed to be a variant of Vim, and that's always a plus - it's even got a vintage mode to make it mimic the command and insert modes of Vim. It's got an amazing speed profile as well as an incredible Python-based extension library. And there are a ton of things that have been written for it.

So this morning I decided to give it a try. I'm glad I did. It's an amazing platform for coding. I can "open" a project - basically a directory that is my git repo, and then with the "Find Anything" command, I can load up a file very simply. Amazingly so. This is my major concern about BBEdit - opening files in a ruby environment is really a pain - as there are a lot of them.

There are so many nice things that it's just too hard to know where to start. There's a GitHub package for dealing with Gists - one of my favorite places to store snippets of code. This guy even gets the file names so you don't have to know the Gist number beforehand.

There is a legacy Vim mode, and a legacy Ex mode. Both are really nice for old school Vim users like me. Also, the app works on Windows, Linux, and OS X.

I'm looking forward to hammering on this and seeing where it takes me. Very excited.