What's it all about, Alfie?

This morning it looks like we were able to run all of North America through the UAT system with a co-worker's changes to Salesforce, and that's a huge step forward - but it wouldn't have happened if it had been up to me. I was thinking it was too big a jump to try and take - to go from 27 divisions to 170+ in one night. I would have done it in a couple of nights. But it worked. In spite of me.

Good experience for me - to be vocal and wrong. I've already apologized to the guy running the test and he laughed… good for me. I'll mention it again in stand-up, and say I was wrong. I want to do it as it drives home the idea that I don't know everything.

But it's a big step. About 5 hours and all of North America. We should be able to get it down from there, but even if we can't, it's workable, and that's a huge win.

Recently, in order to get the kind of performance we needed from CouchDB, we had to drop all sense of updating the existing documents in Couch with the data from subsequent runs. This "insert only" mode turned out to be vastly superior to the update scheme even after we made it server-side updating and sending just what we needed for the update. It was just too slow. So now we are going to have four documents when we had previously had one.

Things had to change.

The biggest concern was the metrics web page and widgets that showed a lot of the different results of the runs - all hitting Couch for their data. In the previous version, we had the one document to look at for all the data, but now we had to be careful about what we were looking at, and gathering up for display.

Thankfully, the views in Couch could be adjusted to make very few code changes, and where there were code changes, we didn't have to change the views in Couch - so it was pretty easy to get things figured out. Not bad, really.

At least now, come Monday, we'll have good data in the metrics app, and that's very important.

I had long suspected that we needed to have tools for correcting problems associated with the reassignment and prioritization phases of the process, and today I finally decided to just make some. There are several interesting pieces in this story, but let's talk about the actual need and how I worked that into the process first.

It's not surprising that again today we had a slight issue with the prioritization of a single sales rep - not due to the code, but due to the incoming data. It would have been really nice to have simply re-run that one sales rep through the prioritizer, and fix them up. But we didn't have any tools to do that. So we had to say "Sorry, it'll be fixed tomorrow".

So after stand-up, I decided that we needed to have tools to:

• Re-prioritize a single sales rep - this will pull all the accounts (merchants) for a single sales rep and then cleanly rank them for their daily call list. This is basically what we do nightly, but we start out getting all the sales reps in a division.
• Clear all the priorities on a single sales rep - this is something that I think is going to become more important as things slip throughout the cracks and we need to clear out account call list priorities en masse. This will simply pull in all the accounts for a single sales rep and then clear out their call list priorities.
• Clear all the priorities on a single sales rep within a division - this is like the last, but in the case of the House Account, which is the same for all divisions, we might want to confine the clearing to a single division for safety sake.

With these three tools, we should be able to do all the quick fixes that have come up since we started showing this to sales reps and their city managers. Thankfully, the code for all this is pretty simple - even battle-tested. If we look at the existing code that gets all the sales reps for a division and then prioritizes them one by one, we can simply make that inner block the prioritize_rep() method, and move the code such that it's simple to call either method - the division-level one, or the per-sales rep one, and get what we need.

Finally, it's simple to copy that method and create clear_rep() where we don't prioritize the accounts for a rep, but simply get them and clear out the requisite fields. It's not bad at all. Pretty simple, really. But that's where the fun ends.

In order to do this, I had to change a lot of specs and other code simply because it wasn't meant to be flexible. This is what I hate most about these unit tests. They really aren't written to be as reusable and flexible as the code they are testing, but they need to be. I spent probably 30 min changing the code, and about another hour fixing the tests. That's messed up.

Buy the real story of the day is when I was talking about doing this, some of the other guys in the group didn't necessarily want to help do it, but they certainly wanted to make sure that their $0.02 was listen to, and done. It's like the unwritten evil side of Agile, or maybe it's just the Code Monkeys, but it's perfectly natural to have a design discussion - even if it's completely one-sided. It's considered "helpful", and "nice". But really, it's about wanting to control the decision without having to burden the responsibility of it being right.

I can work in Agile, and I see some of the benefits, but I think it's like any other evangelical movement - the downsides are completely dismissed by the "faithful" as fringe, and extreme - and certainly not representative of what they do. But it is.

I really long for being on a project where I don't have Code Monkeys. I like the people, just not how they act a lot of the time.

This morning I was able to finish up the re-write of the summary script and I was very pleased with the results: the processing of the pipeline log dropped from 4+ min to less than 5 sec - even with jruby, and the other two are in the sub-2 sec range. The latter are really dominated by the jruby startup time, and if we can move to an RMI ruby in deployment, that will help here too.

In short - fantastic success! Now I need to come up with a better queueing and processing scheme in bash - or re-write that in ruby as well…

This morning I ran into a problem with the ruby re-write of the summary script that I've been working on since late yesterday. The error was occurring on the relatively simple code:

  File.open(src) do |line|
    if line =~ / BEGIN /
    # …
    end
  end

right in the open() method call. The error was cryptic:

  summary:48 in 'block in process_pipeline' invalid byte sequence in UTF-8 (ArgumentError)
      from summary:47" in 'each'

I had to hit google, as it was clear to me there were odd characters in the file, and while I might like to fix that - the key to the previous version was to include the '-a' option to grep to make sure that it looked at the files as binary files. But what would do the trick here?

Turns out there's a StackOverflow answer for that:

  File.open(src, 'r:iso-8859-1') do |line|
    if line =~ / BEGIN /
    # …
    end
  end

which instructs the IO object to read the file with the ISO-8859-1 encoding and that did the trick. No other changes were necessary!

Sweet trick to know.

It's almost the leet version number, but not quite. This morning Google Chrome dev 25.0.1337.0 was released with virtually nonexistent release notes. I guess the same maintainer is back at it. Just gotta wonder what they are thinking. I can write a few paragraphs a day on what I'm doing and they can even be bothered to make decent release notes?

Googlers. Go figure.

With all the efficiency changes in the code recently, the next most inefficient was the bash script that analyzed the run logs and generated some summary statistics for us to view in the morning. When I first created this script, it wasn't all that complex, and the logs weren't nearly as big as they are now. I used the typical assortment of scripting tools: grep, sed, awk, but the problem was that as I added things to the summary script the time it took to execute was getting longer and longer. To the point that it took several minutes to run it on the main pipeline process. That's no good.

So I wanted to rewrite it in something that was going to be fast, but process the file only once. The problem with the current version isn't that it's using bash, or grep, it's that the files are hundreds of megabytes and we need to scan them a dozen or more times for the data. What we needed was to make a single-pass summary script, and that's not happening with bash and grep.

So what then?

Ruby popped to mind, but given that we're using jruby, there's a significant startup penalty. But maybe we can force it to use a compiled MRI ruby in the deployment environments, and that will speed up the loading.

C, C++ both seemed like ideal candidates, but then I know how the rest of the guys in the group would react, and it's just not worth it.

So Ruby is is.

This shouldn't take long, as most of this is pretty simple stuff for ruby. Let's get going...

Finally, moving day has arrived! This morning I've been getting things moved over to the new servers in our own datacenter, and this should provide a very needed boost to the performance of the application. This includes a CouchDB server with 24 cores and 96GB of RAM with a 1.6TB disk array, as well as a nice app server with 24 cores and 96 GB of RAM. There's a mirrored Couch pair for production, and a similar app server there.

It's been a lot of little things, lots of little code changes and pushes. Even some reconfiguring of aliases in the firewalls, but that's where I'm starting to hit a snag. I used to be able to do this, now it's meant to be handled by the production operations group. That's not too bad, but they won't push anything until 4:00 pm today, and if it's not 100% right, then we're going to have a hard time getting it right for tomorrow.

I'm hoping to get a few more tests done today, but I doubt that I'll be able to simply because a co-worker is busy using UAT to test things there. It's a shared environment, and there's no way to run both tests at once, so since he was first, I have to wait.

I'm not the most patient of people.

This afternoon I've been working with a co-worker to get all the new hardware up and going in our own datacenter so that we can move our application from Amazon's EC2 to our own, more reliable, machines. It's a bit of a hassle in that there are now 22 new machines to rebuild, and the folks doing it aren't paying really close attention to the machine names and set-ups, so there have been a lot of re-dos, but it's getting there.

We should be able to get all the critical machines up and going before I have to leave today, and then I can get started on moving the apps in the morning.

Exciting times to be getting out of EC2, and onto far far better hardware. I'm just hoping that it's going to clear up the issues we've been having with Couch. Now that would be really nice!

Today has been a lot of little things to try and get the application's performance good enough so that it can still run in EC2 for the few days that it has left in that datacenter. I'm trying to put in simple, clean fixes to minimize the time spent in an overall run so that we can get more divisions out in the same period of time.

This brings up the point that's been bugging me for a few days, and that's expectations. I'm really getting tired of making extraordinary effort for some management folks that really don't seem to recognize the nature of the effort, or appreciate what it is that I'm really doing.

It's nothing I haven't seen before, but it's always a little sad the first time you see it at a new job. That realization that this guy is no batter than that other guy at the previous place, and they are going to push and make artificial deadlines and then pre tent to "tell Dad" if you don't meet them.

Working last week on Wednesday, Thursday, and Friday to make a deadline that I didn't think was possible, just to make it possible for this guy to tell his superiors that his team "did it" was something I was willing to do - as long as it was appreciated. But it wasn't. So now this guy has marginalized himself. I won't break my back to get him out of his own jam any more.

But hey… what am I doing now, then? I'm trying to make this work as opposed to just letting it fail.

I'm a chump.