What's it all about, Alfie?

For the second morning in a row, I had issues with production. Thankfully, this bug didn't effect any of the production data - only that data that was supposed to be written to Couch. So in a sense, it was bad, but not worthy of a re-run. The bug was pretty simple, and for once, I'm glad it wasn't me. The original code was:

  def to_augmented_hash
    to_hash.delete('accounts').merge({
      :category_counts        => category_counts,
      :number_of_new_accounts => number_of_new_accounts
    })
  end

where we were getting a NilClass error on the merge call. I had remembered reading that the delete() method would return a nil under some cases, so I looked it up again, and sure enough - it only returns the value of the key removed. What needed to happen was:

  def to_augmented_hash
    to_hash.delete_if { |k,v| k == 'accounts' }.merge({
      :category_counts        => category_counts,
      :number_of_new_accounts => number_of_new_accounts
    })
  end

With this in, it all worked just fine. When I went to check it in, I saw that a co-worker also fixed this, but split it out on several lines, and mutated the value with the delete(). I decided to leave mine in as it was cleaner, and more to the way it would have been re-written by someone else anyway.

Today I've spent a good deal of time reading up on clojure and getting up to speed with my co-worker that's got several years of clojure experience on me. I knew going in it was going to be like this, and the learning curve is about what I expected. It's Java, under the hood, and that's got all kinds of pros and cons with it, but there are decent build tools, and I'm able to spend a lot of time getting the build/deployment environment up.

Today I also got the init.d scripts going so that we can start and stop it from a Makefile. Basically, all the things we can do from the rake files of the sister project in Ruby.

There were also issues with building PoestgreSQL 9.2.2, but those weren't hard to solve. Lots of nice progress today.

This morning we had a lot of warnings in the summaries with regards to the Merchant Status Reasons (MSRs). Salesforce exists on string fields, and while they make drop-downs that limit the applicable values, it's still a mess if something goes amiss, and we have unusual string values in fields. Such is the case this morning.

It turns out that our Salesforce team added several new merchant status reasons to the drop-down, and so now we were unable to know where in the sales cycle these merchants were. This is not good because it means that the only thing we can assume is that they are at the beginning, and that's clearly not right for all of them.

The solution was to learn what additions had been made, how they translate into our internal status codes, and put in the mappings so that tomorrow we will not fail. So that's what I had to do this morning.

This afternoon the Google Chrome team released 25.0.1354.0 to the dev track. Unfortunately, the release notes are more than terse, and of no help to someone at all. I wonder why they even bother?

I needed to find out how to list all the active queries in PostgreSQL, so I did a little googling, and found the following:

  SELECT * FROM pg_stat_activity;

and the response comes back with all the activity you can imagine. I'm pretty sure it's by the connection, as a lot of my connections are 'idle', but that's great! This gives me the knowledge of what's going on in the database in case a query it taking forever.

I know this about myself - I'm not afraid to get dirty. I don't mind it, and as long as I'm in for it a little, I'm in all the way. Mud up to my elbows - no problem. So it's come to no shock to me today that Archibald, one of my co-workers, who's a little fussy, and loves clojure is a bit unsettled by some developments in the project today.

Basically, we need to have something that conditionally formats objects into JSON from clojure so that the data flowing out of the system we're building is easily read by the downstream system without modification. That's the key.

Now it's certainly possible that we can modify the downstream system to make it take the data we'll be sending, but previously in the day Archie didn't want to do that either - for obvious coupling reasons (another big clojure design point) - so we discarded that idea, and went full steam ahead. But when we got to the "dirty" part of the code - where we were going to have to have a very customized output formatter to send only the fields we need to send - based on the data we're going to send.

For example, we can have two different location data points. One that's based on a zip code:

  {
    "name": "Downtown",
    "zip": "47664"
  }

and one that's based on a division:

  {
    "name": "Downtown",
    "division": "indianapolis"
  }

The idea is that a zip code is very targeted - geographically, but there are certain demands that are much larger in scope, and for those, we want to make it clear that the entire area is "fair game". The problem is that the table this data is coming from has both columns:

  CREATE TABLE locations (
    id          uuid PRIMARY KEY,
    demand_id   uuid,
    latitude    DOUBLE PRECISION,
    longitude   DOUBLE PRECISION,
    name        VARCHAR,
    zip         VARCHAR,
    division    VARCHAR
  )

so if we sent data directly from the table, we get:

  {
    "latitude": null,
    "longitude": null,
    "name": "Downtown",
    "zip": null,
    "division": "indianapolis"
  }

and if the reader of this data looks at the existence of the keys for what to do, it's going to be confused. We need to have something that intelligently outputs the data so that null fields are only sent when they are required null fields.

This is getting "messy", and I could see it on his face. It was something he was finding quite distasteful. It wasn't clean, like math. And I could tell it was grating on him.

For me, it's just part of the job. Very reasonable, and something we should be doing for our clients. We need to send them what they need, everything that they need, and nothing they don't need. I'd like that if I was a client of the service, so why shouldn't I want to do that for the clients of my service? Seems only reasonable.

I can tell this is going to be one of those projects that I'm going to wish was over before it ever really got started. Archie is a nice guy. He's funny and personable, and smart. But all too often he decides what he is willing to do, and many times that's not what really needs to be done because it's "messy". Please just do the job, or move on… there really is very little room in life for people that only do what they want.

This morning I had a production problem that I'm sad to say I probably should have seen coming. I added some new data from Teradata into the system and it was a bunch of metrics for a bunch of deals, all aggregated up to the Merchant level, and thrown into a JSON file for parsing. The original code didn't allow for there not being data for a division, and it could have easily been solved with something like this:

  @division_cache = source_data[division] || {}

but I forgot the "|| {}", and so it was possible to return a nil. This caused a nil pointer problem, and that hurt.

The solution was simple, and thankfully, I had time to re-run everything, but it was something that again I've strayed from - good, solid, defensive coding. I miss it.

I wish I had more control over this project and could enforce this without the knowledge that it'll get ripped out in a few days by someone looking to "improve" the code.

I know that there's little to nothing I can do about it, but it's a little frightening that as I start to dig more and more into the "World of Clojure" the tools that I didn't like from my previous jobs with Java are rearing their ugly heads: JBoss, Hibernate, etc. Now things may have changed dramatically since I was writing Java code back at a previous job, but I'm guessing it's only changed marginally since then. The problems I had back then were the near mend less use of any package from the net that had anything to do with Java. Certainly if it came from the Apache camp.

Interestingly, it's a lot like the gems I'm seeing in the ruby world - developers do a quick 60-sec google search, see that a gem is written that sounds right, get it and assume it's bug free. Only when it doesn't work do they start to see that often times, Free Software isn't free.

So I'm trying very hard to keep my mouth shut about all this as I know the guys I'm working with on this don't have my experience with these tools, and in general, don't really even think about production issues at all. In fact, when I mentioned logging in clojure to the "clojure guru" here in the Shop, he said "I'm not really sure about any of that"

I just shook my head.

This is the kind of idealistic attitude I see every day. Most of these guys think it's enough to get the code working. Most times. So what's the big deal if you have to hack it up a bit now and again? No biggie, right?

I've seen this time and again in academics, and it's OK there, but if you want to make something that's going to run in the real world, you have to pay attention to all the details. Not just go with the brightest, shiniest, object in your field of vision.

Just keep your mouth shut, Bob. It's not going to change anything, anyway.

I got hit with a nasty little bug this morning. In the multi-stage processing of the app I've been working on, if we have a real show-stopped problem with one of the divisions, we don't want to run subsequent steps in the processing as we know they will be wrong, and it'll make it that much harder to fix things in the morning when I check on them. So I did something like the following:

  function process_list {
    for div in $remaining_divisions; do
      if [ "`echo $failed_divisions | grep $div`" = "" ]; then
        # do the processing
      fi
    done
  }

but this gets us into problems when we have a single failed division in the first step of the processing:

  failed_divisions=" charleston-wv"

and we have another division named charleston.

The grep isn't catching the distinction, and so the failure of charleston-wv is causing charleston to not complete it's processing. Crud.

The solution was to include delimiters in the name, so that a list of failed divisions is really constructed like:

  for div in $remaining_divisions; do
    failed_divisions="$failed_divisions |$div|"
  done

and then the grep can be changed into the more robust:

  function process_list {
    for div in $remaining_divisions; do
      if [ "`echo $failed_divisions | grep \|$div\|`" = "" ]; then
        # do the processing
      fi
    done
  }

This should keep things honest and minimize the problems of these division name collisions.

This morning I noticed that Google Chrome dev is now up to 25.0.1349.2 with another nearly useless set of release notes. While I like that they provide a link to the SVN log, that's not really release notes by any stretch of the imagination. I write very nice commit messages, but even I know those aren't anything like release notes. They're far too detailed. I wish they'd write release notes and not just reference the SVN logs.