What's it all about, Alfie?

I was talking to a couple of guys in The Group today and I heard a few guys talk about extracting the logging and timing metrics from the code itself, and have them be simply meta-programmed into Ruby such that all methods of a certain class would be logged and timed. Now I'm all for simplifications - to a point, but this is really, in my mind, going way too far. There's something about minimalism that I think is attractive to the Math types in a group, and we have them, but that's not at all realistic, as code needs to function in the real world, which means that it has to check inputs, log lots of intermediate state, and in general do all those things that rookie coders don't do and it gets them into trouble when their code doesn't perform well in production.

Simple is one thing. Hidden is another.

Don't hide the complexity of logging. It's in your face, and it's meant to be. There's no way some meta-programmed log system is going to know where I want to put every one of the log messages I want. Timings is a little easier, but it's still a mistaken assumption that simply timing method calls is sufficient as I'll never need to sub-divide a method call.

Fiddlesticks.

You need to have logging and timings sprinkled in your code. It's not homework coding, but I'm coming to believe that there's a lot to be said for the C++/Java world that I come from. Ye,s it's not Ruby, and there are a lot of things to like in Ruby, but there's a lot that I think these guys take for granted and don't properly write code to guard against.

Make code simple, yes. But if you hide too much, then you'll forget what's really being done, and you're going to get hit in the rear pretty soon. Performance is a huge blind spot for the majority of these guys. They just don't see it, and don't see it as needed. Couldn't be more wrong. It's always important in a production system.

So I'm going to try and guide them away from this decision, as I am a firm believer that it's going to get them, and me, into hot water. We just don't need it.

I learned a valuable lesson about Ruby this morning - if you have a module of shared code in Ruby:

  module SharedStuff
    def log
      puts "log"
    end
  end

then:

• include makes the module's methods available to an instance of the class
• extend makes the methods available to the class itself

This means that by using the same base code, you can add them in as class methods with the extend directive, and as instance methods with the include directive.

Totally 'Ruby', as it's a strange, massive difference that might be missed by new users of the language.

Glad I Learned it.

This morning I finished a refactoring that I'd started on Friday when I realized that the UX of the web pages I was making (to be thrown away) were really all wrong. I was doing the more direct scheme - loading a lot of data into memory and then manipulating it quickly to render different graphs. The problem with this is that the data sets are really very large, and the load times for them are longer than you'd want to sit for.

Once of the guys on the team was saying "Break it up into a bunch of small requests." And while I could see that approach, I thought that the overhead of all the calls was going to really kill performance. I never even really considered it.

But about 4:00 pm on Friday, when I was really very frustrated with the code I was working on for that page, I decided to give it a go. What made most sense to me was to break the requests into a few stages:

• Get the list of all runs - based on the selected database, get the division/timestamp pairs for all runs in that database. We'll be able to parse them next, but this is a nice, short little method.
• Parse the executions for divisions and timestamps - take the division/timestamp data for all runs in a database and create a data structure where the list of timestamps will be stored in a descending order for a given division.
• Set HTML run options for a given division - when the user selects a division, take the parsed data and create the drop-down for the runtimes for that guy.
• Query for the specific data - taking the data from the options - the database, the division, the timestamp, hit CouchDB for the exact data we need to visualize. In many cases this is less than 100 documents.
• Parse the documents - once we have the targeted data from CouchDB, parse it into Google DataTable or ZingChart series.
• Render the data - last step.

I was surprised to see that the resulting code was smaller than I'd had. The parsing of the data structures was really a lot more than I thought. Starting at the top of the list, the code to get the list of all runs is really simply:

  function reload_executions() {
    // hit CouchDB for the view of all executions it knowns about
    var svr_opt = document.getElementById('server_opt');
    var url = svr_opt.value + '/_design/general/_view/executions?' +
              opts + '&callback=?';
    $.getJSON(url, function(data) {
      parse_execution_tags(data);
    });
  }

Once again, jQuery really helps me out. Next, I need to parse this data into a structure of all the runs by division:

  function parse_execution_tags(data) {
    divisions = new Array();
    runtimes = new Object();
 
    for(var i in data.rows) {
      // get the execution_tag and exclude the very early ones
      var exec_tag = data.rows[i].key;
      if (!/-\D+$/i.test(exec_tag) || (exec_tag.substring(0,10) < weekAgo)) {
        continue;
      }
      // now get the timestamp and division from the execution_tag
      var runtime = exec_tag.replace(/-\D+/g, '');
      var division = exec_tag.replace(/^.*\.\d\d\d-/g, '');
      if (typeof(runtimes[division]) == 'undefined') {
        runtimes[division] = new Array();
        divisions.push(division);
      }
      runtimes[division].push(runtime);
    }
 
    // sort the divisions and create the contents of the drop-down
    if (divisions.length > 0) {
      divisions.sort();
      var div_opt = document.getElementById('division_opt');
      div_opt.options.length = 0;
      for (var d in divisions) {
        div_opt.options[div_opt.options.length] =
            new Option(divisions[d], divisions[d]);
      }
    }
 
    // given the default division, load up the run times we just parsed
    set_runs_for_division(divisions[0]);
  }

where I'd created the variable weekAgo to be able to let me know what the "recent" data was:

  // get the date a week ago formatted as YYYY-MM-DD
  var when = new Date();
  when.setDate(when.getDate() - 7);
  var weekAgo = when.getFullYear()+'-'
                +('0'+(when.getMonth()+1)).substr(-2,2)+'-'
                +('0'+when.getDate()).substr(-2,2);

Once the data is all parsed into the structures we can then build up the drop down for the runs for a selected division with the function:

  function set_runs_for_division(division) {
    division = (typeof(division) !== 'undefined' ? division :
                document.getElementById('division_opt').value);
    runtimes[division].sort();
    runtimes[division].reverse();
    var run_opt = document.getElementById('run_opt');
    run_opt.options.length = 0;
    for (var i in runtimes[division]) {
      var tag = runtimes[division][i];
      run_opt.options[run_opt.options.length] = new Option(tag, tag);
    }
    // at this point, call back to the the data we need, and then render it
    reload();
  }

Calling to get the actual data is pretty simple:

  function reload() {
    // hit CouchDB for the view we need to process
    var svr_opt = document.getElementById('server_opt');
    var view_opt = document.getElementById('view_opt');
    var run_opt = document.getElementById('run_opt');
    var div_opt = document.getElementById('division_opt');
    var et = run_opt.value + '-' + div_opt.value;
    var url = svr_opt.value + '/' + view_loc + view_opt.value + '?' +
              'startkey=' + JSON.stringify([et,{}]) +
              '&endkey=' + JSON.stringify([et]) + opts + '&callback=?';
    $.getJSON(url, function(data) {
      var tbl = parse_series(data);
      render(tbl);
    });
  }

but parsing it into a Google DataTable is not nearly as simple. The code is complicated by the different requests we need to properly create:

  function parse_series(data) {
    // now put all the data into an object keyed by the execution_tag
    var view_opt = document.getElementById('view_opt');
 
    var table = new google.visualization.DataTable();
    table.addColumn('string', 'Division');
    switch (view_opt.value) {
      case 'merchants_by_existing_merchant':
        table.addColumn('number', 'Deals');
        break;
      case 'merchants_by_research_ranking':
        table.addColumn('number', 'Rank');
        break;
      case 'merchants_by_status':
        table.addColumn('string', 'Status');
        break;
      case 'merchants_by_rep':
        table.addColumn('string', 'Rep SF ID');
        break;
    }
    table.addColumn('string', 'Merchant');
    table.addColumn('number', 'Sales Value');
 
    for(var i in data.rows) {
      var row = data.rows[i];
      var name = (row.value.name.length > 60 ?
                   row.value.name.substring(0,60)+'...' : row.value.name);
      var table_row = new Array();
      table_row.push(row.value.division);
      switch (view_opt.value) {
        case 'merchants_by_existing_merchant':
        case 'merchants_by_research_ranking':
        case 'merchants_by_status':
        case 'merchants_by_rep':
          table_row.push(row.key[1]);
          break;
      }
      table_row.push(name);
      table_row.push(row.value.sales_value);
      table.addRow(table_row);
    }
 
    // now let's apply the formatter to the sales value column
    var fmt = new google.visualization.NumberFormat(sv_format);
    switch (view_opt.value) {
      case 'merchants_by_existing_merchant':
      case 'merchants_by_research_ranking':
      case 'merchants_by_status':
      case 'merchants_by_rep':
        fmt.format(table, 3);
        break;
      default:
        fmt.format(table, 2);
        break;
    }
 
    return table;
  }

but the rendering is very simple:

    function render(tbl) {
      var dest = document.getElementById('table_div');
      var table = new google.visualization.Table(dest);
      table.draw(tbl, table_config);
    }

When I put it all together I was amazed to learn that the hits were exceptionally fast. The page is far more responsive, and in short - I could not possibly have been more wrong. The human lag is sufficient to make the calls invisible, and the sluggishness of the memory load on the old version was horrible. This is a far better solution.

I'm going to remember this for the future.

I did it again today. I pushed myself to get a lot of stuff done today for a big, important demo (again), and along the way a few people interrupted me to ask me to look at a few things. Had I been smarter… had I been wiser… I'd have said "I'm sorry, I'm on this push for the demo, how about I get to it on Monday?" But I didn't.

What I did was to push myself to the point that I was very upset with the things I was working on. Oh, it didn't start out that way. It started out with me thinking that I could easily track down this problem that one of the data science guys pointed out. It was in my code for grouping the merchants where I wanted to be smart and clever. I should have known.

First, I wasn't accounting for the case where two groups of overlapping merchants are built and then a single merchant bridges both groups. I messed up. So I needed to go back and fix a few things. First off, I didn't have a general overlapping service method. So I took the old one and expanded it:

  # this method returns true if ANY service is shared between the two
  # merchants. ANY.
  def self.services_overlap?(ying, yang)
    if ying.is_a?(Array)
      # explode the calls for an array in the first position
      ying.each do |d|
        return true if services_overlap?(d, yang)
      end
      return false
    elsif yang.is_a?(Array)
      # explode the calls for an array in the second position
      yang.each do |d|
        return true if services_overlap?(ying, d)
      end
      return false
    end
    # this is the simple call for a 1:1 check
    !(get_services(ying) & get_services(yang)).empty?
  end

Basically, I just allowed arrays to be passed in, and where necessary, I exploded them to allow the basic logic to be applied to the individual merchants. It's not hard, but at this point, I didn't need to worry about what I was passing in to check for an overlap.

The next thing was to completely redo the group_by_service() method as it was far too complex, and it wasn't even working. I didn't like the fact that it was doing a lot of extra checks, etc. but that seemed to be the Ruby Way. Poo. I changed it into a simple single-pass loop that's far simpler and far faster:

  def self.group_by_service(otcs)
    # start with the array of groups that we'll be returning to the caller.
    groups = []
    otcs.each do |d|
      added = []
 
      # add the OTC to all groups that it has some overlap with
      groups.each_with_index do |g, i|
        if services_overlap?(d, g)
          g << d
          added << i
        end
      end
 
      # if we added it to more than one group, then consolidate those groups
      if added.size > 1
        added[1..-1].each do |i|
          groups[added[0]].concat(groups[i][0..-2])
          groups[i] = nil
        end
        groups.compact!
      end
 
      # if he hadn't been added to anything, make a new group for him
      groups << [d] if added.empty?
    end
 
    groups
  end

The ideas here are a lot clearer - add a new guy in to all the group he'd match, then look for multiple matches. If there are, then simply and cleanly consolidate the groups, and continue. My co-worker in Palo Alto liked this code a lot more as well. I do too. It's ruby, it's just not incomprehensible ruby. And it's right.

But in the midst of this, I'm trying to get more web stuff done for another interruption for the demo, and it's nothing I'm happy with. It's all very lame. I don't like the interface I have to CouchDB, I don't have a lot of time, and it's making me very cranky.

I've run into this before, and it's not the first time I've not been able to say "No", and it's cost me something I didn't want to pay. I know I'm no great graphics designer. I know I can make things that work, but they aren't going to be "Wow!" with anyone but someone looking just at the functionality. That's just not in my DNA. And it is exceptionally frustrating to be in a situation where I'm forced to do this work.

I know what is good. I can appreciate it. But I can't generate it. And to be forced to do it is hard. Because I know they are going to throw it all away. Any half-decent designer will look at what I've done and say "Nice, but let's take out the data collecting and put it in a nice design" - as they should.

So I've got to work harder at keeping my cool.

I've got to say "No" more often.

Or I just won't last.

I was talking with a good friend this morning and I came up with a name for a lot of the ruby devs I've run into - but to be fair, it's not just for a good chunk of the ruby devs I've met - it's for a general class of developers. Let's pretend to be a little more precise about this:

Code Monkey - a developer that is more interested in learning a language and how to solve a few problems in it, than using it to solve real-world problems. This includes, but is not limited to, the clojure devs that have never written a comment, and only solved the zebra/water puzzle, as well as devs that never code defensively, or even think that production is important.

This came up because I'd been battling code that wasn't written at all defensively. It was basically assumed to have been run by a person, with a person to fix any problems as they occur. It's like a glorified Excel spreadsheet - I'm going to hit 'Go', and fix things that come up.

But this doesn't really work for real life, does it? Who wants a system that runs at night that has to be constantly monitored to make sure it doesn't get bad data, etc.

Yet they are the first ones that are onto a new language - like clojure. Saying that the real solution is to use a language that doesn't need all that checking as a functional language simply doesn't require it.

What world are they living in?

How is a language able to do ETL on it's own? Answer: It can't. You still have to do it. But the Code Monkeys are really skipping all that because they start with good data and then the process is clean, and simple.

No kidding? Really? Well, of course it is! The same is true for C++, Java, and any other language you want to pick. Start with clean data, don't worry about exceptions and potential problems, and you're going to be able to write amazingly clean code. But that's not how life really works.

We agreed that there were guys with language knowledge, and skills, but they never really dug in and made it work. It's nice to talk to Code Monkeys, but it's not nice to have to work with them. You're always cleaning up their messes.

Once again, we're gearing up for a big demo tomorrow with some of the users, so today has been full of a lot of things that needed to get done in order to have a successful presentation. I had to make quite a few new views for CouchDB, and then work those into a web page and publish everything up to UAT and production for a run. I'm becoming a big fan of CouchDB and it's views and reductions… those are some very powerful tools for looking at this JSON data in CouchDB. Very nice.

I've also done a little fiddling with the Sublime Text 2 styles, and I'm exceptionally happy how that's all turned out. It's made it much nicer to work with. I can't believe I haven't tried it up to this point, and I can't imagine a better editor for me. I'm going to have to brush up on my Python and write some packages some day.

Finally, I'm just dead tired. Great feeling. Lots of really good work done, and the team is really working together far better than I'd have thought a month ago. This is really quite fun.

Today I was really battling a nasty problem with the CouchRest client for CouchDB. This is a ruby gem, and in general, it's pretty decent, but it really starts to fall down when you need to use a proxy to get to CouchDB, and this guy starts having all kings of problems.

There were timeout issues, so I decided to try and make it work. So I forked the project on GitHub, and started to get to work. THe major point was that the underlying RestClient gem had the ability to set things like the timeout for creating the connection as well as timeouts for reading, etc. It's really very flexible. My goal was to allow the settings to be applied on a per-database basis. Then, for every command, use these as the defaults, but overlay any call-time options as well.

The idea was really nice. I was planning on submitting a pull request for this as it only took me about an hour to do. But when I went to test it, it failed with some 502 Bad Gateway error.

Argh!

More proxy problems!

Then I was talking to one of the guys in the group about this issue and he brought up that I could write to my local CouchDB, and then replicate it to a different database on a different server!

BING!

This is exactly what I'd been looking for. I get fast and efficient writes to my CouchDB, but it gets written up to the shared server as I'm connected to the network. This is great!

The configuration is simple - it's a simple doc in the _replicator database, and I'm in business. This is really quite amazing. First, go to the overview in the CouchDB web page, and select the _replicator database:

then create a new document:

Finally, populate it with these essential key/value pairs:

• source - this is the source for replication - it only goes one-way, so this can be a local database, or a remote one. But it's where the data is coming from
• target - this is the destination for replication - again, local or remote, it makes no difference. Make sure to put in the port and the database name in the URL
• proxy - if needed, put in the URL of the proxy to get to either of these databases
• continuous - set to true if you want it to always be replicating

Save this document and then look at the CouchDB status page to see it replicate. It's like magic! OK, not really, but the handling of the proxy is so far superior to the way that CouchRest was dealing with it it's not even funny. This just works!

I'm more and more convinced that CouchDB is an amazing product.

This morning I decided that it'd be nice to have a complete list of all the merchants that didn't successfully complete their processing. Since we are now processing everything - a recent change to the code to make sure that we know exactly every single merchant is getting completely processed, we can now look at each merchant and make sure that they got through the critical processing phases. If it didn't "pick up" the right data, then we can assume that it didn't get to that point. The point of this is that we can then be sure that every merchant completed processing.

I log this, and write them to CouchDB, so we can keep a complete record of all the issues, and then updated the summary script to list the number of incompletely processed merchants so we can watch them over time.

Nice. This is starting to really get close to verification that all was done as it was supposed to have been done.

Today has been a big day of a lot of little things. I've got five post-it notes on my desk - each filled with little things that need to be updated in the code. There are new rules for how to handle new merchants, more rules about aggregations, different rules about when to ignore merchants… all needed to be in the code as soon as possible, and because no one thing was that horrible to do, it was possible to get all the changes in today.

Yet, while trying to get things done, taking more requests is a little frustrating. No… it's quite frustrating. But I was able to get through it all without getting really upset, which is a nice win for me.

In all, it has been a really nice day - there are a few more things I need to do, but it was a good day.

This afternoon I've written some of the most compact, potentially confusing code I've written in many, many years -- and it's perfect code by a Ruby developer. This is something that may be specific to The Shop, but given that they are such a big Ruby shop, I'm guessing that this is the Ruby Way, and like a lot of the functional code I've seen - completely undocumented. Now that's not to say my code is undocumented, in fact, it's got almost a 1:1 ration of comments to code because of it's compactness, but I've come to realize there's a bit of a blind spot in a solid group of young ruby coders that looks a lot like what I call Homework Problems.

In any case, the code I wrote today was specified by the quantitative analyst in Palo Alto as this:

Group the merchants by the services they offer so that any one merchant in the group shares at least one service with at least one other merchant.

Logically, this means that we can have a series of seemingly unrelated services so long as the group has these pair-wise matchings with at least one service.

If we look at the group of Macy's, the Gap, and a Movie Theatre:

You'd think there's no way the movie theater fits in the same "group" as the Gap, but because Macy's sells Jeans, and so goes the Gap, and because a movie theatre sells candy, and so does Macy's, then the Gap and a Movie Theatre "belong together" in a group.

I'm not making up these rules, I'm just trying to code them up.

Once we get these groups of merchants, we'll then process them and get some data from them. That's not the interesting part. The interesting part is the grouping, and how to get it.

My first idea was to write a few little methods that I knew I was going to need: one to get the services from a merchant into an array, and another to see if there are any overlap (set intersection) between two merchants:

  # this method returns a clean, unique set of services for the provided OTC.
  def self.get_services(otc)
    (otc['taxonomy'] || []).map { |i| i['service'] }.compact.uniq
  end
 
  # this method returns true if ANY service is shared between the two OTCs. ANY.
  def self.services_overlap?(otc_a, otc_b)
    !(get_services(otc_a) & get_services(otc_b)).empty?
  end

At this point, I knew that these were very "ruby-esque" methods - one line each, so it's got to be "minimal", right? At the same time, I was able to then start to deal with the idea of just finding the right pairs to feed to the second method, and then collecting them into the right groups.

But therein was a real problem. If I just looked at the merchants serially, then the order matters. Imagine the order: Gap, Movies, Macy's. In this case, the Movies would not match the Gap, so there'd be two groups, and then Macy's would match the Gap, and strand the Movies. Bad. So I had to have multiple passes, or I had to think up some other way of looking for the sets.

What happened was that I was scanning the ruby Array docs and noticed the product() method. Interesting, and after about another 10 mins of trying to think up a solution, the ideas came to me: use product() to make pairs of merchants to check, and then add things in and remove duplicates.

Sweet idea!

  def self.group_by_service(otcs)
    # start with the array of groups that we'll be returning to the caller.
    groups = []
    # look at all non-identical pairings in the original list and for each
    # pairing, see if there are ANY common services. If there are, try to find
    # a group to place the PAIR in, if we can't, then make a new group of this
    # pair.
    otcs.product(otcs).map do |pair|
      next if pair[0] == pair[1]
      if services_overlap?(pair[0], pair[1])
        groups << pair unless (groups.map do |g|
          (g << pair).flatten!.uniq!.size unless (g & pair).empty?
        end.compact.reduce(:+) || 0) > 0
      end
    end
    # verify that each OTC is in some kind of group - even alone
    otcs.each do |d|
      groups << [d] unless
          (groups.map { |g| g.include?(d) ? 1 : 0 }.reduce(:+) || 0) > 0
    end
    # return the array of groups to the caller
    groups
  end

It's like an expanded APL to me. Compact code. Chained method calls. More work by the CPU, but less code written by the person. It's not something I'd traditionally write because it's excessively wasteful in the work it's doing, but I'm guessing that it'll be seen as evidence of me "getting it" by the other Ruby guys in the group.

I get it, and in certain instances, I don't think it's wrong. But in a production app that's going to hit speed limitations, have code like this is a killer to performance. There's too much that doesn't need to be done. Yeah, it looks nice, but it's going to put a tax on the machines that shouldn't have to be paid.

I get it… I'm just not sure I think it's a great thing.