What's it all about, Alfie?

I did it again today. I pushed myself to get a lot of stuff done today for a big, important demo (again), and along the way a few people interrupted me to ask me to look at a few things. Had I been smarter… had I been wiser… I'd have said "I'm sorry, I'm on this push for the demo, how about I get to it on Monday?" But I didn't.

What I did was to push myself to the point that I was very upset with the things I was working on. Oh, it didn't start out that way. It started out with me thinking that I could easily track down this problem that one of the data science guys pointed out. It was in my code for grouping the merchants where I wanted to be smart and clever. I should have known.

First, I wasn't accounting for the case where two groups of overlapping merchants are built and then a single merchant bridges both groups. I messed up. So I needed to go back and fix a few things. First off, I didn't have a general overlapping service method. So I took the old one and expanded it:

  # this method returns true if ANY service is shared between the two
  # merchants. ANY.
  def self.services_overlap?(ying, yang)
    if ying.is_a?(Array)
      # explode the calls for an array in the first position
      ying.each do |d|
        return true if services_overlap?(d, yang)
      end
      return false
    elsif yang.is_a?(Array)
      # explode the calls for an array in the second position
      yang.each do |d|
        return true if services_overlap?(ying, d)
      end
      return false
    end
    # this is the simple call for a 1:1 check
    !(get_services(ying) & get_services(yang)).empty?
  end

Basically, I just allowed arrays to be passed in, and where necessary, I exploded them to allow the basic logic to be applied to the individual merchants. It's not hard, but at this point, I didn't need to worry about what I was passing in to check for an overlap.

The next thing was to completely redo the group_by_service() method as it was far too complex, and it wasn't even working. I didn't like the fact that it was doing a lot of extra checks, etc. but that seemed to be the Ruby Way. Poo. I changed it into a simple single-pass loop that's far simpler and far faster:

  def self.group_by_service(otcs)
    # start with the array of groups that we'll be returning to the caller.
    groups = []
    otcs.each do |d|
      added = []
 
      # add the OTC to all groups that it has some overlap with
      groups.each_with_index do |g, i|
        if services_overlap?(d, g)
          g << d
          added << i
        end
      end
 
      # if we added it to more than one group, then consolidate those groups
      if added.size > 1
        added[1..-1].each do |i|
          groups[added[0]].concat(groups[i][0..-2])
          groups[i] = nil
        end
        groups.compact!
      end
 
      # if he hadn't been added to anything, make a new group for him
      groups << [d] if added.empty?
    end
 
    groups
  end

The ideas here are a lot clearer - add a new guy in to all the group he'd match, then look for multiple matches. If there are, then simply and cleanly consolidate the groups, and continue. My co-worker in Palo Alto liked this code a lot more as well. I do too. It's ruby, it's just not incomprehensible ruby. And it's right.

But in the midst of this, I'm trying to get more web stuff done for another interruption for the demo, and it's nothing I'm happy with. It's all very lame. I don't like the interface I have to CouchDB, I don't have a lot of time, and it's making me very cranky.

I've run into this before, and it's not the first time I've not been able to say "No", and it's cost me something I didn't want to pay. I know I'm no great graphics designer. I know I can make things that work, but they aren't going to be "Wow!" with anyone but someone looking just at the functionality. That's just not in my DNA. And it is exceptionally frustrating to be in a situation where I'm forced to do this work.

I know what is good. I can appreciate it. But I can't generate it. And to be forced to do it is hard. Because I know they are going to throw it all away. Any half-decent designer will look at what I've done and say "Nice, but let's take out the data collecting and put it in a nice design" - as they should.

So I've got to work harder at keeping my cool.

I've got to say "No" more often.

Or I just won't last.

I was talking with a good friend this morning and I came up with a name for a lot of the ruby devs I've run into - but to be fair, it's not just for a good chunk of the ruby devs I've met - it's for a general class of developers. Let's pretend to be a little more precise about this:

Code Monkey - a developer that is more interested in learning a language and how to solve a few problems in it, than using it to solve real-world problems. This includes, but is not limited to, the clojure devs that have never written a comment, and only solved the zebra/water puzzle, as well as devs that never code defensively, or even think that production is important.

This came up because I'd been battling code that wasn't written at all defensively. It was basically assumed to have been run by a person, with a person to fix any problems as they occur. It's like a glorified Excel spreadsheet - I'm going to hit 'Go', and fix things that come up.

But this doesn't really work for real life, does it? Who wants a system that runs at night that has to be constantly monitored to make sure it doesn't get bad data, etc.

Yet they are the first ones that are onto a new language - like clojure. Saying that the real solution is to use a language that doesn't need all that checking as a functional language simply doesn't require it.

What world are they living in?

How is a language able to do ETL on it's own? Answer: It can't. You still have to do it. But the Code Monkeys are really skipping all that because they start with good data and then the process is clean, and simple.

No kidding? Really? Well, of course it is! The same is true for C++, Java, and any other language you want to pick. Start with clean data, don't worry about exceptions and potential problems, and you're going to be able to write amazingly clean code. But that's not how life really works.

We agreed that there were guys with language knowledge, and skills, but they never really dug in and made it work. It's nice to talk to Code Monkeys, but it's not nice to have to work with them. You're always cleaning up their messes.

Once again, we're gearing up for a big demo tomorrow with some of the users, so today has been full of a lot of things that needed to get done in order to have a successful presentation. I had to make quite a few new views for CouchDB, and then work those into a web page and publish everything up to UAT and production for a run. I'm becoming a big fan of CouchDB and it's views and reductions… those are some very powerful tools for looking at this JSON data in CouchDB. Very nice.

I've also done a little fiddling with the Sublime Text 2 styles, and I'm exceptionally happy how that's all turned out. It's made it much nicer to work with. I can't believe I haven't tried it up to this point, and I can't imagine a better editor for me. I'm going to have to brush up on my Python and write some packages some day.

Finally, I'm just dead tired. Great feeling. Lots of really good work done, and the team is really working together far better than I'd have thought a month ago. This is really quite fun.

This morning I was getting tired of the pretty lame syntax highlighting of YAML files in Sublime Text 2 - and I know it could be better. So I started digging. The first thing I looked at was the tmTheme file that I can cloned of the Eiffel theme in the standard release package. It's close… white background, nice colors, but it's not perfect, and I wanted perfect. So here's what I found out.

The matching of the language is really in the tmLanguage files in the packages. These are a bunch of regexs, and it's fine, but each pattern match then pins the color to use to some "classification" - a dotted-notation similar to a Java package. The idea is that if you specify only the first part or parts, then the last parts are up for specialization.

For instance, if you want to have a numeric constant style, it makes sense to build them hierarchically: constant -> numeric -> yaml, this leads to the classification: constant.numeric.yaml. But if you want all constants to be a certain style (by default), you can simply specify the constant style in your tmTheme file.

Alternatively, if you want all your numeric constants to be a certain style except those in java, you make a style for constant.numeric and then a new one for constant.numeric.java. Simple. But certainly not simple to figure out by looking at the files.

So I realized that for YAML, I didn't want the 'Embedded source' to have a colored background. So I added:

  <dict>
    <key>name</key>
    <string>Embedded source</string>
    <key>scope</key>
    <string>source.php.embedded.block.html, string.unquoted.yaml</string>
    <key>settings</key>
    <dict>
      <key>background</key>
      <string>#FFFFFF</string>
    </dict>
  </dict>

so now it's got a white background. Nice.

The next thing was to notice that I didn't like that the keys in YAML were red like almost all the text (strings, constants, etc.) so I wanted to make those keys blue:

  <dict>
    <key>name</key>
    <string>Markup name of tag</string>
    <key>scope</key>
    <string>entity.name.tag.yaml</string>
    <key>settings</key>
    <dict>
      <key>fontStyle</key>
      <string>bold</string>
      <key>foreground</key>
      <string>#1C02FF</string>
    </dict>
  </dict>

and now the keys are a nice blue. Much better!

All this is just in my clone of the Eiffel theme in the Packages/User/ directory in the Application Support for Sublime Text 2. Very nice.

UPDATE: I realized it should be easy to do the same for PHP - which has the annoying background color, and it was! You simply have to look into the tmLanguage file and see the tag name that's used and place it in the string in a simple comma-delimited list. Very slick!

UPDATE: I noticed a few more that I wanted to add - all from the HTML syntax highlighting. The code became:

  <dict>
    <key>name</key>
    <string>Embedded source</string>
    <key>scope</key>
    <string>
      source.php.embedded.block.html,
      source.css.embedded.html,
      source.js.embedded.html,
      source.python.embedded.html,
      source.ruby.embedded.html,
      string.unquoted.yaml
    </string>
    <key>settings</key>
    <dict>
      <key>background</key>
      <string>#FFFFFF</string>
    </dict>
  </dict>

This morning I noticed that Java 1.6.0_35 was updated on Software Updates most likely due to a security issue that's been patched. Given the way Oracle is handling Java, I'm really wishing that Apple would retain control of Java for OS X. Right now, I'm wishing they had it slightly better integrated into the OS such that starting a JVM instance wasn't so time-consuming. Linux handles this with all the shared libs being loaded. Then it's a very lightweight thing to spin up the JVM. On OS X, it's a lot more.

Still, it's nice to see that they have at least one more update. Maybe cooler heads will prevail in the coming months? Most likely not, but a guy can wish, can't he?

I had to do a special gem today, and I wanted to get down how it was build and deployed. This is because I didn't remember it and I'd already done this once. So here goes.

First off, make sure that you have the skeleton of a gem - including the gemspec file - it comes with all GitHub repos, and if you make a new gem with the Bundler, it gives you a simple skeleton as well.

Next, write the code and then build it:

  $ rake build

this should deposit it in your pkg directory.

Upload it to a site - like ruby gems. Simple.

It's worth noting that it might help to delete older versions of the gem. This is easily done with:

  $ gem uninstall couchrest

If there are multiple versions of the gem installed, it'll give you a choice. If there's only one, it's gone.

Fix your Gemfile and then reload all the gems with:

  $ bundle install

Today I was really battling a nasty problem with the CouchRest client for CouchDB. This is a ruby gem, and in general, it's pretty decent, but it really starts to fall down when you need to use a proxy to get to CouchDB, and this guy starts having all kings of problems.

There were timeout issues, so I decided to try and make it work. So I forked the project on GitHub, and started to get to work. THe major point was that the underlying RestClient gem had the ability to set things like the timeout for creating the connection as well as timeouts for reading, etc. It's really very flexible. My goal was to allow the settings to be applied on a per-database basis. Then, for every command, use these as the defaults, but overlay any call-time options as well.

The idea was really nice. I was planning on submitting a pull request for this as it only took me about an hour to do. But when I went to test it, it failed with some 502 Bad Gateway error.

Argh!

More proxy problems!

Then I was talking to one of the guys in the group about this issue and he brought up that I could write to my local CouchDB, and then replicate it to a different database on a different server!

BING!

This is exactly what I'd been looking for. I get fast and efficient writes to my CouchDB, but it gets written up to the shared server as I'm connected to the network. This is great!

The configuration is simple - it's a simple doc in the _replicator database, and I'm in business. This is really quite amazing. First, go to the overview in the CouchDB web page, and select the _replicator database:

then create a new document:

Finally, populate it with these essential key/value pairs:

• source - this is the source for replication - it only goes one-way, so this can be a local database, or a remote one. But it's where the data is coming from
• target - this is the destination for replication - again, local or remote, it makes no difference. Make sure to put in the port and the database name in the URL
• proxy - if needed, put in the URL of the proxy to get to either of these databases
• continuous - set to true if you want it to always be replicating

Save this document and then look at the CouchDB status page to see it replicate. It's like magic! OK, not really, but the handling of the proxy is so far superior to the way that CouchRest was dealing with it it's not even funny. This just works!

I'm more and more convinced that CouchDB is an amazing product.

I just read this tweet this morning:

which leads to this article advocating that young people look to entering an apprenticeship program and not continue school. It says, in part:

Universities are the typical place that established businesses expect to find these high-potential beginners. While many software developers finish college with a good education, they’re often burned out, deep in debt, and understandably eager to cash in on their hard work. Apprentices, on the other hand, inject enthusiasm, hard work, and a thirst for knowledge into your teams. They will consistently launch from your apprenticeship program with context, momentum, and loyalty to your organization.

While I can understand the point of the article, and you should read it to get that it's not saying people shouldn't go to higher education, it's saying that you, as a business owner, can capitalize on the cost of higher education, and get those people that might go to college and get them into the workforce.

But is that what we want to have happen, as an industry? I don't think so. I think it's robbing the future to staff the present, and that's a mistake. A big one.

I'm biased. I've got the higher education and the advanced degrees, and I think they are the right thing to do. But even if you discount my position, and do what the author suggests, aren't we just creating a bunch of Software Plumbers? They'll know what they see, and will be able to work with it, but their understanding of how to solve new and unusual problems will be very limited. Oh sure, you'll have a few percent that naturally think outside the box, but their exposure to new things and new ideas will be incredibly limited.

This is the exact purpose of those liberal arts classes for engineers - to broaden a student's horizons. If we just allow people to learn what we want them to learn, aren't we really just forcing ourselves to re-train them when we want to change technologies? Of course we are.

While there are times to have an apprenticeship program - for those that can't make it into college, I think it'll be overused and draw the real future of the profession into one where only a few can really think creatively. And that would be very bad.

This morning I decided that it'd be nice to have a complete list of all the merchants that didn't successfully complete their processing. Since we are now processing everything - a recent change to the code to make sure that we know exactly every single merchant is getting completely processed, we can now look at each merchant and make sure that they got through the critical processing phases. If it didn't "pick up" the right data, then we can assume that it didn't get to that point. The point of this is that we can then be sure that every merchant completed processing.

I log this, and write them to CouchDB, so we can keep a complete record of all the issues, and then updated the summary script to list the number of incompletely processed merchants so we can watch them over time.

Nice. This is starting to really get close to verification that all was done as it was supposed to have been done.

It didn't take long - just a few days, and now Google Chrome dev 23.0.1255.0 is out with a nice array of fixes for crashing bugs - including a video problem on retina MacBook Pros. There are a few things about the security of apps in the browser, which I don't use, but I'm sure there are quite a few Angry Birds fans out there.