What's it all about, Alfie?

Just finessed refactoring a class where I had used class variables. The code was very simple:

  class Demand
    def self.load(division, data)
      @@division = division
      @@data     = data
    end
 
    def self.reset
      @@division = nil
      @@data     = nil
    end
 
    def self.has_division(division)
      defined?(@@division) && !@@division.nil? && (@@division == division)
    end
  end

there's more to the class, but this points out that I'm using the class variables, and the API is really pretty simple. When I posted this as a pull request on GitHub, a guy on the team pointed out that others are going to want to modify the class variables to class methods as they are less dangerous.

I don't see the danger, when the API is the same, but I can understand the advantage of the class method approach - it makes it possible to stub() out the method in an spec test - which is nice as it simplifies the testing.

So I went back in and modified the code to look like:

  class Demand
    def self.load(division, data)
      @division = division
      @data     = data
    end
 
    def self.reset
      @division = nil
      @data     = nil
    end
 
    def self.has_division(division)
      !self.division.nil? && (self.division == division)
    end
 
    private
 
    def self.division
      @division ||= nil
    end
 
    def self.data
      @data ||= nil
    end
  end

The code is slightly simpler because I don't have to worry about the possibility of an undefined reference in the has_division method, but that's about it. The real advantage has to be in the stub-factor for tests.

OK… I'm easy… I can do it that way.

This morning I've spent a good deal of time factoring out the Singleton nature of the processing out application is doing. Rather than have singletons with thread-safe data structures, it's very simple to make a clear delineation of the loading, processing, and persistence phases of the app, and then apply thread pools where appropriate to work on the data in a parallel fashion without having to lock a single thing. It's a simple processing problem with a queue built into the thread pool doing the work.

Once all the work is done, we can then run the aggregation steps that cut across the data, in a single thread so as not to cause problems there as well. It's not rocket science, and I spent the vast majority of my time trying to unravel the code written for the atomic and immutable classes, but that's partly because of my unfamiliarity with the classes, and partly because it's a complete mess and looks nothing like the native ruby containers.

I'm glad it done - all in a branch and I've sent a pull request for debate. I think it's the right thing to do simply because of the simplification in the code and containers. The data flow is cleaner, and it'll be far clearer where to add in new functionality. It's really much better code.

UPDATE: I decided that I needed to have some sense of real measurement of the difference, so I added in a quick timing of the core of the code - the augmentation and matching. What I found was that the new, atomic and immutable-free way of doing things is significantly faster:

  processed 4605 merchants in 30310.0 ms = 6.5820 ms/merchant

for the old, atomic and immutable data way, and:

  processed 4605 merchants in 6198.0 ms = 1.3459 ms/merchant

with the cleaner, clearer, no-atomics and no-immutables way. So there's a real difference. True, the total runtime is hardly different, but that's because it's really being dictated by I/O, and we can't do a lot about that.

OK, it's nothing special, but today I took on a story where I needed to pull out the Salesforce client into a gem and put them gem in our local ruby-gems server. It's pretty nice how it all works - you create a simple 'stub' gem, populate it with files and data, package it up, check it into git, and publish it. It's really pretty simple. Clearly, it's very tightly coupled to git and the ruby-gems server, but that's not too bad, considering.

The class needed very little tweaking, but it needed a bit. I was able to easily retrofit the gem into our code and it was all done in about an hour. Clearly, it'll be faster the next time, but the process is easy:

  $ bundle gem my_gem

then edit the gemspec and put in the code as needed. Build it with:

  $ rake build

and there should be a gem file ready to deploy.

I'll take better notes the next time to make sure I get all the details, but it wasn't too hard.

We ran intro a very interesting problem today with Warbler, Bundler and a bug in Hamster. To set the stage, Bundler is able to have gems specified as git/GitHub resources with the simple Gemfile line:

  gem "Hamster",
       :git => "git://github.com/harukizaemon/hamster.git"

and you can even specify a SHA for the commit point you want. Very nice.

The problem is that when you do this, Bundler places the resulting Gem into a Bundler/gems/ subdirectory of your current /gems/ directory, and that makes finding it impossible for the standard gem tools - which includes ruby itself.

What you must do is to let Bundler "fix" the GEM_PATH as soon as possible in your app. Simply require it's setup code and you are good to go:

  require 'bundler/setup'
 
  # use Hamster as normal

This is OK, if you're going to include Bundler in your deployment package and you aren't using Warbler to place everything into a Jar for Java to execute. Then we get into some nasty problems.

Our situation is that we want to deploy a single jar file. This means using Warbler to jar things up easily. Again, this doesn't sound so bad, but it's this patching issue that Bundler uses that makes things very difficult - certainly when dealing with a jar. You see, Bundler doesn't "do jars" at all. It's looking for a file system to work on, so when you try to do the same thing within a jar, you get error messages on trying to change directories, etc. It's trying to move around the file system, but a jar isn't a file system, and JRuby doesn't make it any easier, so Bundler fails.

Warbler doesn't help here because it's a problem with JRuby and Bundler and the jar.

Our solution was to build the gem from the fixes and place is in a local gem server and then reference it in the Gemfile without using the git/GitHub scheme. This places the gem in the right directory, and that means we don't need Bundler to augment the GEM_PATH and that means it all works.

I'm convinced that the real solution is to fix/patch JRuby to allow all ruby-based file and directory operations to operate on a jar. However, that's probably not the typical deployment scheme as we really hardly ever leave a WAR file unexploded, and deploying Jars is pretty limited.

Lesson: Don't use Git Gems with Bundler if you're deploying to a Jar. It's just not going to work.

Today I ran into a nasty problem with a JRuby app where we're deploying the complete app as a single jar file to the server. That's a really nice idea - one atomic unit to move around, roll-back easily, all the things you'd expect… but it's got at least a few very nasty downsides, and it's got nothing to do with ruby - it's JRuby and how Java handles resources located within the jar as opposed to the filesystem outside the jar.

In short, it's not a seamless transition, and it'd be great if JRuby would handle this in all the File.open code so that we wouldn't have to. But that's probably asking a little much.

Still… to the problem at hand.

The code for reading a CSV file into a map in ruby is pretty simple:

  def self.read_csv(filename)
    res = {}
    CSV.read(filename, :headers => true).each do |rec|
      k = [rec['Size'], rec['Weight'], rec['Height']]
      res[k] = rec
    end
    res
  end

but it assumes that the file is located on the filesystem, and specifically, relative to the current directory of the running ruby VM. This isn't new, it's pretty standard, and very convenient.

But files in jar files aren't in the filesystem. They have to be located and read in as a byte stream:

  require 'java'
 
  def self.read_csv(filename)
    res = {}
 
    # get the contents of the file - no matter where it is
    contents = ''
    if File.exists?(filename)
      File.open(filename) do |file|
        contents = file.read
      end
    else
      # We appear not to have this file - but it's quite possible that
      # the file exists in the deployed jar, and if that's the case,
      # we need to access it in a more java-esque manner. This will be
      # a line at a time, but the results should be the same.
      f = java.lang.Object.new
      stream = f.java_class.class_resource_as_stream('/jar_root/' + filename)
      br = java.io.BufferedReader.new(java.io.InputStreamReader.new(stream))
      while (line = br.read_line())
        contents << "#{line}\n"
      end
      br.close()
    end
 
    # now we can take the contents of the file and process it...
    CSV.parse(contents, :headers => true).each do |rec|
      k = [rec['Size'], rec['Weight'], rec['Height']]
      res[k] = rec
    end
    res
  end

Here, the bulk of the code is about getting the file into a string that we can then parse. It first tries to see if it's on the filesystem, and if that fails, it tries the jar to see if it happens to be there. Unfortunately, it's got to be the full path to the file in the jar, and if you're using a packager that tacks something on the front, you need to be aware of this.

Not horrible, but it was an hour to figure this all out and get it nicely coded up so we didn't have too much redundant code.

This morning I noticed that Google Chrome dev 22.0.1229.0 was out, and the release notes are getting to be somewhat of a disappointment. To say they are sparse is an understatement. Take the UI change of the 'wrench' to the 'pancakes':

that's a change, and there's got to be a reason for it, but the release notes are totally silent about it. "Read the SVN logs" is hardly an answer - at least it better not be. There should be a lot more information in the SVN logs than this, and these are higher-level issues that need addressing.

Yeah, the quality of the release notes is really slipping on this project.

I can really appreciate that building things in a large-scale web system means decentralization. You need to have services, and those services have to be well-defined and walled off so that changes in one service are easily adopted by another, but don't break existing apps. In short - I get it. But that doesn't mean it's easy.

The problem I'm running into today is that in order to accomplish my goals, I need other people to add things to their services so that I can get the data from them and then process it. In the past, I haven't liked this either, but it's typically been the case that I simply figured out how to update that code, and made the changes myself. Alternatively, I just didn't have the external dependencies as management wanted me to build something from the ground up.

It's not that I think these people are bad, or can't do the work. It's that they are required to make progress. It's the difference in working in a large, disparate team, and working in a small, focused team. I like the latter because it means you do very little sitting around, and a whole lot of getting stuff done.

But the flip side is that you're the key man, and there are very few people that understand the project. Like I said, I can see the flip-side.

Today has just been one of those days where I'm doing a lot of waiting for people, and I'd like to be doing a lot of coding. After all, I'm a coder, not a professional waiter.

This morning I've finally gotten tired of the font choice for the Pragmatic Programmer's Programming Ruby 1.9 book. It's a lot smaller and not nearly as readable as every other Pragmatic Programmer's book I've got. I mean really bad. First, I read all my tech books on my MacBook Pro. I have a size I like to give them on the screen. I know the font is fixed on a PDF, but still… I find this very readable:

yet here's the PickAxe book at even more screen real estate:

To me, there's no contest.

So I wrote to the PP guys - on the off-chance that they might be interested in my feedback:

Guys,

I love your books. They are by far the "go to" books for learning anything I need for a new project or job. I have several. Wonderful stuff.

However, I have a slight bone to pick with you on the font choice for the Ruby 1.9 book.

What's up with the font? It's not the same font that I see on every single one of my other PragProg books. I like the "other" font. It's easy to read on computer screens as a PDF, and it's very nice and legible at a somewhat reduced size.

But the font for the Ruby 1.9 book is almost the exact opposite. Very hard to read. Nowhere near as clear.

It may be stylish, but as for me - give me back the old font and layout with the blue borders and the style that made PragProg the best tech books around.

I know it's just one voice, but hey... if you don't hear it from your friends, then others will just talk about you behind your back, right?

Dave Thomas wrote back:

As the person who picked the font, let me say I really appreciate the feedback.

The choice of font was difficult. The PickAxe had grown from a hefy 450 pages to a massive 950, and I really wanted to stop the trend. So I looked for ways to thin it down. One choice I made was to go for a font which was a little skinnier and a little more open, which would let me sometimes squeeze just a few more lines on the page. This one difference let me save about 60 pages overall. I also changed the layout of the standard library section and saved another 40 or so.

It's always a tricky compromise, but I hope that the marginal decrease in legibility is offset by the many thousands of pages that won't be printed 🙂

I can see his point - it would be a monster printed book, and a lot of people buy his books like this. I wrote back saying so, and he suggested reading it as ePub or mobi where I could set the font as I pleased. The problem with that is that there are no good ePub or mobi readers for the Mac that preserve the formatted text of the code samples. None. So I can read PDF that at least looks right, or nothing.

Not happy with the alternatives, but I don't have a lot of say about it either. It is what it is, and it's up to me to just suck it up and deal with it.

Shucks.

Today I got the latest news from GitHub about the new Notifications and Stars features, and how they work together to make it much easier for you to tag your interest in a repo without getting all the notifications from every one. Very nice.

These guys are really quite impressive. I'm glad to have code there - both open source and private. It's an amazing place that's coordinating all the source control, issue tracking, and documentation functions of a project under one hood. It's really quite impressive.

And to see that they are continuing to improve it with new features and better style is just amazing. I can't remember the last time I was this impressed with Jira. OK… I have never been that impressed with Jira.

Great work for an amazing service.

It really is nice to be busy. Maybe this is feeding my finance problem, and I'd be better off de-toxing all the way, but it really is nice to be responding to requests quickly and turning around answers to folks quickly and efficiently. It gives me a sense of accomplishment, and that's really nice when you're still The New Guy at a place. Feeling like you make a difference, like you matter - that's nice stuff.

It's all about how we feel about ourselves, isn't it? No matter what you're doing, where you're doing it… if you feel good about yourself it's meaningful work. Good enough.