What's it all about, Alfie?

We ran intro a very interesting problem today with Warbler, Bundler and a bug in Hamster. To set the stage, Bundler is able to have gems specified as git/GitHub resources with the simple Gemfile line:

  gem "Hamster",
       :git => "git://github.com/harukizaemon/hamster.git"

and you can even specify a SHA for the commit point you want. Very nice.

The problem is that when you do this, Bundler places the resulting Gem into a Bundler/gems/ subdirectory of your current /gems/ directory, and that makes finding it impossible for the standard gem tools - which includes ruby itself.

What you must do is to let Bundler "fix" the GEM_PATH as soon as possible in your app. Simply require it's setup code and you are good to go:

  require 'bundler/setup'
 
  # use Hamster as normal

This is OK, if you're going to include Bundler in your deployment package and you aren't using Warbler to place everything into a Jar for Java to execute. Then we get into some nasty problems.

Our situation is that we want to deploy a single jar file. This means using Warbler to jar things up easily. Again, this doesn't sound so bad, but it's this patching issue that Bundler uses that makes things very difficult - certainly when dealing with a jar. You see, Bundler doesn't "do jars" at all. It's looking for a file system to work on, so when you try to do the same thing within a jar, you get error messages on trying to change directories, etc. It's trying to move around the file system, but a jar isn't a file system, and JRuby doesn't make it any easier, so Bundler fails.

Warbler doesn't help here because it's a problem with JRuby and Bundler and the jar.

Our solution was to build the gem from the fixes and place is in a local gem server and then reference it in the Gemfile without using the git/GitHub scheme. This places the gem in the right directory, and that means we don't need Bundler to augment the GEM_PATH and that means it all works.

I'm convinced that the real solution is to fix/patch JRuby to allow all ruby-based file and directory operations to operate on a jar. However, that's probably not the typical deployment scheme as we really hardly ever leave a WAR file unexploded, and deploying Jars is pretty limited.

Lesson: Don't use Git Gems with Bundler if you're deploying to a Jar. It's just not going to work.

Today I ran into a nasty problem with a JRuby app where we're deploying the complete app as a single jar file to the server. That's a really nice idea - one atomic unit to move around, roll-back easily, all the things you'd expect… but it's got at least a few very nasty downsides, and it's got nothing to do with ruby - it's JRuby and how Java handles resources located within the jar as opposed to the filesystem outside the jar.

In short, it's not a seamless transition, and it'd be great if JRuby would handle this in all the File.open code so that we wouldn't have to. But that's probably asking a little much.

Still… to the problem at hand.

The code for reading a CSV file into a map in ruby is pretty simple:

  def self.read_csv(filename)
    res = {}
    CSV.read(filename, :headers => true).each do |rec|
      k = [rec['Size'], rec['Weight'], rec['Height']]
      res[k] = rec
    end
    res
  end

but it assumes that the file is located on the filesystem, and specifically, relative to the current directory of the running ruby VM. This isn't new, it's pretty standard, and very convenient.

But files in jar files aren't in the filesystem. They have to be located and read in as a byte stream:

  require 'java'
 
  def self.read_csv(filename)
    res = {}
 
    # get the contents of the file - no matter where it is
    contents = ''
    if File.exists?(filename)
      File.open(filename) do |file|
        contents = file.read
      end
    else
      # We appear not to have this file - but it's quite possible that
      # the file exists in the deployed jar, and if that's the case,
      # we need to access it in a more java-esque manner. This will be
      # a line at a time, but the results should be the same.
      f = java.lang.Object.new
      stream = f.java_class.class_resource_as_stream('/jar_root/' + filename)
      br = java.io.BufferedReader.new(java.io.InputStreamReader.new(stream))
      while (line = br.read_line())
        contents << "#{line}\n"
      end
      br.close()
    end
 
    # now we can take the contents of the file and process it...
    CSV.parse(contents, :headers => true).each do |rec|
      k = [rec['Size'], rec['Weight'], rec['Height']]
      res[k] = rec
    end
    res
  end

Here, the bulk of the code is about getting the file into a string that we can then parse. It first tries to see if it's on the filesystem, and if that fails, it tries the jar to see if it happens to be there. Unfortunately, it's got to be the full path to the file in the jar, and if you're using a packager that tacks something on the front, you need to be aware of this.

Not horrible, but it was an hour to figure this all out and get it nicely coded up so we didn't have too much redundant code.

This morning I noticed that Google Chrome dev 22.0.1229.0 was out, and the release notes are getting to be somewhat of a disappointment. To say they are sparse is an understatement. Take the UI change of the 'wrench' to the 'pancakes':

that's a change, and there's got to be a reason for it, but the release notes are totally silent about it. "Read the SVN logs" is hardly an answer - at least it better not be. There should be a lot more information in the SVN logs than this, and these are higher-level issues that need addressing.

Yeah, the quality of the release notes is really slipping on this project.

Today I got the latest news from GitHub about the new Notifications and Stars features, and how they work together to make it much easier for you to tag your interest in a repo without getting all the notifications from every one. Very nice.

These guys are really quite impressive. I'm glad to have code there - both open source and private. It's an amazing place that's coordinating all the source control, issue tracking, and documentation functions of a project under one hood. It's really quite impressive.

And to see that they are continuing to improve it with new features and better style is just amazing. I can't remember the last time I was this impressed with Jira. OK… I have never been that impressed with Jira.

Great work for an amazing service.

This morning I found a problem with the Ruby gem VCR. It turns out that if it hits a service that reports it's data as ASCII encoded, but actually sends UTF-8, then the data will be stored by VCR as ASCII, but will be unable to be read out of the cassette. Very nasty. The solution is to force VCR to record the actual bytes and base64 encode them. This is easily done with the code:

  require 'vcr'
 
  VCR.configure do |c|
    c.cassette_library_dir = 'spec/cassettes'
    c.hook_into :webmock
    c.default_cassette_options = {
      :record => (ENV["VCR_RECORD"] ? :new_episodes : :none),
      :match_requests_on => [:method, :url, :path, :host, :body]
    }
    c.ignore_localhost = true
    c.allow_http_connections_when_no_cassette = true
    c.preserve_exact_body_bytes do |http_message|
      http_message.body.encoding.name == 'ASCII-8BIT' ||
      !http_message.body.valid_encoding?
    done
  done

The code is simple - if the message body is reported from the service as ASCII-8BIT, then don't trust it. Likewise, if the body has no reported valid encoding.

This is a nice, conservative way to ensure that the cassettes get written in a way that it's ensured that we'll be able to read it back out. While it might be nice to have a human-readable version, it's not worth the problems of badly behaved services that say they are ASCII and really return UTF-8.

I'm a big fan of refactoring - heck rewriting, code. It's good because the first time you write something, you're going to make mistakes, and the more you re-work it, the better it's going to be. Or so the theory goes… and I tend to agree with that theory… so long as some kind of limits are imposed.

I'm not talking about being unreasonable here, either. If you are feeling that a class or method needs to be changed in order to make it more readable, more maintainable, or fix a serious performance issue, then by all means - do it. But if you're changing 10 lines for 10 different lines… or doing meta programming to try and make things neater or cooler, then I want to stop you right there and say What's the benefit of this change?

Is it really better? More readable? More maintainable? Or is it that you thought of another way to write the same thing, and wanted to write it just because you could? I'm running into quite a bit of that today, and because I'm the new guy, and don't want to rock the boat, I sat back and learned a lot about how to write 10 lines of ruby code about six different ways. I learned a lot, but the code didn't benefit one little bit.

The developers discussing this - there were three of us on this at that time, were talking about the relative merits of the indentation style, and the warranted differentiation of the methods of generating the methods dynamically in ruby. All might be considered valid issues, but none of this makes the code better.

None.

So after we got all this checked in and done, we started talking a little and I just brought up the point that in the past, this would have never been allowed in the places I've been. There is just too much to do to allow three high-powered developers to be tied up for more than an hour for 10 lines of code that were fine before anyone got involved. It would have been seen as a massive waste of time.

But in truth, there was benefit. The methods are smaller by a bit. I learned a lot. And the calling arguments are now more general. This is all nice, but it certainly doesn't compare favorably to the cost. But maybe that's the difference outside of Finance. Maybe the sustainability is more reasonable here. Maybe it's OK to do this and "relax", and allow everyone to recharge from time to time.

It's certainly different.

Yesterday I spent a good deal of time tracking down something that appears now to be an error in JRuby 1.6.7, which is what we're using on this code. The problem was a little nasty in that the code was generating a java exception and depending on the code we were using, it was either getting swallowed completely, or only showing the message of the exception - and not the complete stack trace as you'd expect in a Java exception.

But let's set the stage, and in doing so expose the error that led to this horrible problem. The code starts out with a seemingly innocent ruby module:

  module Bun
    private
 
    def wrap(dog)
      puts "wrapping up #{dog}"
    end
 
    def box(dogs)
      puts "boxing up #{dogs.join(', ')}"
    end
  end

The idea is that the Bun is a module the has a few methods that we'll need in some other classes, and similar to an ObjC Category, it's placed here and just re-used in the classes with the 'include' directive.

Next comes a class that uses the Bun module - just as you'd expect.

  require 'bun'
 
  class Fixer
    include Bun
 
    def fix(count)
      count.times wrap('beef dog')
    end
  end

Nothing fancy here.

The next class was intended to use this module when it had more functionality, but refactoring took that functionality out of the module and into the first class. Still, the refactoring didn't catch the difference, and so the code of the second class looks like this:

  require 'bun'
 
  class Shipper
    include Bun
 
    def ship(count)
      fix(10)
    end
  end

Technically, the following code should not work:

  require 'shipper'
 
  s = Shipper.new
  s.ship(1)

and in truth, it doesn't. But in the larger context of the app where there are many more things happening, it passed the compile and run tests only to find that there was a stack crash exception. So far, this was annoying, but that's not where it stopped.

The real code we had looked more like this:

  require 'java'
  require 'java.util.concurrent.Executors'
  require 'java.util.concurrent.TimeUnit'
 
  require 'shipper'
 
  executor = Executors.new_fixed_thread_pool(4)
  10.times do
    executor.execute do
      begin
        s = Shipper.new
        s.ship(1)
      rescue => e
        puts "caught: #{e.to_s}"
      end
    end
  end

So we have a multi-threaded app, using the Java Executors to help things out, and we got an exception that didn't get caught by the rescue clause. Very odd. Even worse, if we changed the call to use the submit method:

  10.times do
    executor.submit do
      begin
        s = Shipper.new
        s.ship(1)
      rescue => e
        puts "caught: #{e.to_s}"
      end
    end
  end

we got NOTHING. The exception was thrown, but completely swallowed up by the Executor's submit call.

After a bunch of reading, the latter at least makes sense in the context of the JVM. All exceptions will get swallowed up in the submit call, and it's up to you to keep the future, and interrogate it. So that was explained. What wasn't explained was this: When I took out the Executor, I got the message and the stack trace!

OK, to summarize: Use the Executors and you get the exception message - Don't use it and you get the message and the stack trace.

I've tried to make a simple test case, the one above, but it's properly seen as not being able to find the method on the object. But in our more complex code, that's not the case. When I talked to a friend about this, I ended up getting this tweet in response:

As I've seen this guy today on the JRuby bug tracker, I think he really knows what he's talking about. Sadly, I can't easily reproduce the problem, and it's often these complex ones that are the really tough ones to reproduce. However, it's fixed, and we have a work-around (simply comment out the Executor block) so we're able to proceed. Also, I trust that it's fixed in JRuby 1.7, so when we want to go there, that'll be waiting for us.

In retrospect, this has been a lot of work, but a real insight into ruby, JRuby, and the support community. It's been a real enlightening experience.

This morning I read an interesting fact about OS X 10.8 Mountain Lion - they removed the Web Sharing from System Preferences -> Sharing. Odd. I didn't use it a lot, but they also reset the config files for Apache, so it's something that I'm going to need to work on when I finally upgrade to 10.8.

Thankfully, the post talks about how to get all this functionality back and includes nice screen snapshots, so when I need to, I'll have a nice reference to walk me through anything I haven't seen before.

I know I need to get Apache, PHP, PostgreSQL all working together as my work on SyncKit requires it. So I'll have to factor that into the time for the upgrade.

Still planning on doing it, just not right today.

I just noticed that Google Chrome dev 22.0.1221.0 is out and save the inclusion of a new version of the V8 javascript engine (3.12.16.0), the release notes are a little skimpy to say the least. I mean, would it really kill them to put more than 2 mins into the release notes? Are SVN commit logs really sufficient for release notes?

I'd like to think not. But I'm not in their group, so it's not my call. Still… it makes you wonder what their attention to detail level really is...

This afternoon I wanted to create a log4j-like logger for the ruby app we're working on. I was hoping to find a gem that did it - singleton, thread-safe, the works. What I found was that the default logger: "logger" is really pretty close, and it's easy to make it what we need.

First, it's thread-safe. Doesn't say that in the docs, but the code has synchronize blocks on a mutex, and that appears to be working, so while I can't guarantee that it's implemented correctly, it appears that it is, and that's good enough for now.

Second, it's not a singleton, but that's easy to fix:

  require "singleton"
  require "logger"
  require "app_config"
 
  class AppLog
    include SIngleton
 
    def initialize()
      # get the location to log to and the level
      where = AppConfig.application.log_to || "stout"
      level = AppConfig.application.log_level || "WARN"
      # now make the logger for our singleton
      case
      when where == "stdout"
        self.log = Logger.new($stdout)
      when where == "stderr"
        self.log = Logger.new($stderr)
      else
        self.log = Logger.new(where)
      end
      # now set the log level
      case
        where level == "FATAL"
          self.log.level = Logger::FATAL
        where level == "ERROR"
          self.log.level = Logger::ERROR
        where level == "WARN"
          self.log.level = Logger::WARN
        where level == "INFO"
          self.log.level = Logger::INFO
        where level == "DEBUG"
          self.log.level = Logger::DEBUG
      end
    end
  end

This can then be placed in all our code and we get singleton-based logging for next to nothing. Very nice. At the same time, we can control the logging location and level very easily in the app config file.

I then took this and integrated it into the code I was working on for matching up the demand data with the merchants. It's something that we really need to do to productionize our app, and this is an important step.