Archive for the ‘Coding’ Category

Google Chrome dev 22.0.1229.6 is Out

Wednesday, August 15th, 2012

Google Chrome

This morning I noticed that Google Chrome dev 22.0.1229.6 was out, but without any release notes. None. Zip. I did notice that the "Stable" channel of Google Chrome is on 21.x, so it's possible that the 'dev' channel is about to just to 23.x, and this is a last little patch before the big jump, but that's just a guess.

Something changed, and it's nothing I can visually see, so I'm guessing it's not a major deal, but it's interesting that they've gone from "sparse" to "nonexistent" release notes. Maybe they're cutting back at Google? Can't afford the digital ink? Crazy.

Making Really Great Progress Today

Tuesday, August 14th, 2012

GeneralDev.jpg

Today we've had a lot of really great work done. Things are starting to accelerate, in part because we're down a few guys due to a clojure-rewrite of the main app, but also because we're focused on making real and significant progress on the project. It's been all-(remaining)-hands on deck, and it's been really nice.

I've done a lot of email to folks about things we need their assistance with. In some cases it's just knowledge transfer, in others it's liable to be a little more complex. Still, we need to get all these issues figured out before we release this to the users, and since someone has to do it, it might as well be me.

At the same time I've made some nice progress on closing some stories for the calculations done and logged in the app. Nothing amazing, but due to the optimistic nature of 90% of the ruby code I've seen, it really means tracking down the potential errors and making the code more robust as well as implementing the actual features. It's nice to see things really start to take the shape of a completed app - especially when there are so many people working on it, as it more often resembles swiss cheese than a working app.

I still would like to get some answers from a few folks, but that's what I'm waiting on now - external dependencies. I've never been very patient, but I'm going to try hard to last this out with a nice, and gentle attitude. Personal growth time.

Sublime Text 2 – An Amazing Editor

Tuesday, August 14th, 2012

Sublime Text 2

I've been talking to a few friends about editors on Mac OS X, and one editor that's getting a lot of press lately is Sublime Text 2. It's supposed to be a variant of Vim, and that's always a plus - it's even got a vintage mode to make it mimic the command and insert modes of Vim. It's got an amazing speed profile as well as an incredible Python-based extension library. And there are a ton of things that have been written for it.

So this morning I decided to give it a try. I'm glad I did. It's an amazing platform for coding. I can "open" a project - basically a directory that is my git repo, and then with the "Find Anything" command, I can load up a file very simply. Amazingly so. This is my major concern about BBEdit - opening files in a ruby environment is really a pain - as there are a lot of them.

There are so many nice things that it's just too hard to know where to start. There's a GitHub package for dealing with Gists - one of my favorite places to store snippets of code. This guy even gets the file names so you don't have to know the Gist number beforehand.

There is a legacy Vim mode, and a legacy Ex mode. Both are really nice for old school Vim users like me. Also, the app works on Windows, Linux, and OS X.

I'm looking forward to hammering on this and seeing where it takes me. Very excited.

Class Variables vs. Class Methods

Monday, August 13th, 2012

Unit Testing

Just finessed refactoring a class where I had used class variables. The code was very simple:

  class Demand
    def self.load(division, data)
      @@division = division
      @@data     = data
    end
 
    def self.reset
      @@division = nil
      @@data     = nil
    end
 
    def self.has_division(division)
      defined?(@@division) && !@@division.nil? && (@@division == division)
    end
  end

there's more to the class, but this points out that I'm using the class variables, and the API is really pretty simple. When I posted this as a pull request on GitHub, a guy on the team pointed out that others are going to want to modify the class variables to class methods as they are less dangerous.

I don't see the danger, when the API is the same, but I can understand the advantage of the class method approach - it makes it possible to stub() out the method in an spec test - which is nice as it simplifies the testing.

So I went back in and modified the code to look like:

  class Demand
    def self.load(division, data)
      @division = division
      @data     = data
    end
 
    def self.reset
      @division = nil
      @data     = nil
    end
 
    def self.has_division(division)
      !self.division.nil? && (self.division == division)
    end
 
    private
 
    def self.division
      @division ||= nil
    end
 
    def self.data
      @data ||= nil
    end
  end

The code is slightly simpler because I don't have to worry about the possibility of an undefined reference in the has_division method, but that's about it. The real advantage has to be in the stub-factor for tests.

OK… I'm easy… I can do it that way.

Refactoring Singletons Out of the Code

Monday, August 13th, 2012

Ruby

This morning I've spent a good deal of time factoring out the Singleton nature of the processing out application is doing. Rather than have singletons with thread-safe data structures, it's very simple to make a clear delineation of the loading, processing, and persistence phases of the app, and then apply thread pools where appropriate to work on the data in a parallel fashion without having to lock a single thing. It's a simple processing problem with a queue built into the thread pool doing the work.

Once all the work is done, we can then run the aggregation steps that cut across the data, in a single thread so as not to cause problems there as well. It's not rocket science, and I spent the vast majority of my time trying to unravel the code written for the atomic and immutable classes, but that's partly because of my unfamiliarity with the classes, and partly because it's a complete mess and looks nothing like the native ruby containers.

I'm glad it done - all in a branch and I've sent a pull request for debate. I think it's the right thing to do simply because of the simplification in the code and containers. The data flow is cleaner, and it'll be far clearer where to add in new functionality. It's really much better code.

UPDATE: I decided that I needed to have some sense of real measurement of the difference, so I added in a quick timing of the core of the code - the augmentation and matching. What I found was that the new, atomic and immutable-free way of doing things is significantly faster:

  processed 4605 merchants in 30310.0 ms = 6.5820 ms/merchant

for the old, atomic and immutable data way, and:

  processed 4605 merchants in 6198.0 ms = 1.3459 ms/merchant

with the cleaner, clearer, no-atomics and no-immutables way. So there's a real difference. True, the total runtime is hardly different, but that's because it's really being dictated by I/O, and we can't do a lot about that.

Made my First Gem!

Friday, August 10th, 2012

Ruby

OK, it's nothing special, but today I took on a story where I needed to pull out the Salesforce client into a gem and put them gem in our local ruby-gems server. It's pretty nice how it all works - you create a simple 'stub' gem, populate it with files and data, package it up, check it into git, and publish it. It's really pretty simple. Clearly, it's very tightly coupled to git and the ruby-gems server, but that's not too bad, considering.

The class needed very little tweaking, but it needed a bit. I was able to easily retrofit the gem into our code and it was all done in about an hour. Clearly, it'll be faster the next time, but the process is easy:

  $ bundle gem my_gem

then edit the gemspec and put in the code as needed. Build it with:

  $ rake build

and there should be a gem file ready to deploy.

I'll take better notes the next time to make sure I get all the details, but it wasn't too hard.

Ruby Gems, the Bundler, and Deploying a Jar File

Thursday, August 9th, 2012

JRuby

We ran intro a very interesting problem today with Warbler, Bundler and a bug in Hamster. To set the stage, Bundler is able to have gems specified as git/GitHub resources with the simple Gemfile line:

  gem "Hamster",
       :git => "git://github.com/harukizaemon/hamster.git"

and you can even specify a SHA for the commit point you want. Very nice.

The problem is that when you do this, Bundler places the resulting Gem into a Bundler/gems/ subdirectory of your current /gems/ directory, and that makes finding it impossible for the standard gem tools - which includes ruby itself.

What you must do is to let Bundler "fix" the GEM_PATH as soon as possible in your app. Simply require it's setup code and you are good to go:

  require 'bundler/setup'
 
  # use Hamster as normal

This is OK, if you're going to include Bundler in your deployment package and you aren't using Warbler to place everything into a Jar for Java to execute. Then we get into some nasty problems.

Our situation is that we want to deploy a single jar file. This means using Warbler to jar things up easily. Again, this doesn't sound so bad, but it's this patching issue that Bundler uses that makes things very difficult - certainly when dealing with a jar. You see, Bundler doesn't "do jars" at all. It's looking for a file system to work on, so when you try to do the same thing within a jar, you get error messages on trying to change directories, etc. It's trying to move around the file system, but a jar isn't a file system, and JRuby doesn't make it any easier, so Bundler fails.

Warbler doesn't help here because it's a problem with JRuby and Bundler and the jar.

Our solution was to build the gem from the fixes and place is in a local gem server and then reference it in the Gemfile without using the git/GitHub scheme. This places the gem in the right directory, and that means we don't need Bundler to augment the GEM_PATH and that means it all works.

I'm convinced that the real solution is to fix/patch JRuby to allow all ruby-based file and directory operations to operate on a jar. However, that's probably not the typical deployment scheme as we really hardly ever leave a WAR file unexploded, and deploying Jars is pretty limited.

Lesson: Don't use Git Gems with Bundler if you're deploying to a Jar. It's just not going to work.

Parsing a CSV File in Ruby within a JAR

Thursday, August 9th, 2012

JRuby

Today I ran into a nasty problem with a JRuby app where we're deploying the complete app as a single jar file to the server. That's a really nice idea - one atomic unit to move around, roll-back easily, all the things you'd expect… but it's got at least a few very nasty downsides, and it's got nothing to do with ruby - it's JRuby and how Java handles resources located within the jar as opposed to the filesystem outside the jar.

In short, it's not a seamless transition, and it'd be great if JRuby would handle this in all the File.open code so that we wouldn't have to. But that's probably asking a little much.

Still… to the problem at hand.

The code for reading a CSV file into a map in ruby is pretty simple:

  def self.read_csv(filename)
    res = {}
    CSV.read(filename, :headers => true).each do |rec|
      k = [rec['Size'], rec['Weight'], rec['Height']]
      res[k] = rec
    end
    res
  end

but it assumes that the file is located on the filesystem, and specifically, relative to the current directory of the running ruby VM. This isn't new, it's pretty standard, and very convenient.

But files in jar files aren't in the filesystem. They have to be located and read in as a byte stream:

  require 'java'
 
  def self.read_csv(filename)
    res = {}
 
    # get the contents of the file - no matter where it is
    contents = ''
    if File.exists?(filename)
      File.open(filename) do |file|
        contents = file.read
      end
    else
      # We appear not to have this file - but it's quite possible that
      # the file exists in the deployed jar, and if that's the case,
      # we need to access it in a more java-esque manner. This will be
      # a line at a time, but the results should be the same.
      f = java.lang.Object.new
      stream = f.java_class.class_resource_as_stream('/jar_root/' + filename)
      br = java.io.BufferedReader.new(java.io.InputStreamReader.new(stream))
      while (line = br.read_line())
        contents << "#{line}\n"
      end
      br.close()
    end
 
    # now we can take the contents of the file and process it...
    CSV.parse(contents, :headers => true).each do |rec|
      k = [rec['Size'], rec['Weight'], rec['Height']]
      res[k] = rec
    end
    res
  end

Here, the bulk of the code is about getting the file into a string that we can then parse. It first tries to see if it's on the filesystem, and if that fails, it tries the jar to see if it happens to be there. Unfortunately, it's got to be the full path to the file in the jar, and if you're using a packager that tacks something on the front, you need to be aware of this.

Not horrible, but it was an hour to figure this all out and get it nicely coded up so we didn't have too much redundant code.

Google Chrome dev 22.0.1229.0 is Out

Wednesday, August 8th, 2012

Google Chrome

This morning I noticed that Google Chrome dev 22.0.1229.0 was out, and the release notes are getting to be somewhat of a disappointment. To say they are sparse is an understatement. Take the UI change of the 'wrench' to the 'pancakes':

Chrome Releases: Dev Channel Update

that's a change, and there's got to be a reason for it, but the release notes are totally silent about it. "Read the SVN logs" is hardly an answer - at least it better not be. There should be a lot more information in the SVN logs than this, and these are higher-level issues that need addressing.

Yeah, the quality of the release notes is really slipping on this project.

Building Systems in a Service Orientated Architecture Way

Tuesday, August 7th, 2012

cubeLifeView.gif

I can really appreciate that building things in a large-scale web system means decentralization. You need to have services, and those services have to be well-defined and walled off so that changes in one service are easily adopted by another, but don't break existing apps. In short - I get it. But that doesn't mean it's easy.

The problem I'm running into today is that in order to accomplish my goals, I need other people to add things to their services so that I can get the data from them and then process it. In the past, I haven't liked this either, but it's typically been the case that I simply figured out how to update that code, and made the changes myself. Alternatively, I just didn't have the external dependencies as management wanted me to build something from the ground up.

It's not that I think these people are bad, or can't do the work. It's that they are required to make progress. It's the difference in working in a large, disparate team, and working in a small, focused team. I like the latter because it means you do very little sitting around, and a whole lot of getting stuff done.

But the flip side is that you're the key man, and there are very few people that understand the project. Like I said, I can see the flip-side.

Today has just been one of those days where I'm doing a lot of waiting for people, and I'd like to be doing a lot of coding. After all, I'm a coder, not a professional waiter.