What's it all about, Alfie?

This afternoon I wanted to create a log4j-like logger for the ruby app we're working on. I was hoping to find a gem that did it - singleton, thread-safe, the works. What I found was that the default logger: "logger" is really pretty close, and it's easy to make it what we need.

First, it's thread-safe. Doesn't say that in the docs, but the code has synchronize blocks on a mutex, and that appears to be working, so while I can't guarantee that it's implemented correctly, it appears that it is, and that's good enough for now.

Second, it's not a singleton, but that's easy to fix:

  require "singleton"
  require "logger"
  require "app_config"
 
  class AppLog
    include SIngleton
 
    def initialize()
      # get the location to log to and the level
      where = AppConfig.application.log_to || "stout"
      level = AppConfig.application.log_level || "WARN"
      # now make the logger for our singleton
      case
      when where == "stdout"
        self.log = Logger.new($stdout)
      when where == "stderr"
        self.log = Logger.new($stderr)
      else
        self.log = Logger.new(where)
      end
      # now set the log level
      case
        where level == "FATAL"
          self.log.level = Logger::FATAL
        where level == "ERROR"
          self.log.level = Logger::ERROR
        where level == "WARN"
          self.log.level = Logger::WARN
        where level == "INFO"
          self.log.level = Logger::INFO
        where level == "DEBUG"
          self.log.level = Logger::DEBUG
      end
    end
  end

This can then be placed in all our code and we get singleton-based logging for next to nothing. Very nice. At the same time, we can control the logging location and level very easily in the app config file.

I then took this and integrated it into the code I was working on for matching up the demand data with the merchants. It's something that we really need to do to productionize our app, and this is an important step.

This morning I wanted to correct the CVS import problem that I didn't have the correct entries in the Authors file for the import, and so I wanted to update the converted repos to have my name and email. It's a little thing, but it's something that's been nagging at me for a while.

Anyway, the solution is pretty simple. Get into the directory of the git repo, and issue the once command:

  $ git filter-branch --env-filter 'GIT_AUTHOR_NAME="Bob Beaty"; \
      GIT_AUTHOR_EMAIL="drbob@themanfromspud.com"; \
      GIT_COMMITTER_NAME="Bob Beaty"; \
      GIT_COMMITTER_EMAIL="drbob@themanfromspud.com";' HEAD

and then I needed to do the following to push these changes to the server:

  $ git pull
  Merge made by the 'recursive' strategy.
  $ git pull
  Already up-to-date.
  $ git push
  Counting objects: 1591, done.
  Delta compression using up to 8 threads.
  Compressing objects: 100% (400/400), done.
  Writing objects: 100% (1490/1490), 614.05 KiB, done.
  Total 1490 (delta 1103), reused 1263 (delta 1089)
  To git@git.themanfromspud.com:CKit.git
     b5ffd5d..ef9724e  master -> master

Then it's all done. Nice and clean. Pretty slick!

[7/31] UPDATE: I noticed that the repos aren't universally fixed. In fact, there are still commits that have the old username. I did it again, and it's fixed - locally, but when I do the pull in preparation for the push, it "unfixed" several, and I can't figure out the reason. At this point, it's not an issue for me because most of the commits are right, and that's fine for me. Close enough.

I've been working with Jeff on some changes to the threading model of our app, and I have to hand it to Ruby/JRuby - the exposure of Java's Executors is really very nice:

  require 'java'
  java_import 'java.util.concurrent.Executors'
 
  executor = Executors.new_fixed_thread_pool(10)
  work.each do
    executor.execute do
      begin
        # do all my work here
      rescue => e
        puts e.backtrace.join("\n")
      end
    end
  end

and the beauty of the design is that we now have 10 threads for any number of tasks that need to be done in 'work', and the Executors will handle all the processing and thread management and all the stuff you'd once have to have done manually - it's now free.

This isn't new to Java, but it's really nice that it's sitting in JRuby for free. Very nice.

The problem is that when we get a Java stack trace, it's next to impossible to figure out what's going on. Face it, ruby is dynamic, so the Java classes that are created have a passing resemblance to the original ruby class, but they have been munged into what's needed by a non-dynamic language: Java. It's possible to get some idea, but it's not a lot, and today it's turned out to be not nearly enough to figure out what's happening.

Again, the best debugger is a sharp developer, and that's what we have to go with, but it's just too bad that the JRuby crew didn't foresee this and make something that can take the Java stack trace and turn it into what a ruby developer's stack trace might look like.

Now that would be neat.

This morning I was looking at some JSON output from a service and realized that my hand-cleaning of JSON was really not a good use of my time. So I googled JSON beautification BBEdit and found this:

  #!/usr/bin/python
  import fileinput
  import json
  if __name__ == "__main__":
    jsonStr = ''
    for a_line in fileinput.input():
      jsonStr = jsonStr + ' ' + a_line.strip()
    jsonObj = json.loads(jsonStr)
    print json.dumps(jsonObj, sort_keys=True, indent=2)

It's a little python script that I can put into ~/Library/Application Support/BBEdit/Text Filters/, call it PrettyJSON.py and then restart BBEdit and get a wonderful reformatter in Text -> Apply Text Filter -> PrettyJSON.

It's impressive. Fast too. I had a pretty big JSON file and it made it look nice and readable in under a second. Very impressive. This is certainly something to keep around.

Today I had a need to install ruby 1.8.7 to work in a repo here at The Shop. Normally, rvm would make this easy by allowing me to simply load it up:

  $ rvm install ree-1.8.7-2010.02

but the upshot of this is a problem in the compilation:

  Error running './installer -a /Users/me/.rvm/rubies/
  ree-1.8.7-2010.02 --no-tcmalloc --dont-install-useful-gems -c',
  please read /Users/me/.rvm/log/ree-1.8.7-2010.02/install.log

  There has been an error trying to run the see installer,
  Halting the installation.

After a ton of trying things, the solution is to use Homebrew's gcc 4.2 as it's not based on the LLVM, as the one that ships with Xcode 4.2 is. It turns out that there's a problem with the compile of some of the ree-1.8.7-2010.02 components when using the LLVM-based compiler.

OK, so we need to get the see what we're trying to get:

  $ brew search gcc
  homebrew/dupes/apple-gcc42
  (among other things)

If the homebrew/dupes doesn't appear, then you don't need this step, but I did:

  $ brew tap homebrew/dupes

and then we can install GCC 4.2 without the LLVM:

  $ brew tap apple-gcc42

This places gcc-4.2 in the path if you have Homebrew set up properly.

All this was to be able to properly compile ree-1.8.7-2010.02, so now we can do that:

  $ CC=gcc-4.2 rvm install ree-1.8.7-2010.02 --with-gcc=gcc-4.2

At this point, it will all install nicely and you'll have ree-1.8.7-2010.02 in your RVM set-up. Lots of work, but all worth it.

Today I did quite a bit of work on refactoring the existing code which represents a simple buffering storage container for the output of the pipeline in our project. The original version had a lot of things in it that had no real need - the output was ordered (why?), there were a lot of methods and nested classes that really didn't need to be there. So Jeff and I started peeling the layers back and cleaning up a lot of this code.

What we came up with was a very nice design. The store had two basic methods, and two optional ones, if it happened to maintain a state:

  class Store
    attr_accessor :backend
 
    def store(block)
      self.backend.store(block)
    end
 
    def bulk_store(blocks)
      self.backend.bulk_store(blocks)
    end
 
    def clear
      self.backend.clear if self.backend.respond_to?(:clear)
    end
 
    def flush
      self.backend.flush if self.backend.respond_to?(:flush)
    end
  end

With this, we can have a very simple store class and then use it as the "backend" to the main store that just happens to be a Singleton (not shown):

  class StreamStore
    attr_accessor :stream
 
    def initialize(stream = $stdout)
      self.stream = stream
    end
 
    def store(block)
      write_one(block)
    end
 
    def bulk_store(blocks)
      blocks.each do |b|
        write_one(b)
      end
    end
 
    private
 
    def write_one(b)
      stream.puts b.to_s
    end
  end

All this is nice, but it doesn't address the buffering, and that's where I've come to find one of Ruby's really nice features: Modules. You can write a module - like a class, and then you can augment an existing instance with this behavior without having to include this in the class inheritance.

Methods will be added, ivars as well - it's just like the categories in ObjC - but maybe even one step better: in Ruby you can add this to a single instance, and in ObjC you have to add it to the class everywhere in the runtime.

I'm really impressed. This is really powerful, but at the same time, it's possible to have one instance of an object have entirely different methods and ivars than another. One might think this is really cool, but from a production stand-point, it's also very dangerous. This kind of power can be abused so easily.

This afternoon it's been tough as Jeff and I have been doing some pair work on one component of the system we're building at The Shop. It's a small ruby app, and the Ruby Way is to have lots and tests, but the problem is it's impossible to have complete test coverage, and bad tests are a nightmare. For example, we're trying to make the code functional in nature - immutable objects, queues and threads to balance the load, and that kind of stuff. All good goals, to be sure, and worthy of the effort. But as I've always said, there's no such thing as a complete test suite, and the ones I've seen are just complex enough to make the addition of a simple, single feature so daunting that it almost defies inclusion.

There was a Java Swing app, for instance, that every time I added a new attribute to the system I had to spend more than 5x the time of adding the feature to updating the tests. This is not bad, if it's only 10 sec to add a feature, but when it's an hour to add a feature, and 5 hours of work to update the tests, it gets out of hand. And since there's no system that analyzes the code and generates the test code, the tests are just as fallible as the code itself.

After all, who's writing the tests?

Do we need tests on the tests? Meta-Tests?

It can get out of hand very soon. And if it's just a simple set of sanity checks, then that's one thing, but when it includes end-to-end tests and complex integration tests, it's going to get out of hand very quickly.

Such is the case, I fear, for the project I'm on now.

Today we were trying to figure out why the continuous-integration server was failing on the tests. All the tests ran just fine on our dev machines, but on the CI server, they failed. All the time. Why?

Jeff and I started digging into this and found that the CI server was running the rspec tests in a specific order - as opposed to letting rspec run them as it saw fit out of the complete directory. We suspected the order of the tests was the problem, and sure enough it was. This is clearly hysteresis, or test pollution at work. Something in one of these tests was setting state that wasn't getting cleared properly, and then another test was coming in and it was failing. Reverse the order and both tests worked just fine.

So what was it?

We spent about an hour at this - getting it down to about 10 lines of code split across two methods. This is the flip-side of ruby… 10 lines is a method, but 10 lines is an hour long mystery. Finally, I realized that one test was capturing stdout and the other was using it, and if the non-capture went first, then it's singleton was set up for non-capture on stdout, and the test failed. Reverse them, and all was well.

Singletons. Not great for functional coding because of just this kind of stuff. Also, spec should have some way of resetting the complete environment for each spec (test) file. That would be far preferable, but I can see why they do it this way - write good code, and you don't have this problem, but it allows you to have tests that "build" on one another. Makes sense.

So tests - not the code itself, was the reason for a half day of work. This is nasty. Very nasty in my book. Maybe I'm still decompressing from the world of finance, but that's a long time to be stuck on something that adds no value to the project. Still… I have to be patient and learn why they do things this way, as there's bound to be some reason. Gotta be.

I've been working with a co-worker on some code for the project I'm on, and I'm learning a lot about the Ruby way of doing things. It's a lot different from the typical Java/C++ coding I've been used to over the years. In short - things are meant to be simple, but there can be a lot of them. So if you have a class that holds Requests, and queries them and loads them, that's really three classes in Ruby. You want to isolate the roles and responsibilities to one per thing.

This is very different from C++ where the overhead of a class is 50 lines, and then there's the implementation file as well. It'd end up being more boilerplate than functional code if it was written with the guidelines of methods less than 10 lines and classes less than a page. It's just an entirely different world.

And I'm enjoying learning this process and style very much. I can see the goals, and the reasons for it. It's pretty simple, really. But it's hard to do in a lot of other languages. Hard because of the overhead or boilerplate needed for those languages. Ruby is nice in that regard, and I'm excited about learning more.

It's a process, and I'm happy to be on the path.

I've been using tcsh for years - back to my first MacBook when it was the default for new accounts on Mac OS X. It's been a long time. But there are just too many things that are looking for bash as the shell, and I'm just tired of having to fix things, or re-do things, or just have to roll it all myself, so I switched.

This makes a lot of sense as it's the same shell I'm going to be using on Ubuntu, and all the other platforms I'll ever use. It was a pain to get going as I needed to convert my login scripts to bash, and that took a little bit of fiddling, but it's now worth it. I'm converted, and things are running smoothly again.

Great. Now I can get to running Ruby from RVM. Very important for the work I'm doing.

Turns out The Shop is a big fan of RVM, and it's ilk, where you have to install something small, and then have that install all the things you need. It's a nice setup, and I can see why they do it - so I'm going to remove the JRuby Framework and try using RVM.

The setup is simple:

  $ curl -L https://get.rvm.io | bash -s stable --ruby

and once that's all done, we can just install jruby with:

  $ source .rvm/scripts/rvm
  $ rvm install jruby-1.6.7

to get jruby 1.6.7 (the preferred version at this time) for the box.

It all installs in !/.rvm and that's really very convenient, and you can put simple .rvmrc files in project directories to force the use of a certain version, etc. Also, if that version isn't installed, then rvm will go get it for you to make sure you can run what you need.

Not bad. I can go with this.