What's it all about, Alfie?

I've been at The Shop for a couple weeks now. I barely know how to find the bathroom (joking), and I'm finding that some days the process of acclimation is easier than others. Today has been one of those days that was harder than average. I was talking to a good friend that's still in finance and he said something that so incredibly true:

The one thing I like about finance is that it teaches urgency. Now, the average developer interprets that to mean "do the bare minimum and move on", which sucks. But for those of us that care, we learn to write great code in a short period of time. We truly deliver. My fear, in a culture like [The Shop], is that after a while, people would lose their "edge", and revert to their lackadaisical way of life.

and then went on to say:

Or, you will emerge so much stronger than the rest of them, you'll own them.

This is exactly what's happened in the jobs I've had in the past even in finance. It's so easy to work 12+ hr days, write 2000-4000 lines of code a day, and take about two projects to totally change management's impression of me from the "new guy" to the "shining star". Period. I've done it so many times, it's almost a formula to me.

But I didn't want to do it here because I didn't like how the story seems to unfold. I start to isolate myself - and management agrees: Keep him locked away and producing code! It's in their best interests, and I didn't mind the peace and quiet. Plus, it meant that I didn't have to deal with a lot of people monkeying around with my code base.

Now I don't mind real help, but if you're going to do something that's not in the design of the app - don't do it. Ask. I've written about this more times than I can remember. So I decided that this time I'm going to work hard at fitting in.

I'm not so sure I'm going to be successful.

Case in point this week: let's call it The Case of Singletons and Thread Safety.

The Case of Singletons and Thread Safety

We have a process in the current application that needs to gather some statistics on the data running through it. Like a logger, it makes good design sense to have a singleton that does this. In the same way that you need one thing controlling the output on a logger to make the output look reasonable and make sense, it makes a lot of sense to have one aggregator, and then have it be responsible for serving up those aggregated values at any time.

I suppose that it's possible to have an aggregator that is simply applied to a list of things, and gathers data from each as it visits them, but that's not the same. That visitor pattern requires that all members be in memory at once, or that there's some reference to the ones that have been processed, and those that haven't - so you don't process the same one twice.

In either case, it's a lot more difficult design to understand. While it may have fewer lines, I've found that there's a significant point of diminishing returns on the simplicity angle - and it's possible to try to simplify something too much and end up with something that's small, but far from simple.

So we have this singleton. I simply put in a simple mutex at the proper points - three, I'm pretty sure, and that would provide all the thread safety we needed. They were even scoped locks, so there was no chance of deadlocks or any other issues arising from the use of the muteness. It's simple.

But not from Steve's point of view.

Steve is a guy on the team that's a nice guy. Fun-loving and quick with a joke, he's a good person to be around to keep things from becoming too serious. At the same time he's like many coding bigots I've known in that there is only one real language, and all the rest are junk, and there's only one real OS, and the rest are junk, and so on… It's sad to see someone with such great potential limit themselves so totally in life. The minute I heard him talk about technology, he marginalized himself to me as I know he's never going to be able to think outside the box that he's voluntarily placed himself in.

So Steve didn't like muteness. Thought they made the code more difficult to read.

Now recall there are only three uses of this mutex in this one class. Three. And the entire class is less than 50 lines - including whitespace. So it's not like this is going to take a lot to understand - and they are ruby muteness, well documented in the specs.

But that was too much for Steve.

So Steve and Fred spent two on replacing the mutex with atomic references. When they started down this road, I advised them why it was the way it was, and that it was a good solution. I advised them not to mess with atomics unless it's necessary because they are difficult to get right. But they didn't listen to me.

I even asked them not to do it.

But now we have atomics in the code.

I take that back… we have use of the Atomic Reference gem in the code. Within that code, as I've read, they use muteness to control access because you can't really have atomic operations in a reference counting language - look at ARC in ObjC 2.0 - can't be done. You can't do the compare and swap at the same time as handling the reference count. Period. So you can fake it, or use muteness, but you can't do it like you can when you don't have a reference counting VM.

So what's the upshot of all this? Well… I had the mutex in there to control adding to the singleton. If they made the container atomic it has to be atomic with respect to it's contents and that's a lot different that having atomic references to the container. My belief is that the Atomic gem is doing just what I was doing, but in the gem, if it's doing it right. If not, then it's controlling access to the container and isn't properly controlling inserts and removals from the container.

In short - they have in all cases a worse implementation. But they are happy. Why? Because they got to remove three lines of code in the 75 line class.

If I were the manager, I'd have a serious heart-to-heart with them about wasting time.

But I'm not the manager. I'm trying to fit in. But this kind of stuff is very hard for me because I see them making mistakes I've told them not to. They aren't listening. I can't save them, or their project. I can only step in when it's not working to say "I told you about this, and you didn't listen. Change it back and try it again. Next time, shut up and listen!"

But I won't. Because I want to fit in.

This adjustment is hard today. I had to deal with this. I had to deal with refactoring that wasn't done right. I mean really… if you're going to refactor, then bloody well do it RIGHT! You don't have classes that are 5 lines in total. Two lines are the definition! You've got a 5 line class?! That's got to be a method or function somewhere else - I guarantee it.

Like I said… it's hard. I don't know how long I'll last. I want to last. I really do. But I don't know how long I can last in a place where there are guys like this.

This morning I found a problem with the Ruby gem VCR. It turns out that if it hits a service that reports it's data as ASCII encoded, but actually sends UTF-8, then the data will be stored by VCR as ASCII, but will be unable to be read out of the cassette. Very nasty. The solution is to force VCR to record the actual bytes and base64 encode them. This is easily done with the code:

  require 'vcr'
 
  VCR.configure do |c|
    c.cassette_library_dir = 'spec/cassettes'
    c.hook_into :webmock
    c.default_cassette_options = {
      :record => (ENV["VCR_RECORD"] ? :new_episodes : :none),
      :match_requests_on => [:method, :url, :path, :host, :body]
    }
    c.ignore_localhost = true
    c.allow_http_connections_when_no_cassette = true
    c.preserve_exact_body_bytes do |http_message|
      http_message.body.encoding.name == 'ASCII-8BIT' ||
      !http_message.body.valid_encoding?
    done
  done

The code is simple - if the message body is reported from the service as ASCII-8BIT, then don't trust it. Likewise, if the body has no reported valid encoding.

This is a nice, conservative way to ensure that the cassettes get written in a way that it's ensured that we'll be able to read it back out. While it might be nice to have a human-readable version, it's not worth the problems of badly behaved services that say they are ASCII and really return UTF-8.

I'm a big fan of refactoring - heck rewriting, code. It's good because the first time you write something, you're going to make mistakes, and the more you re-work it, the better it's going to be. Or so the theory goes… and I tend to agree with that theory… so long as some kind of limits are imposed.

I'm not talking about being unreasonable here, either. If you are feeling that a class or method needs to be changed in order to make it more readable, more maintainable, or fix a serious performance issue, then by all means - do it. But if you're changing 10 lines for 10 different lines… or doing meta programming to try and make things neater or cooler, then I want to stop you right there and say What's the benefit of this change?

Is it really better? More readable? More maintainable? Or is it that you thought of another way to write the same thing, and wanted to write it just because you could? I'm running into quite a bit of that today, and because I'm the new guy, and don't want to rock the boat, I sat back and learned a lot about how to write 10 lines of ruby code about six different ways. I learned a lot, but the code didn't benefit one little bit.

The developers discussing this - there were three of us on this at that time, were talking about the relative merits of the indentation style, and the warranted differentiation of the methods of generating the methods dynamically in ruby. All might be considered valid issues, but none of this makes the code better.

None.

So after we got all this checked in and done, we started talking a little and I just brought up the point that in the past, this would have never been allowed in the places I've been. There is just too much to do to allow three high-powered developers to be tied up for more than an hour for 10 lines of code that were fine before anyone got involved. It would have been seen as a massive waste of time.

But in truth, there was benefit. The methods are smaller by a bit. I learned a lot. And the calling arguments are now more general. This is all nice, but it certainly doesn't compare favorably to the cost. But maybe that's the difference outside of Finance. Maybe the sustainability is more reasonable here. Maybe it's OK to do this and "relax", and allow everyone to recharge from time to time.

It's certainly different.

Yesterday I spent a good deal of time tracking down something that appears now to be an error in JRuby 1.6.7, which is what we're using on this code. The problem was a little nasty in that the code was generating a java exception and depending on the code we were using, it was either getting swallowed completely, or only showing the message of the exception - and not the complete stack trace as you'd expect in a Java exception.

But let's set the stage, and in doing so expose the error that led to this horrible problem. The code starts out with a seemingly innocent ruby module:

  module Bun
    private
 
    def wrap(dog)
      puts "wrapping up #{dog}"
    end
 
    def box(dogs)
      puts "boxing up #{dogs.join(', ')}"
    end
  end

The idea is that the Bun is a module the has a few methods that we'll need in some other classes, and similar to an ObjC Category, it's placed here and just re-used in the classes with the 'include' directive.

Next comes a class that uses the Bun module - just as you'd expect.

  require 'bun'
 
  class Fixer
    include Bun
 
    def fix(count)
      count.times wrap('beef dog')
    end
  end

Nothing fancy here.

The next class was intended to use this module when it had more functionality, but refactoring took that functionality out of the module and into the first class. Still, the refactoring didn't catch the difference, and so the code of the second class looks like this:

  require 'bun'
 
  class Shipper
    include Bun
 
    def ship(count)
      fix(10)
    end
  end

Technically, the following code should not work:

  require 'shipper'
 
  s = Shipper.new
  s.ship(1)

and in truth, it doesn't. But in the larger context of the app where there are many more things happening, it passed the compile and run tests only to find that there was a stack crash exception. So far, this was annoying, but that's not where it stopped.

The real code we had looked more like this:

  require 'java'
  require 'java.util.concurrent.Executors'
  require 'java.util.concurrent.TimeUnit'
 
  require 'shipper'
 
  executor = Executors.new_fixed_thread_pool(4)
  10.times do
    executor.execute do
      begin
        s = Shipper.new
        s.ship(1)
      rescue => e
        puts "caught: #{e.to_s}"
      end
    end
  end

So we have a multi-threaded app, using the Java Executors to help things out, and we got an exception that didn't get caught by the rescue clause. Very odd. Even worse, if we changed the call to use the submit method:

  10.times do
    executor.submit do
      begin
        s = Shipper.new
        s.ship(1)
      rescue => e
        puts "caught: #{e.to_s}"
      end
    end
  end

we got NOTHING. The exception was thrown, but completely swallowed up by the Executor's submit call.

After a bunch of reading, the latter at least makes sense in the context of the JVM. All exceptions will get swallowed up in the submit call, and it's up to you to keep the future, and interrogate it. So that was explained. What wasn't explained was this: When I took out the Executor, I got the message and the stack trace!

OK, to summarize: Use the Executors and you get the exception message - Don't use it and you get the message and the stack trace.

I've tried to make a simple test case, the one above, but it's properly seen as not being able to find the method on the object. But in our more complex code, that's not the case. When I talked to a friend about this, I ended up getting this tweet in response:

As I've seen this guy today on the JRuby bug tracker, I think he really knows what he's talking about. Sadly, I can't easily reproduce the problem, and it's often these complex ones that are the really tough ones to reproduce. However, it's fixed, and we have a work-around (simply comment out the Executor block) so we're able to proceed. Also, I trust that it's fixed in JRuby 1.7, so when we want to go there, that'll be waiting for us.

In retrospect, this has been a lot of work, but a real insight into ruby, JRuby, and the support community. It's been a real enlightening experience.

This morning I read an interesting fact about OS X 10.8 Mountain Lion - they removed the Web Sharing from System Preferences -> Sharing. Odd. I didn't use it a lot, but they also reset the config files for Apache, so it's something that I'm going to need to work on when I finally upgrade to 10.8.

Thankfully, the post talks about how to get all this functionality back and includes nice screen snapshots, so when I need to, I'll have a nice reference to walk me through anything I haven't seen before.

I know I need to get Apache, PHP, PostgreSQL all working together as my work on SyncKit requires it. So I'll have to factor that into the time for the upgrade.

Still planning on doing it, just not right today.

I've had a couple of really good days at The Shop. It's a really neat environment, and while I can't say it's been without friction, it's all my fault, and what I'm used to from the world of Finance. I've learned a ton about ruby, rvm, rspec, all the tools used in what I think is a traditional ruby stack. It's fun, it's very different, but what's struck me the most has been the really solid skill set of the people I'm working with.

These are some really solid guys. Amazingly so, when compared to the finance world. Maybe it's too incestutious, I don't know. But the folks I was used to working with on a daily basis in the past are not nearly as skilled in the fine art of software development as these guys. Maybe it's ruby?

There's far more refactoring. Far more iteration. Far more devotion to the best answer - and they don't sacrifice too much speed in the process. I'll admit it's not nearly the same stress level as Finance, but it's still getting done, and the business is happy, so who am I to argue with the speed of development?

It's amazing. Really. I like these guys. I really do. That's a new feeling for me.

And it's a nice feeling to be sure.

I think I'm going to really like being out of Finance for a while. Who knows… maybe it's permanent. That would certainly be fine with me.

I just noticed that Google Chrome dev 22.0.1221.0 is out and save the inclusion of a new version of the V8 javascript engine (3.12.16.0), the release notes are a little skimpy to say the least. I mean, would it really kill them to put more than 2 mins into the release notes? Are SVN commit logs really sufficient for release notes?

I'd like to think not. But I'm not in their group, so it's not my call. Still… it makes you wonder what their attention to detail level really is...

This afternoon I wanted to create a log4j-like logger for the ruby app we're working on. I was hoping to find a gem that did it - singleton, thread-safe, the works. What I found was that the default logger: "logger" is really pretty close, and it's easy to make it what we need.

First, it's thread-safe. Doesn't say that in the docs, but the code has synchronize blocks on a mutex, and that appears to be working, so while I can't guarantee that it's implemented correctly, it appears that it is, and that's good enough for now.

Second, it's not a singleton, but that's easy to fix:

  require "singleton"
  require "logger"
  require "app_config"
 
  class AppLog
    include SIngleton
 
    def initialize()
      # get the location to log to and the level
      where = AppConfig.application.log_to || "stout"
      level = AppConfig.application.log_level || "WARN"
      # now make the logger for our singleton
      case
      when where == "stdout"
        self.log = Logger.new($stdout)
      when where == "stderr"
        self.log = Logger.new($stderr)
      else
        self.log = Logger.new(where)
      end
      # now set the log level
      case
        where level == "FATAL"
          self.log.level = Logger::FATAL
        where level == "ERROR"
          self.log.level = Logger::ERROR
        where level == "WARN"
          self.log.level = Logger::WARN
        where level == "INFO"
          self.log.level = Logger::INFO
        where level == "DEBUG"
          self.log.level = Logger::DEBUG
      end
    end
  end

This can then be placed in all our code and we get singleton-based logging for next to nothing. Very nice. At the same time, we can control the logging location and level very easily in the app config file.

I then took this and integrated it into the code I was working on for matching up the demand data with the merchants. It's something that we really need to do to productionize our app, and this is an important step.

This morning I wanted to correct the CVS import problem that I didn't have the correct entries in the Authors file for the import, and so I wanted to update the converted repos to have my name and email. It's a little thing, but it's something that's been nagging at me for a while.

Anyway, the solution is pretty simple. Get into the directory of the git repo, and issue the once command:

  $ git filter-branch --env-filter 'GIT_AUTHOR_NAME="Bob Beaty"; \
      GIT_AUTHOR_EMAIL="drbob@themanfromspud.com"; \
      GIT_COMMITTER_NAME="Bob Beaty"; \
      GIT_COMMITTER_EMAIL="drbob@themanfromspud.com";' HEAD

and then I needed to do the following to push these changes to the server:

  $ git pull
  Merge made by the 'recursive' strategy.
  $ git pull
  Already up-to-date.
  $ git push
  Counting objects: 1591, done.
  Delta compression using up to 8 threads.
  Compressing objects: 100% (400/400), done.
  Writing objects: 100% (1490/1490), 614.05 KiB, done.
  Total 1490 (delta 1103), reused 1263 (delta 1089)
  To git@git.themanfromspud.com:CKit.git
     b5ffd5d..ef9724e  master -> master

Then it's all done. Nice and clean. Pretty slick!

[7/31] UPDATE: I noticed that the repos aren't universally fixed. In fact, there are still commits that have the old username. I did it again, and it's fixed - locally, but when I do the pull in preparation for the push, it "unfixed" several, and I can't figure out the reason. At this point, it's not an issue for me because most of the commits are right, and that's fine for me. Close enough.

I've been working with Jeff on some changes to the threading model of our app, and I have to hand it to Ruby/JRuby - the exposure of Java's Executors is really very nice:

  require 'java'
  java_import 'java.util.concurrent.Executors'
 
  executor = Executors.new_fixed_thread_pool(10)
  work.each do
    executor.execute do
      begin
        # do all my work here
      rescue => e
        puts e.backtrace.join("\n")
      end
    end
  end

and the beauty of the design is that we now have 10 threads for any number of tasks that need to be done in 'work', and the Executors will handle all the processing and thread management and all the stuff you'd once have to have done manually - it's now free.

This isn't new to Java, but it's really nice that it's sitting in JRuby for free. Very nice.

The problem is that when we get a Java stack trace, it's next to impossible to figure out what's going on. Face it, ruby is dynamic, so the Java classes that are created have a passing resemblance to the original ruby class, but they have been munged into what's needed by a non-dynamic language: Java. It's possible to get some idea, but it's not a lot, and today it's turned out to be not nearly enough to figure out what's happening.

Again, the best debugger is a sharp developer, and that's what we have to go with, but it's just too bad that the JRuby crew didn't foresee this and make something that can take the Java stack trace and turn it into what a ruby developer's stack trace might look like.

Now that would be neat.