What's it all about, Alfie?

This morning I ran into a problem with the ruby re-write of the summary script that I've been working on since late yesterday. The error was occurring on the relatively simple code:

  File.open(src) do |line|
    if line =~ / BEGIN /
    # …
    end
  end

right in the open() method call. The error was cryptic:

  summary:48 in 'block in process_pipeline' invalid byte sequence in UTF-8 (ArgumentError)
      from summary:47" in 'each'

I had to hit google, as it was clear to me there were odd characters in the file, and while I might like to fix that - the key to the previous version was to include the '-a' option to grep to make sure that it looked at the files as binary files. But what would do the trick here?

Turns out there's a StackOverflow answer for that:

  File.open(src, 'r:iso-8859-1') do |line|
    if line =~ / BEGIN /
    # …
    end
  end

which instructs the IO object to read the file with the ISO-8859-1 encoding and that did the trick. No other changes were necessary!

Sweet trick to know.

It's almost the leet version number, but not quite. This morning Google Chrome dev 25.0.1337.0 was released with virtually nonexistent release notes. I guess the same maintainer is back at it. Just gotta wonder what they are thinking. I can write a few paragraphs a day on what I'm doing and they can even be bothered to make decent release notes?

Googlers. Go figure.

With all the efficiency changes in the code recently, the next most inefficient was the bash script that analyzed the run logs and generated some summary statistics for us to view in the morning. When I first created this script, it wasn't all that complex, and the logs weren't nearly as big as they are now. I used the typical assortment of scripting tools: grep, sed, awk, but the problem was that as I added things to the summary script the time it took to execute was getting longer and longer. To the point that it took several minutes to run it on the main pipeline process. That's no good.

So I wanted to rewrite it in something that was going to be fast, but process the file only once. The problem with the current version isn't that it's using bash, or grep, it's that the files are hundreds of megabytes and we need to scan them a dozen or more times for the data. What we needed was to make a single-pass summary script, and that's not happening with bash and grep.

So what then?

Ruby popped to mind, but given that we're using jruby, there's a significant startup penalty. But maybe we can force it to use a compiled MRI ruby in the deployment environments, and that will speed up the loading.

C, C++ both seemed like ideal candidates, but then I know how the rest of the guys in the group would react, and it's just not worth it.

So Ruby is is.

This shouldn't take long, as most of this is pretty simple stuff for ruby. Let's get going...

In a continuing effort to make the code more efficient and really, just plain faster, this afternoon I've been working with a teammate to update CouchRest, our ruby client to Couch, to handle server-side updates. Couch allows server-side updates - you basically write a javascript function that takes the document and the request and you can update the document as you see fit, and return something to the caller.

It's not bad, really. It should certainly make the updates a ton faster as right now we're reading, updating and writing back the complete document for a very small change - in one case just a single field. This is really where the document database falls down, and you long for a SQL statement where you can simply UPDATE and be done with it.

Still, it's nice to be able to write:

  function(doc, req) {
    var ans = false;
    var fld = 'lead_assignment';
    if (doc) {
      doc[fld] = JSON.parse(req.body);
      and = true;
    }
    return [doc, JSON.stringify({'updated': ans})];
  }

and be able to make a change with:

  def update_merchant_assignment(division, sf_id, stuff)
    return nil if (id = get_latest_results_docID(division, sf_id)).nil?
    Database.update('merchant_updater/add_assignment', :id => id, :body => stuff)
  end

It really simplifies the code, and it certainly cuts the bytes moved for an update way down. I'm hoping it's enough… we'll have to see how it goes.

This afternoon I found a set of tests in the code that weren't implemented, and should be. They were stubbed out by one of the guys on the Team, but he didn't have time to really implement them, he just wanted to remind himself that we needed these tests, and so he stubbed them out and then went on about what he needed to do. I noticed them, and decided that since I didn't have a lot going on at this time, it made sense to give it a go at implementing the tests.

After all, I know there's a lot about rspec I don't know, and this would be a nice way to learn about it.

Some of the tests were really pretty clear: make sure the main routine returns something. So how to do that - simply? Well… we can always stub out the methods with simple return values and then just make sure that you get back what you expect.

  require 'lead_assignment/entry_point'
 
  describe LeadAssignment::EntryPoint do
    describe ".unassign_and_assign" do
     before(:each) do
       class FauxRepClient
         def get_reps(division)
           []
         end
       end
       LeadAssignment::EntryPoint.stub(:reps_client => FauxRepClient.new)
       LeadAssignment::EntryPoint.stub(:fetch_accounts => [])
       LeadAssignment::EntryPoint.stub(:add_accounts => [])
     end

on this first test, I noticed that I wanted to start all my tests with this little configuration, so it was easy to put into a before() block and then it was going to be done before each test within the scope of the enclosing describe. That's nice to remember.

Then I can do the end-to-end test:

     it "returns a result" do
       LeadAssignment::EntryPoint.stub(:sink => nil)
       results = LeadAssignment::EntryPoint.unassign_and_reassign('cleveland')
       results.should == { :unassignments => [], :assignments => [] }
     end

and my first test is done!

I learned a lot about testing writing that, and it was going to pay off as I did the others. They all looked about the same - you stub out certain methods, run certain sections, and then check the output. Not bad at all. Just need to be careful and methodical about what you're doing.

Then I came to a more challenging problem: I needed to know when a specific instance was being called, and with a certain set of arguments. That's not too bad - you have to make the instance, and then you can use it:

     it "writes the results to salesforce" do
       class FauxSFClient
         def bulk_store(accounts)
           nil
         end
       end
       my_store = FauxSFStore.new
       LeadAssignment::EntryPoint.stub(:send_assignments_to_sf? => true)
       LeadAssignment::EntryPoint.stub(:store => my_store)
       my_store.should_receive(:bulk_store).with([]).exactly(2).times
 
       LeadAssignment::EntryPoint.unassign_and_reassign('cleveland')
     end

and again, this works great! I like that I can specify the args to the method, and the instance doesn't need to be exactly what's in the code - my faux class is just as good for this as anything.

The final trick I learned has to do with passing blocks to methods and testing the contents of those blocks. You can't actually tell what's in a block, but you can evaluate it and then test that value:

     it "writes log messages properly for summary script" do
       LeadAssignment::EntryPoint.stub(:send_assignments_to_sf? => false)
       LeadAssignment::EntryPoint.stub(:unassign => [])
       LeadAssignment::EntryPoint.stub(:assign => [])
       LeadAssignment::EntryPoint.stub(:sink => nil)
 
       QuantumLead::Application.logger.should_receive(:info) do |method, &block|
         method.should == "LeadAssignment::EntryPoint.unassign_and_reassign"
         block.call.should == "Starting LeadAssignment in cleveland"
       end
 
       LeadAssignment::EntryPoint.unassign_and_reassign('cleveland')
     end

and you can have as many of those logger checking blocks as you believe you will have calls to the logger. It's pretty nice.

With all this in place, I was able to whip up the necessary tests for the code in short order. Pretty nice tools.

We are going to be writing a project in Clojure, and I've been told I'm going to be on this project, so I need to get up to speed on Clojure. Thankfully, I had the Pragmatic Programmer's book on Programming Clojure, so I've been re-reading it these past few days.

It's interestingly a lot like Erlang, which I had to learn at a previous position for some work I was doing there. In fact, that's when I picked up the Clojure book, but I also had to pick up the Erlang book, and since there was work to do on that, it took top priority for my time, so I learned it more completely.

Now I'm looking at Clojure, and it's really a lot like Erlang in a bunch of ways. The record data structure, the gen_server ideas, and the functional code all make for a pretty quick learn for me. There's still a lot I need to read - I'm only about one-third of the way through, but it's something that I've been working on these days. Just to be ready.

I've got an old install of Erlang, and Clojure, and I need to update them for work I'm about to do, but I don't feel like doing the same old installs… I'm going to try Homebrew for package management because it's working so well for my work laptop. So I cleared out the old installs of these packages, which was a chore - basically, complete directories in /usr/local/ or in ~/Library/ and I also took the time to clean up my .bash_login and .bashrc because they had additions for the PATH, and even DYLD_LIBRARY_PATH that needed to be removed as well.

Once I had the old stuff removed, I installed Homebrew with the simple command:

  $ ruby -e "$(curl -fsSkL raw.github.com/mxcl/homebrew/go)"

and it installed itself just fine. Having done this already, I knew what to expect, but the next steps were really nice:

  $ brew install erlang
  $ brew install leiningen

where Leiningen is the Clojure package manager and REPL tool. Once I had this installed, I noticed that /usr/local/bin wasn't early enough in my PATH to make sure that I picked off the Homebrew commands and not the native OS X commands.

Actually, Homebrew itself, pointed this out to me. Nice installer. So I had to track down where this was happening. Interestingly enough, I wasn't adding /usr/local/bin/ to my path - the system was! In /etc/paths there's a list of paths to add:

  /usr/bin
  /bin
  /usr/sbin
  /sbin
  /usr/local/bin

and I needed to change it to:

  /usr/local/bin
  /usr/bin
  /bin
  /usr/sbin
  /sbin

to get things right. Now, I had the PATH right, and both Erlang (erl) and Clojure (lein repl) started up just fine. Sounds like a no-op, but I'm on more recent versions, and for the work I'm about to get into, switching to Leiningen is a must.

But I didn't stop there… Oh no… I kept on cleaning things up. I don't even have Qt on this box, but that was in my PATH, and the Groovy, and a lot of other things that I don't have and don't need. All cleaned up.

By now my .bash_login and .bashrc are looking almost spartan. But then I was wondering about PostgreSQL. Was that on Homebrew? Would it work with Apache2 on my OS X box? Since I had the time, I decided to try it. So once again, I followed the simple steps to migrate from one package to the other:

Step 1 - make a complete backup. I went into my home directory and backed up everything in my server:

  $ /usr/local/pgsql/bin/pg_dumpall -U _postgres -o > pgbackup

Step 2 - shut down the old version, and remove it's startup script from the system-wide install location:

  $ sudo launchctl unload \
      /Library/LaunchDaemons/org.postgresql.postgres.plist
  $ sudo rm /Library/LaunchDaemons/org.postgresql.postgres.plist

Step 3 - remove the old install and all the symlinks in the man pages and the /usr/local/bin directory that I did myself with this install:

  $ cd /usr/local
  $ sudo rm -rf pgsql-9.1

there was some shell magic in the removal of the links - like an ls piped into a grep for 'pgsql' and then removing them. Nothing fancy, but it took a little time.

Now that the old PostgreSQL install was really gone - even from my .bash_login and .bashrc, I was ready to install the PostgreSQL from Homebrew. One of the reasons was that it was 9.2.1 and the previous install was 9.1.

Step 4 - install PostgreSQL:

  $ brew install postgresql

Step 5 - create initial database for Homebrew PostgreSQL install:

  $ initdb /usr/local/var/postgres -E utf8

Step 6 - set it to start on my login, and start it now:

  $ mkdir -p ~/Library/LaunchAgents
  $ cp /usr/local/Cellar/postgresql/9.2.1/homebrew.mxcl.postgresql.plist \
        ~/Library/LaunchAgents/
  $ launchctl load -w ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist

Step 7 - reload databases from initial dump:

  $ psql -d template1 -f ~/pgbackup

At this point, we can run psql and access the databases, and I'm sure I'm up and running, I needed to see about the integration with Apache2 - I have to have that working for some projects I've done, and are still working on.

Step 8 - activating PHP in Apache2 config on my box. Edit the file: /etc/apache2/httpd.conf and uncomment the line that looks like:

  LoadModule php5_module libexec/apache2/libphp5.so

and restart apache:

  $ sudo apachectl graceful

Step 9 - make my ~/Sites directory executable again. Create the file /etc/apache2/users/drbob.conf:

  <Directory "/Users/drbob/Sites/">
    Options FollowSymLinks Indexes MultiViews
    AllowOverride None
    Order allow,deny
    Allow from all
  </Directory>

and at this point, I had the quite familiar PHP info screen up and my simple database accessing page worked like a charm. I'd successfully completed the migration!

But was I done? No!

I've been running boost 1.49.0 for a while, and I like that I figured out how to do universal binaries of the libraries. Very nice. But then I checked Homebrew:

  $ brew info boost
  boost: stable 1.52.0 (bottled), HEAD
  www.boost.org
  Not installed
  github.com/mxcl/homebrew/commits/master/Library/Formula/boost.rb
  ==> Options
  --with-icu
    Build regexp engine with icu support
  --without-python
    Build without Python
  --with-mpi
    Enable MPI support
  --universal
    Build a universal binary

so I could update to boost 1.52.0 and get the same universal binaries without missing a beat! This might be really nice. So I removed my own boost install:

  $ cd /usr/local/include
  $ sudo rm -rf boost
  $ cd /usr/local/lib
  $ sudo rm -rf libboost_*

and then I installed boost from Homebrew:

  $ brew install boost --universal

Odd… I got:

  ...failed updating 22 targets...
  ...skipped 12 targets...
  ...updated 10743 targets...
 
  READ THIS: github.com/mxcl/homebrew/wiki/troubleshooting
 
  These open issues may also help:
    github.com/mxcl/homebrew/issues/14749

The hint was to run brew doctor and correct all the errors. Well… I had a lot of them - all from my manual boost and gfortran installs. So I ditched my old gfortran install and cleaned up all the problems and then I re-ran the install:

  /usr/local/Cellar/boost/1.52.0: 9086 files, 362M, built in 6.1 minutes

When I looked in /usr/local/include and /usr/local/lib I see all the boost code, and I even checked that I got the universal binaries:

  $ file /usr/local/lib/libboost_wave-mt.dylib 
  /usr/local/lib/libboost_wave-mt.dylib: Mach-O universal binary with 2 architectures
  /usr/local/lib/libboost_wave-mt.dylib (for architecture i386): Mach-O dynamically
    linked shared library i386
  /usr/local/lib/libboost_wave-mt.dylib (for architecture x86_64): Mach-O
    64-bit dynamically linked shared library x86_64

Excellent!

Now to put back gfortran from Homebrew:

  $ brew install gfortran

and after cleaning up more cruft from the old gfortran install, it installed and worked just fine!

I have now successfully removed all the third-party builds I once used with Homebrew. This is amazing stuff.

This morning I noticed that Google Chrome dev 25.0.1323.1 was out and the release notes have returned to a more spartan style. Yes, people can read SVN logs, but that's not the point - really, is it? If you take the time to make a blog post about the release, you should be able to make release notes that say more than "Read the SVN logs".

Still… progress is nice to see continuing.

Today I've been having real troubles trying to get our system scaled up to new hardware in the datacenter. Moving from hosts in Amazon EC2 to machines in our own datacenter are a big step in the right direction, but going from iffy bandwidth in EC2 to solid switches in the datacenter and 2 cores to 24 are something we simply have to do in order to scale up to handling the global data load that we're going to need.

The problems I've run into today are all about loading. I've already added a queueing system to the CouchDB interface, in order to minimize the connections to the Couch server so as not to overload it - there were times when the Couch server would simply shutdown it's socket listener, and therefore refuse all updates sent from the process. Not good.

There's also a lot of problems today with Salesforce. I don't think they expected the kind of loads we're delivering. This morning, at 3:00 am, Salesforce called the support guys at the shop and told them that a process was bringing one of their sandbox clusters to it's knees, and that, it turns out, is but one of four boxes we need to bring online. They're having a hard time handling this much - I can't imagine what's going to happen when we try to really ramp it up.

I'm starting to have real concerns about both these endpoints of the project. I know there's a lot of data getting moved around, and while we're able to handle it, it's these endpoints that are having the hardest time. I've talked with the project manager and the technical manager about this, and I think we need to start thinking about potential bail-out scenarios.

It's certainly possible to read from Salesforce. We're planning on re-doing the complete demand system, so there shouldn't be an issue there. Persistence? Go back to MySQL or PostgreSQL and store it all in tables. The data is getting pretty nicely finalized, so a nice schema should be able to be made. Save it all in a SQL database, make a simple service that reads/caches this data and offers it up to the clients, and the pages already built on the existing data sources are pretty easily modified.

Odd to think that I was looking at MySQL before Couch popped up. Funny thing is, I have a strong feeling that Salesforce can come up with hardware that makes the grade, but it's the bugs in Couch that worry me the most. You just gotta wonder if it's in the Erlang code, or on the boxes, or what.

So many unknowns, but it's clear that we can't scale to one nice box - there's no way we're going to make it work globally without some serious changes.

I've been fighting a problem with our CouchDB installation using the CouchRest interface for ruby/jruby. What I'm seeing is that when I have a lot of updates to CouchDB from the REST API in CouchRest, we start to get socket connection errors to the CouchDB server. I've gone through a lot of different configurations of open file handles on both boxes, and nothing seems to really 'fix' the problem.

So what I wanted to do was to make a queueing database class. Currently, we have something that abstracts away the CouchDB-centric features, and adds in necessary metadata so that we can more easily track all the data held in CouchDB. This is a nice start, in that I only really needed to add to the bulk of the code was a simple flush method:

  Database.flush

where I initially started with it being implemented as:

  def flush
    nil
  end

At this point, I was free to do just about anything with the Database class - as long as I didn't change the public API. What I came up with is this:

What I really like about this is that I can use it either way - as a cacheing/flushing endpoint, or as a straight pass-through with thread-local database connections. This means that I can start with a simple config file setting:

  data_to: 'database'
  writers: 3

which will give me three writer threads, as specified in the start method argument. Then, when I fix the CouchDB issues, I can switch to the simpler, thread-local connection storage with the code:

  Database.start(config.writers) if config.writers > 0

The call to flush is a no-op if we didn't start anything, so there's no harm in always calling it at the end of the process. Pretty nice. I've easily verified that this gets me what I need, and it's just a matter of how 'throttling' I want to be with the writing of the data to CouchDB. But I tell you this… I'm closer to MySQL than ever before - just because of this. There's no reason in the world to put up with this kind of software if it can't do the job.