What's it all about, Alfie?

Today, while working on a bunch of other stuff, I got a chat from the data scientist in Palo Alto asking me why the service tag changed to primary_service, and what else changed. Turns out that the folks I'm getting the data from has decided to change the data format I was getting from them, without telling me about it. Nice.

The problem wasn't that it only took me an hour or so to change the ETL on the data to get it back to where I needed it, it was that they decided to change it in ways that weren't required without telling us anything about it! I know this is often the case, but we're supposed to be on the same team, and they just didn't bother telling us.

I know the point of the ETL code is to fix that up, and allow us to insulate ourselves from changes on their part, but it's also the timing. These guys could not possibly have been in my position yesterday or they'd never have done it. It's one of those cardinal rules of making an API - once you have it, you stick to it! If you have to change it, it's because it doesn't work, and then you open a discussion - even a simple "heads-up" email, to make sure that people are aware of the change that's coming, and why you had to make it.

It's not horrible, and I liked that I got new data that made the results better, but it was the way they did it that was annoying. I fear that this isn't the last such occurrence from these guys.

I just got out of a meeting, and I have to say that I'm really quite shocked to see someone at The Shop so incredibly inflexible as Jim. This is the first time I've met Jim, and while I haven't been impressed by his emails, and specifically his responsiveness to my requests, I always wrote it off as him being exceptionally busy. Busy people have a lot going on - I get it. But today was the first face-to-face meeting, and during that, Jim really blew my socks off.

Jim is vending some demand data, and it makes perfect sense to everyone but Jim to have him include the current inventory in his service. After all, if there's some change in inventory, it's going to really effect what we need to get because excess inventory will shrink demand, and a big run on something will push it up. But not in Jim's mind.

No, Jim was making a theoretical demand calculation, and in that, I can see a value. But he's also including the inventory on hand at the time of his run - which is once a week, so it's going to make it harder for us to know what current inventory-effecting events to include in the demand we get from Jim. We have to look at the time of his run, and then look at all events to see how they might effect the demand.

It'd be far simpler to have Jim vend the raw demand and then we can always correct it by the current inventory. Much simpler. No problems in potentially lost transactions. Better.

But not to Jim. Holy Cow! It's been a long time since I've seen a professional programmer push back so hard on a feature. It was speculated that it was all about the level-of-service issue… after all, hitting Jim's service once a week and saving the results is far easier on Jim than having to make sure it's up all the time and vending the right data. I just can't get over the sheer laziness of this guy.

He's a developer. What's he got to do but develop? How hard is it to make it work 24x7? Not too hard, I've done it in a bunch of technologies over the years. Stop being so bloody lazy, Jim!

Holy Cow!

This morning I'm riding in on the train, bus, and walking and got a text from Joseph thanking me for waking him up this morning:

I want to reiterate how much I appreciate you waking me up. I always feel like I seem really pissed at you because I'm late, but you've saved me a lot of trouble many times. So thank you.

I sent back:

Son, and I mean that with all the love in my heart, it's my pleasure, and I totally understand. You're very welcome.

Now don't get me wrong, I'm glad to wake him up. And deal with waking the girls, and their nasty morning attitudes. It needs to be done, but I have to say that it gets more than a little annoying to see and hear nothing but grief from them. Which brings me to this morning's realization.

Life is not an "even" game. Not even close.

The most contented people I know have given up the idea of "fair", and accepted that people will hurt them far more than they will, in turn, hurt others. It's not the "takers" that are happy, it's the "givers". Even when it's not a gift, these "givers" accept that they cannot expect fairness and equality in the world. They accept that life is what it is, and they can only do what they can. And just move on.

My Mom is a classic example. When I was growing up, my sisters picked on her all the time, and were merciless. It was bad. I wished they would stop, but I was the younger brother, and they did it to me as well. I could do nothing to stop it, and it always amazed me that my Mom didn't seem to mind.

Years later, I found that it was breaking her heart, and it made me all the more sad for her. I didn't understand at the time that she was doing the best she could, and rather than confront them - which would be pointless, they'd just make fun of her more, she chose to accept that life wasn't fair, and that her children were going to be making fun of her for this period in her, and their, lives.

Today I'm faced with the same thing. The kids are intelligent, and so are vicious in their attacks on Liza and me. It's to the point that I just don't speak in the car. When I forget, and say even the most unassuming thing, it's "source" for the girls. I'll hear about it for the rest of the day. I want to yell at them, but that'll just be more fodder for the guns.

While some will say "Fight back" - that just teaches them that anger and attacks are "OK". They know - deep down, that what they are doing is not nice. It's fun, but it's not nice. And they don't do it to most people - just family. So some day, like my sisters, they'll stop, and while they'll probably never apologize, they'll stop, and that will be good enough.

I can't even the score with them. That's not right, and it'll never happen. So I need to accept that life, certainly my life, at 50 is not really about fair. It's about making it through the day with as few a set of negatives as possible. Try to find something enjoyable about the day before I go home and have to listen to them.

I'm going to work really hard on this… I think it's really the key to long-term happiness.

This morning I noticed that Google Chrome dev 22.0.1229.6 was out, but without any release notes. None. Zip. I did notice that the "Stable" channel of Google Chrome is on 21.x, so it's possible that the 'dev' channel is about to just to 23.x, and this is a last little patch before the big jump, but that's just a guess.

Something changed, and it's nothing I can visually see, so I'm guessing it's not a major deal, but it's interesting that they've gone from "sparse" to "nonexistent" release notes. Maybe they're cutting back at Google? Can't afford the digital ink? Crazy.

Today we've had a lot of really great work done. Things are starting to accelerate, in part because we're down a few guys due to a clojure-rewrite of the main app, but also because we're focused on making real and significant progress on the project. It's been all-(remaining)-hands on deck, and it's been really nice.

I've done a lot of email to folks about things we need their assistance with. In some cases it's just knowledge transfer, in others it's liable to be a little more complex. Still, we need to get all these issues figured out before we release this to the users, and since someone has to do it, it might as well be me.

At the same time I've made some nice progress on closing some stories for the calculations done and logged in the app. Nothing amazing, but due to the optimistic nature of 90% of the ruby code I've seen, it really means tracking down the potential errors and making the code more robust as well as implementing the actual features. It's nice to see things really start to take the shape of a completed app - especially when there are so many people working on it, as it more often resembles swiss cheese than a working app.

I still would like to get some answers from a few folks, but that's what I'm waiting on now - external dependencies. I've never been very patient, but I'm going to try hard to last this out with a nice, and gentle attitude. Personal growth time.

I've been talking to a few friends about editors on Mac OS X, and one editor that's getting a lot of press lately is Sublime Text 2. It's supposed to be a variant of Vim, and that's always a plus - it's even got a vintage mode to make it mimic the command and insert modes of Vim. It's got an amazing speed profile as well as an incredible Python-based extension library. And there are a ton of things that have been written for it.

So this morning I decided to give it a try. I'm glad I did. It's an amazing platform for coding. I can "open" a project - basically a directory that is my git repo, and then with the "Find Anything" command, I can load up a file very simply. Amazingly so. This is my major concern about BBEdit - opening files in a ruby environment is really a pain - as there are a lot of them.

There are so many nice things that it's just too hard to know where to start. There's a GitHub package for dealing with Gists - one of my favorite places to store snippets of code. This guy even gets the file names so you don't have to know the Gist number beforehand.

There is a legacy Vim mode, and a legacy Ex mode. Both are really nice for old school Vim users like me. Also, the app works on Windows, Linux, and OS X.

I'm looking forward to hammering on this and seeing where it takes me. Very excited.

Just finessed refactoring a class where I had used class variables. The code was very simple:

  class Demand
    def self.load(division, data)
      @@division = division
      @@data     = data
    end
 
    def self.reset
      @@division = nil
      @@data     = nil
    end
 
    def self.has_division(division)
      defined?(@@division) && !@@division.nil? && (@@division == division)
    end
  end

there's more to the class, but this points out that I'm using the class variables, and the API is really pretty simple. When I posted this as a pull request on GitHub, a guy on the team pointed out that others are going to want to modify the class variables to class methods as they are less dangerous.

I don't see the danger, when the API is the same, but I can understand the advantage of the class method approach - it makes it possible to stub() out the method in an spec test - which is nice as it simplifies the testing.

So I went back in and modified the code to look like:

  class Demand
    def self.load(division, data)
      @division = division
      @data     = data
    end
 
    def self.reset
      @division = nil
      @data     = nil
    end
 
    def self.has_division(division)
      !self.division.nil? && (self.division == division)
    end
 
    private
 
    def self.division
      @division ||= nil
    end
 
    def self.data
      @data ||= nil
    end
  end

The code is slightly simpler because I don't have to worry about the possibility of an undefined reference in the has_division method, but that's about it. The real advantage has to be in the stub-factor for tests.

OK… I'm easy… I can do it that way.

This morning I've spent a good deal of time factoring out the Singleton nature of the processing out application is doing. Rather than have singletons with thread-safe data structures, it's very simple to make a clear delineation of the loading, processing, and persistence phases of the app, and then apply thread pools where appropriate to work on the data in a parallel fashion without having to lock a single thing. It's a simple processing problem with a queue built into the thread pool doing the work.

Once all the work is done, we can then run the aggregation steps that cut across the data, in a single thread so as not to cause problems there as well. It's not rocket science, and I spent the vast majority of my time trying to unravel the code written for the atomic and immutable classes, but that's partly because of my unfamiliarity with the classes, and partly because it's a complete mess and looks nothing like the native ruby containers.

I'm glad it done - all in a branch and I've sent a pull request for debate. I think it's the right thing to do simply because of the simplification in the code and containers. The data flow is cleaner, and it'll be far clearer where to add in new functionality. It's really much better code.

UPDATE: I decided that I needed to have some sense of real measurement of the difference, so I added in a quick timing of the core of the code - the augmentation and matching. What I found was that the new, atomic and immutable-free way of doing things is significantly faster:

  processed 4605 merchants in 30310.0 ms = 6.5820 ms/merchant

for the old, atomic and immutable data way, and:

  processed 4605 merchants in 6198.0 ms = 1.3459 ms/merchant

with the cleaner, clearer, no-atomics and no-immutables way. So there's a real difference. True, the total runtime is hardly different, but that's because it's really being dictated by I/O, and we can't do a lot about that.

OK, it's nothing special, but today I took on a story where I needed to pull out the Salesforce client into a gem and put them gem in our local ruby-gems server. It's pretty nice how it all works - you create a simple 'stub' gem, populate it with files and data, package it up, check it into git, and publish it. It's really pretty simple. Clearly, it's very tightly coupled to git and the ruby-gems server, but that's not too bad, considering.

The class needed very little tweaking, but it needed a bit. I was able to easily retrofit the gem into our code and it was all done in about an hour. Clearly, it'll be faster the next time, but the process is easy:

  $ bundle gem my_gem

then edit the gemspec and put in the code as needed. Build it with:

  $ rake build

and there should be a gem file ready to deploy.

I'll take better notes the next time to make sure I get all the details, but it wasn't too hard.

We ran intro a very interesting problem today with Warbler, Bundler and a bug in Hamster. To set the stage, Bundler is able to have gems specified as git/GitHub resources with the simple Gemfile line:

  gem "Hamster",
       :git => "git://github.com/harukizaemon/hamster.git"

and you can even specify a SHA for the commit point you want. Very nice.

The problem is that when you do this, Bundler places the resulting Gem into a Bundler/gems/ subdirectory of your current /gems/ directory, and that makes finding it impossible for the standard gem tools - which includes ruby itself.

What you must do is to let Bundler "fix" the GEM_PATH as soon as possible in your app. Simply require it's setup code and you are good to go:

  require 'bundler/setup'
 
  # use Hamster as normal

This is OK, if you're going to include Bundler in your deployment package and you aren't using Warbler to place everything into a Jar for Java to execute. Then we get into some nasty problems.

Our situation is that we want to deploy a single jar file. This means using Warbler to jar things up easily. Again, this doesn't sound so bad, but it's this patching issue that Bundler uses that makes things very difficult - certainly when dealing with a jar. You see, Bundler doesn't "do jars" at all. It's looking for a file system to work on, so when you try to do the same thing within a jar, you get error messages on trying to change directories, etc. It's trying to move around the file system, but a jar isn't a file system, and JRuby doesn't make it any easier, so Bundler fails.

Warbler doesn't help here because it's a problem with JRuby and Bundler and the jar.

Our solution was to build the gem from the fixes and place is in a local gem server and then reference it in the Gemfile without using the git/GitHub scheme. This places the gem in the right directory, and that means we don't need Bundler to augment the GEM_PATH and that means it all works.

I'm convinced that the real solution is to fix/patch JRuby to allow all ruby-based file and directory operations to operate on a jar. However, that's probably not the typical deployment scheme as we really hardly ever leave a WAR file unexploded, and deploying Jars is pretty limited.

Lesson: Don't use Git Gems with Bundler if you're deploying to a Jar. It's just not going to work.