What's it all about, Alfie?

I just got out of a meeting, and I have to say that I'm really quite shocked to see someone at The Shop so incredibly inflexible as Jim. This is the first time I've met Jim, and while I haven't been impressed by his emails, and specifically his responsiveness to my requests, I always wrote it off as him being exceptionally busy. Busy people have a lot going on - I get it. But today was the first face-to-face meeting, and during that, Jim really blew my socks off.

Jim is vending some demand data, and it makes perfect sense to everyone but Jim to have him include the current inventory in his service. After all, if there's some change in inventory, it's going to really effect what we need to get because excess inventory will shrink demand, and a big run on something will push it up. But not in Jim's mind.

No, Jim was making a theoretical demand calculation, and in that, I can see a value. But he's also including the inventory on hand at the time of his run - which is once a week, so it's going to make it harder for us to know what current inventory-effecting events to include in the demand we get from Jim. We have to look at the time of his run, and then look at all events to see how they might effect the demand.

It'd be far simpler to have Jim vend the raw demand and then we can always correct it by the current inventory. Much simpler. No problems in potentially lost transactions. Better.

But not to Jim. Holy Cow! It's been a long time since I've seen a professional programmer push back so hard on a feature. It was speculated that it was all about the level-of-service issue… after all, hitting Jim's service once a week and saving the results is far easier on Jim than having to make sure it's up all the time and vending the right data. I just can't get over the sheer laziness of this guy.

He's a developer. What's he got to do but develop? How hard is it to make it work 24x7? Not too hard, I've done it in a bunch of technologies over the years. Stop being so bloody lazy, Jim!

Holy Cow!

Today we've had a lot of really great work done. Things are starting to accelerate, in part because we're down a few guys due to a clojure-rewrite of the main app, but also because we're focused on making real and significant progress on the project. It's been all-(remaining)-hands on deck, and it's been really nice.

I've done a lot of email to folks about things we need their assistance with. In some cases it's just knowledge transfer, in others it's liable to be a little more complex. Still, we need to get all these issues figured out before we release this to the users, and since someone has to do it, it might as well be me.

At the same time I've made some nice progress on closing some stories for the calculations done and logged in the app. Nothing amazing, but due to the optimistic nature of 90% of the ruby code I've seen, it really means tracking down the potential errors and making the code more robust as well as implementing the actual features. It's nice to see things really start to take the shape of a completed app - especially when there are so many people working on it, as it more often resembles swiss cheese than a working app.

I still would like to get some answers from a few folks, but that's what I'm waiting on now - external dependencies. I've never been very patient, but I'm going to try hard to last this out with a nice, and gentle attitude. Personal growth time.

Just finessed refactoring a class where I had used class variables. The code was very simple:

  class Demand
    def self.load(division, data)
      @@division = division
      @@data     = data
    end
 
    def self.reset
      @@division = nil
      @@data     = nil
    end
 
    def self.has_division(division)
      defined?(@@division) && !@@division.nil? && (@@division == division)
    end
  end

there's more to the class, but this points out that I'm using the class variables, and the API is really pretty simple. When I posted this as a pull request on GitHub, a guy on the team pointed out that others are going to want to modify the class variables to class methods as they are less dangerous.

I don't see the danger, when the API is the same, but I can understand the advantage of the class method approach - it makes it possible to stub() out the method in an spec test - which is nice as it simplifies the testing.

So I went back in and modified the code to look like:

  class Demand
    def self.load(division, data)
      @division = division
      @data     = data
    end
 
    def self.reset
      @division = nil
      @data     = nil
    end
 
    def self.has_division(division)
      !self.division.nil? && (self.division == division)
    end
 
    private
 
    def self.division
      @division ||= nil
    end
 
    def self.data
      @data ||= nil
    end
  end

The code is slightly simpler because I don't have to worry about the possibility of an undefined reference in the has_division method, but that's about it. The real advantage has to be in the stub-factor for tests.

OK… I'm easy… I can do it that way.

This morning I've spent a good deal of time factoring out the Singleton nature of the processing out application is doing. Rather than have singletons with thread-safe data structures, it's very simple to make a clear delineation of the loading, processing, and persistence phases of the app, and then apply thread pools where appropriate to work on the data in a parallel fashion without having to lock a single thing. It's a simple processing problem with a queue built into the thread pool doing the work.

Once all the work is done, we can then run the aggregation steps that cut across the data, in a single thread so as not to cause problems there as well. It's not rocket science, and I spent the vast majority of my time trying to unravel the code written for the atomic and immutable classes, but that's partly because of my unfamiliarity with the classes, and partly because it's a complete mess and looks nothing like the native ruby containers.

I'm glad it done - all in a branch and I've sent a pull request for debate. I think it's the right thing to do simply because of the simplification in the code and containers. The data flow is cleaner, and it'll be far clearer where to add in new functionality. It's really much better code.

UPDATE: I decided that I needed to have some sense of real measurement of the difference, so I added in a quick timing of the core of the code - the augmentation and matching. What I found was that the new, atomic and immutable-free way of doing things is significantly faster:

  processed 4605 merchants in 30310.0 ms = 6.5820 ms/merchant

for the old, atomic and immutable data way, and:

  processed 4605 merchants in 6198.0 ms = 1.3459 ms/merchant

with the cleaner, clearer, no-atomics and no-immutables way. So there's a real difference. True, the total runtime is hardly different, but that's because it's really being dictated by I/O, and we can't do a lot about that.

OK, it's nothing special, but today I took on a story where I needed to pull out the Salesforce client into a gem and put them gem in our local ruby-gems server. It's pretty nice how it all works - you create a simple 'stub' gem, populate it with files and data, package it up, check it into git, and publish it. It's really pretty simple. Clearly, it's very tightly coupled to git and the ruby-gems server, but that's not too bad, considering.

The class needed very little tweaking, but it needed a bit. I was able to easily retrofit the gem into our code and it was all done in about an hour. Clearly, it'll be faster the next time, but the process is easy:

  $ bundle gem my_gem

then edit the gemspec and put in the code as needed. Build it with:

  $ rake build

and there should be a gem file ready to deploy.

I'll take better notes the next time to make sure I get all the details, but it wasn't too hard.

We ran intro a very interesting problem today with Warbler, Bundler and a bug in Hamster. To set the stage, Bundler is able to have gems specified as git/GitHub resources with the simple Gemfile line:

  gem "Hamster",
       :git => "git://github.com/harukizaemon/hamster.git"

and you can even specify a SHA for the commit point you want. Very nice.

The problem is that when you do this, Bundler places the resulting Gem into a Bundler/gems/ subdirectory of your current /gems/ directory, and that makes finding it impossible for the standard gem tools - which includes ruby itself.

What you must do is to let Bundler "fix" the GEM_PATH as soon as possible in your app. Simply require it's setup code and you are good to go:

  require 'bundler/setup'
 
  # use Hamster as normal

This is OK, if you're going to include Bundler in your deployment package and you aren't using Warbler to place everything into a Jar for Java to execute. Then we get into some nasty problems.

Our situation is that we want to deploy a single jar file. This means using Warbler to jar things up easily. Again, this doesn't sound so bad, but it's this patching issue that Bundler uses that makes things very difficult - certainly when dealing with a jar. You see, Bundler doesn't "do jars" at all. It's looking for a file system to work on, so when you try to do the same thing within a jar, you get error messages on trying to change directories, etc. It's trying to move around the file system, but a jar isn't a file system, and JRuby doesn't make it any easier, so Bundler fails.

Warbler doesn't help here because it's a problem with JRuby and Bundler and the jar.

Our solution was to build the gem from the fixes and place is in a local gem server and then reference it in the Gemfile without using the git/GitHub scheme. This places the gem in the right directory, and that means we don't need Bundler to augment the GEM_PATH and that means it all works.

I'm convinced that the real solution is to fix/patch JRuby to allow all ruby-based file and directory operations to operate on a jar. However, that's probably not the typical deployment scheme as we really hardly ever leave a WAR file unexploded, and deploying Jars is pretty limited.

Lesson: Don't use Git Gems with Bundler if you're deploying to a Jar. It's just not going to work.

I can really appreciate that building things in a large-scale web system means decentralization. You need to have services, and those services have to be well-defined and walled off so that changes in one service are easily adopted by another, but don't break existing apps. In short - I get it. But that doesn't mean it's easy.

The problem I'm running into today is that in order to accomplish my goals, I need other people to add things to their services so that I can get the data from them and then process it. In the past, I haven't liked this either, but it's typically been the case that I simply figured out how to update that code, and made the changes myself. Alternatively, I just didn't have the external dependencies as management wanted me to build something from the ground up.

It's not that I think these people are bad, or can't do the work. It's that they are required to make progress. It's the difference in working in a large, disparate team, and working in a small, focused team. I like the latter because it means you do very little sitting around, and a whole lot of getting stuff done.

But the flip side is that you're the key man, and there are very few people that understand the project. Like I said, I can see the flip-side.

Today has just been one of those days where I'm doing a lot of waiting for people, and I'd like to be doing a lot of coding. After all, I'm a coder, not a professional waiter.

It really is nice to be busy. Maybe this is feeding my finance problem, and I'd be better off de-toxing all the way, but it really is nice to be responding to requests quickly and turning around answers to folks quickly and efficiently. It gives me a sense of accomplishment, and that's really nice when you're still The New Guy at a place. Feeling like you make a difference, like you matter - that's nice stuff.

It's all about how we feel about ourselves, isn't it? No matter what you're doing, where you're doing it… if you feel good about yourself it's meaningful work. Good enough.

I've been at The Shop for a couple weeks now. I barely know how to find the bathroom (joking), and I'm finding that some days the process of acclimation is easier than others. Today has been one of those days that was harder than average. I was talking to a good friend that's still in finance and he said something that so incredibly true:

The one thing I like about finance is that it teaches urgency. Now, the average developer interprets that to mean "do the bare minimum and move on", which sucks. But for those of us that care, we learn to write great code in a short period of time. We truly deliver. My fear, in a culture like [The Shop], is that after a while, people would lose their "edge", and revert to their lackadaisical way of life.

and then went on to say:

Or, you will emerge so much stronger than the rest of them, you'll own them.

This is exactly what's happened in the jobs I've had in the past even in finance. It's so easy to work 12+ hr days, write 2000-4000 lines of code a day, and take about two projects to totally change management's impression of me from the "new guy" to the "shining star". Period. I've done it so many times, it's almost a formula to me.

But I didn't want to do it here because I didn't like how the story seems to unfold. I start to isolate myself - and management agrees: Keep him locked away and producing code! It's in their best interests, and I didn't mind the peace and quiet. Plus, it meant that I didn't have to deal with a lot of people monkeying around with my code base.

Now I don't mind real help, but if you're going to do something that's not in the design of the app - don't do it. Ask. I've written about this more times than I can remember. So I decided that this time I'm going to work hard at fitting in.

I'm not so sure I'm going to be successful.

Case in point this week: let's call it The Case of Singletons and Thread Safety.

The Case of Singletons and Thread Safety

We have a process in the current application that needs to gather some statistics on the data running through it. Like a logger, it makes good design sense to have a singleton that does this. In the same way that you need one thing controlling the output on a logger to make the output look reasonable and make sense, it makes a lot of sense to have one aggregator, and then have it be responsible for serving up those aggregated values at any time.

I suppose that it's possible to have an aggregator that is simply applied to a list of things, and gathers data from each as it visits them, but that's not the same. That visitor pattern requires that all members be in memory at once, or that there's some reference to the ones that have been processed, and those that haven't - so you don't process the same one twice.

In either case, it's a lot more difficult design to understand. While it may have fewer lines, I've found that there's a significant point of diminishing returns on the simplicity angle - and it's possible to try to simplify something too much and end up with something that's small, but far from simple.

So we have this singleton. I simply put in a simple mutex at the proper points - three, I'm pretty sure, and that would provide all the thread safety we needed. They were even scoped locks, so there was no chance of deadlocks or any other issues arising from the use of the muteness. It's simple.

But not from Steve's point of view.

Steve is a guy on the team that's a nice guy. Fun-loving and quick with a joke, he's a good person to be around to keep things from becoming too serious. At the same time he's like many coding bigots I've known in that there is only one real language, and all the rest are junk, and there's only one real OS, and the rest are junk, and so on… It's sad to see someone with such great potential limit themselves so totally in life. The minute I heard him talk about technology, he marginalized himself to me as I know he's never going to be able to think outside the box that he's voluntarily placed himself in.

So Steve didn't like muteness. Thought they made the code more difficult to read.

Now recall there are only three uses of this mutex in this one class. Three. And the entire class is less than 50 lines - including whitespace. So it's not like this is going to take a lot to understand - and they are ruby muteness, well documented in the specs.

But that was too much for Steve.

So Steve and Fred spent two on replacing the mutex with atomic references. When they started down this road, I advised them why it was the way it was, and that it was a good solution. I advised them not to mess with atomics unless it's necessary because they are difficult to get right. But they didn't listen to me.

I even asked them not to do it.

But now we have atomics in the code.

I take that back… we have use of the Atomic Reference gem in the code. Within that code, as I've read, they use muteness to control access because you can't really have atomic operations in a reference counting language - look at ARC in ObjC 2.0 - can't be done. You can't do the compare and swap at the same time as handling the reference count. Period. So you can fake it, or use muteness, but you can't do it like you can when you don't have a reference counting VM.

So what's the upshot of all this? Well… I had the mutex in there to control adding to the singleton. If they made the container atomic it has to be atomic with respect to it's contents and that's a lot different that having atomic references to the container. My belief is that the Atomic gem is doing just what I was doing, but in the gem, if it's doing it right. If not, then it's controlling access to the container and isn't properly controlling inserts and removals from the container.

In short - they have in all cases a worse implementation. But they are happy. Why? Because they got to remove three lines of code in the 75 line class.

If I were the manager, I'd have a serious heart-to-heart with them about wasting time.

But I'm not the manager. I'm trying to fit in. But this kind of stuff is very hard for me because I see them making mistakes I've told them not to. They aren't listening. I can't save them, or their project. I can only step in when it's not working to say "I told you about this, and you didn't listen. Change it back and try it again. Next time, shut up and listen!"

But I won't. Because I want to fit in.

This adjustment is hard today. I had to deal with this. I had to deal with refactoring that wasn't done right. I mean really… if you're going to refactor, then bloody well do it RIGHT! You don't have classes that are 5 lines in total. Two lines are the definition! You've got a 5 line class?! That's got to be a method or function somewhere else - I guarantee it.

Like I said… it's hard. I don't know how long I'll last. I want to last. I really do. But I don't know how long I can last in a place where there are guys like this.

I was fixing a production problem that the Unix Admins hadn't remembered to fix after the last kernel update, and my Boss asked to see me. I asked him to hold on a second to make sure that I had the production systems back up and emails sent, and then I followed him. Right past his office.

I was going to say "You takin' me out back to whack me?", but as we turned the corner, it was clear that, in a way, that's exactly what he was doing. I was lead into the room with an HR lady with a folder, and it was clear - I was being fired! Wild.

The reason was lame - I was unhappy. He elaborated to say that eventually I was going to leave or they were going to have to fire me, so do it now and get it over with. Seems a little wild given that less than two weeks ago, I got a $75,000.00 bonus check. Hmmm… could it be that I was looking and they found out?

Seems so.

I had a lunch meeting with a friend - totally unrelated to the search, but it could have look suspicious. Also, a clumsy recruiter called me at work - bone-head move, but it could have exposed me. Also, there's the very real possibility that someone blabbed on me. The Chicago finance industry is tight-nit, and it's happened to me before, so I know it can happen.

So it goes. Thankfully, I have a few things to look at, and I'm sure I'll be employed soon, and that's OK with me. This was not the place for me long-term. Too many people that are in the wrong jobs.