What's it all about, Alfie?

Well… here again today I got the hint that maybe I'm living right. Maybe. The manager that was asking for 8 million log statements came by and thanked me for looking into the problem and getting it all solved. Sure, this included a lot of work from others as well, but I was happy that he made the effort.

Really happy. This is nice. Have a nice place to work, and decent people to work with. I just need to be a little firmer about pushing back. I don't think I'm doing anyone any favors by getting upset and grumbling through something.

I get to leave with a smile on my face.

This might sound harsh, and I'll concede that it's certainly something that's bugged me at more than one job over the years, but it's also why I end up being highly paid and one of the more influential people at those jobs as well. It's really simple.

Thinking has become a lost art.

It's like everything else in life, you have to work at it to get good at it. You can't do it just a bit, and think you're 'good to go' for a while. Nope. Thinking is something you have to really work on, and really apply on a daily basis to keep sharp on the skill.

Why is it so important? Because no matter what you do in life, thinking will make you more efficient and better at it. Period.

You want to manage technical people? You need to think. In something that happened to be recently, a manager asked to look at all the outcomes of a matching. But didn't even stop to do the simple math that what he was asking for was in excess of 8 million matches. Even at a few minutes per match, that's way way too long to really be a useful strategy. Had I followed his advice, we'd have a giant Excel spread sheet that no one could open, and we'd still be going through the data.

Think, people! Think!

Don't just act… Think!

I recently read a story about two very famous original Unix architects that were sitting at a workstation (because they were scarce) and when there was a bug, one started hitting the debugger, and the other stopped doing anything -- he started thinking. His approach was that if you understood the system, then debugging was in your head. Thinking was more powerful than doing.

This isn't an invitation to analysis paralysis - that's taking something to an extreme. But you can't shoot first and ask questions later, either. Certainly when you're doing a lot of technical managing.

So let's all practice this a little more, OK?

Now this is probably not amazing news to long-time Ruby developers, but the simplicity and ease with which it's possible to make thread-local variables in Ruby is simply shocking. The ruby developers just don't know how good they have it. This morning I was looking at a threading problem with the CouchRest CouchDB client, and realized that it's not thread-safe. This isn't really shocking as thread-safety is something I've come to realize is not standard in Ruby libraries.

Still… I was determined to make it work.

What seemed logical was to have multiple database connections - one per thread, and then just have thread-local database connections. As the threads are born, they need a connection, create it, and use it. When they die, the connections are cleaned up automatically. Sweet. Simple.

But I know that dealing with thread-local storage in pthreads is not horrible, but it's certainly not "easy". I dig into the Ruby support for thread-local storage, and it's trivial:

  Thread.current[:foo] = {}

This creates the tagged thread-local variable foo. How simple! This is something that I never expected to see. Never! So why are these Ruby guys having so much trouble with thread-safety? I have no idea.

With the tools I've seen in Ruby, there's really no excuse for why there aren't more thread-safe libraries. All the tools are there - they're just unused. Lazy coders.

This morning I read that the latest Gimp on OS X had a native GUI build - very exciting! They didn't abandon the X11 GUI, they simply made a version that looks the same, but has the native Cocoa GUI so that you don't have to be running X11 in order to use it.

This is really neat! I'm not a huge Gimp user any more, as I've found some really great replacements, but it's got a lot of power, and some of the provided scripts are simply amazing. So to see them port the GUI to Cocoa is a really nice feat. I'm very impressed with these guys.

Well done!

I know it needed to be done, and I know someone had to do it, but that doesn't make it any more fun than it already isn't. Digging in the data to find out why we aren't matching up merchants and demand in Philadelphia is no fun at all. It's a lot of data with very little pattern to it, and a whole lot of problems. But that's what I was doing for several hours today. The pain and suffering was really compounded by the complete lack of real thought put into this as we headed into the meeting.

Overall, I was very angry at myself for not pushing back. I should have. I know that now, but it's that blasted work ethic thing that causes me to say "Yes" when I should be saying "Hold on a sec…"

The problem is that we're getting demand and merchants to fulfill that demand, and the assumed "match" here should be very high. Why? Just because. They really have never looked at this and have no idea what it should be, but "instinctually" many think that it should be "very high" - like 90%. So when the first runs came out with it being more like 50%, they wanted to know why. I totally agree.

But where we diverge is in the How?

Once program manager suggested I send a 5000+ line Excel file where the hierarchical JSON data was somehow magically "flattened" to make it easy for anyone to look at the data and determine why the merchants weren't matching. Thankfully, I had the strength of character to say "No" to that.

But that wasn't until after I heard another request to log all 5000+ merchants against all 1500+ demands - yielding more than 8 million log lines. Nope. That's just plain silly.

I wanted to get to the bottom of this to be sure, but I wanted to do it in a way that makes at least a little sense. And looking at 8 million log lines isn't it. So I started building a few CouchDB temporary views and started looking for what wasn't being matched and why.

Turns out there were two major issues: the demand wasn't supplying sufficient 'service' coverage to pin enough merchants, and the zip codes on the merchant data was really pretty horrible. Call me 'Indy' on this - it only took me about 90 mins to find these reasons and document them up for the group. Nice. Clean. Efficient.

Nothing like looking at 8 million log lines.

My manager at The Shop forwarded something he read today about an interesting little package called drip. It's essentially something that will pre-launch a JVM instance for a set of command options so that repeated calls of the same command will not have the overhead of starting a JVM. This would be ideal for JRuby - if it was supported. Unfortunately, this is a bash script and you need to be able to hook it into the code you're using - or at least replace the java command with drip.

Sadly, JRuby hides the java command, so we can't easily replace it. The JRuby team will have to make it possible with some kind of environment variable, etc. Given that Java on most platforms I've used starts pretty nicely, I'm guessing they are not going to spend a lot of time with this. It's really a bad problem on Mac OS X, but maybe that will be changing in the future. Who knows?

But it's certainly something to hang onto and maybe it'll be useful in the future.

This morning I noticed that Google Chrome dev 23.0.1246.0 was out, and the release notes are back to being a little more descriptive. This guy a new V8 javascript engine as well as a new cut of WebKit and addresses a few other bugs. Nice to see the notes are back to what they used to be, and that we're still seeing progress. Nice.

I've been building a little visualization based on ZingChart, and using jQuery as the general-purpose support library which I've come to realize is simply amazing, and deserving of the acceptance it's received. The code to do a simple AJAX request is amazingly simple:

  $.getJSON(url, function(data) {
    parse_series(data);
    redraw();
  });

where the code in the block is an anonymous function that runs on the return data which will be called data in the block. Very slick. I remember what I had to do to get this working in Chrome a few years ago… it was not pretty.

But there's so much more as well. I think I'm a convert. If I end up doing more javascript pages, I'm going to use jQuery and even if I end up using Google Visualizations, I'll use jQuery to get the data and do some of the nastier stuff.

Today I've started working with a new javascript charting library called ZingChart. It looks pretty complete, and certainly has a lot of charting styles, plus it's touting it's speed with large data sets, so that's nice. But as with anything like this, the learning curve is steep because the graph has to be configured in javascript, and it's always a lot of work to know not only How to do something, but also What's possible to be done.

Compounding this is that the documentation and the examples have contradictory information - I mean not even close! I spent several hours on trying to just get a nice pie chart going. But in the end, all the knobs are there and we don't have to worry that there's something we simply cannot do.

As an interesting note, they include the complete jQuery library in their 'resources' directory. It's something I've read a lot about, and it was available when I was doing my last web development work a few lifetimes ago, but we didn't use it because the Google Visualization Toolkit was more than enough for what we needed, and the need just didn't arise.

But it's interesting that it's all there. Very nice looking graphs.

Odd that they don't have a table. Guess I'll have to fall back to Google Visualizations for that.

I've been at The Shop about six weeks, and I thought this morning it was about time I updated my resumes and my web site. It's something I like to keep up to date because going back and figuring out what you did when, and for whom, is one of those things that you have to do when you're looking for a new position, and that's typically a pretty stressful time. There's no need to add to the stress by figuring all this out when it's possible to do it a little at a time, when it's all fresh in your memory.

So I took the time to update my three resumes, and the four web pages as well as my WordPress 'About me' page. I had updated LinkedIn a few weeks ago as I was still getting a lot of interest on that front, and it was only fair to shut that down by posting the new position.

It's really quite nice using the tools I have for this: Pages and Coda, as I really only need to write the text once and then I can just move it from document to page to document and simply reformat it. Sweet tools. But hey… it's a Mac - it's gotta be sweet!

All done and it only took me a few minutes for everything. Love to stay up to date.