What's it all about, Alfie?

I'm sitting here this morning smiling quietly to myself after working a while on some problems. I realize I'm smiling because the keyboard is just such a wonderful piece of work - the Apple (small) wireless keyboard and wireless mouse, to be precise. These are simply the best input devices that I've ever used. The keyboard is low to the desktop, responsive, small, and without a cord. The trackpad is large, the gestures wonderfully thought-out. It's amazing what I can do so effortlessly on this machine.

Then there's the machine. I'll admit I'm not a fan of the glossy Apple displays. I never have been. The HP 30" is really the best I've used in a long while. I think Apple could make a nice 30" display, but they don't want to, and there's no making them do something they don't want to do.

But back to the laptops… they are, without a doubt, the best in the industry - and have been for a long while. Really, the only solid competition was the IBM ThinkPad, and IBM sold that to Levono, and the quality has become so very ordinary now. Nothing to write home about.

But great hardware gets out of your way. You stop thinking about how to do something, and focus on the doing. In some cases, it even enables you to see things that you might have not seen - large displays with more code in them really are very beneficial.

So I have to tip my hat to the folks at The Shop that see this as well, and outfit the developers this way. It's really quite amazing.

Well… here again today I got the hint that maybe I'm living right. Maybe. The manager that was asking for 8 million log statements came by and thanked me for looking into the problem and getting it all solved. Sure, this included a lot of work from others as well, but I was happy that he made the effort.

Really happy. This is nice. Have a nice place to work, and decent people to work with. I just need to be a little firmer about pushing back. I don't think I'm doing anyone any favors by getting upset and grumbling through something.

I get to leave with a smile on my face.

This might sound harsh, and I'll concede that it's certainly something that's bugged me at more than one job over the years, but it's also why I end up being highly paid and one of the more influential people at those jobs as well. It's really simple.

Thinking has become a lost art.

It's like everything else in life, you have to work at it to get good at it. You can't do it just a bit, and think you're 'good to go' for a while. Nope. Thinking is something you have to really work on, and really apply on a daily basis to keep sharp on the skill.

Why is it so important? Because no matter what you do in life, thinking will make you more efficient and better at it. Period.

You want to manage technical people? You need to think. In something that happened to be recently, a manager asked to look at all the outcomes of a matching. But didn't even stop to do the simple math that what he was asking for was in excess of 8 million matches. Even at a few minutes per match, that's way way too long to really be a useful strategy. Had I followed his advice, we'd have a giant Excel spread sheet that no one could open, and we'd still be going through the data.

Think, people! Think!

Don't just act… Think!

I recently read a story about two very famous original Unix architects that were sitting at a workstation (because they were scarce) and when there was a bug, one started hitting the debugger, and the other stopped doing anything -- he started thinking. His approach was that if you understood the system, then debugging was in your head. Thinking was more powerful than doing.

This isn't an invitation to analysis paralysis - that's taking something to an extreme. But you can't shoot first and ask questions later, either. Certainly when you're doing a lot of technical managing.

So let's all practice this a little more, OK?

Now this is probably not amazing news to long-time Ruby developers, but the simplicity and ease with which it's possible to make thread-local variables in Ruby is simply shocking. The ruby developers just don't know how good they have it. This morning I was looking at a threading problem with the CouchRest CouchDB client, and realized that it's not thread-safe. This isn't really shocking as thread-safety is something I've come to realize is not standard in Ruby libraries.

Still… I was determined to make it work.

What seemed logical was to have multiple database connections - one per thread, and then just have thread-local database connections. As the threads are born, they need a connection, create it, and use it. When they die, the connections are cleaned up automatically. Sweet. Simple.

But I know that dealing with thread-local storage in pthreads is not horrible, but it's certainly not "easy". I dig into the Ruby support for thread-local storage, and it's trivial:

  Thread.current[:foo] = {}

This creates the tagged thread-local variable foo. How simple! This is something that I never expected to see. Never! So why are these Ruby guys having so much trouble with thread-safety? I have no idea.

With the tools I've seen in Ruby, there's really no excuse for why there aren't more thread-safe libraries. All the tools are there - they're just unused. Lazy coders.

I know it needed to be done, and I know someone had to do it, but that doesn't make it any more fun than it already isn't. Digging in the data to find out why we aren't matching up merchants and demand in Philadelphia is no fun at all. It's a lot of data with very little pattern to it, and a whole lot of problems. But that's what I was doing for several hours today. The pain and suffering was really compounded by the complete lack of real thought put into this as we headed into the meeting.

Overall, I was very angry at myself for not pushing back. I should have. I know that now, but it's that blasted work ethic thing that causes me to say "Yes" when I should be saying "Hold on a sec…"

The problem is that we're getting demand and merchants to fulfill that demand, and the assumed "match" here should be very high. Why? Just because. They really have never looked at this and have no idea what it should be, but "instinctually" many think that it should be "very high" - like 90%. So when the first runs came out with it being more like 50%, they wanted to know why. I totally agree.

But where we diverge is in the How?

Once program manager suggested I send a 5000+ line Excel file where the hierarchical JSON data was somehow magically "flattened" to make it easy for anyone to look at the data and determine why the merchants weren't matching. Thankfully, I had the strength of character to say "No" to that.

But that wasn't until after I heard another request to log all 5000+ merchants against all 1500+ demands - yielding more than 8 million log lines. Nope. That's just plain silly.

I wanted to get to the bottom of this to be sure, but I wanted to do it in a way that makes at least a little sense. And looking at 8 million log lines isn't it. So I started building a few CouchDB temporary views and started looking for what wasn't being matched and why.

Turns out there were two major issues: the demand wasn't supplying sufficient 'service' coverage to pin enough merchants, and the zip codes on the merchant data was really pretty horrible. Call me 'Indy' on this - it only took me about 90 mins to find these reasons and document them up for the group. Nice. Clean. Efficient.

Nothing like looking at 8 million log lines.

Today I've started working with a new javascript charting library called ZingChart. It looks pretty complete, and certainly has a lot of charting styles, plus it's touting it's speed with large data sets, so that's nice. But as with anything like this, the learning curve is steep because the graph has to be configured in javascript, and it's always a lot of work to know not only How to do something, but also What's possible to be done.

Compounding this is that the documentation and the examples have contradictory information - I mean not even close! I spent several hours on trying to just get a nice pie chart going. But in the end, all the knobs are there and we don't have to worry that there's something we simply cannot do.

As an interesting note, they include the complete jQuery library in their 'resources' directory. It's something I've read a lot about, and it was available when I was doing my last web development work a few lifetimes ago, but we didn't use it because the Google Visualization Toolkit was more than enough for what we needed, and the need just didn't arise.

But it's interesting that it's all there. Very nice looking graphs.

Odd that they don't have a table. Guess I'll have to fall back to Google Visualizations for that.

I've been at The Shop about six weeks, and I thought this morning it was about time I updated my resumes and my web site. It's something I like to keep up to date because going back and figuring out what you did when, and for whom, is one of those things that you have to do when you're looking for a new position, and that's typically a pretty stressful time. There's no need to add to the stress by figuring all this out when it's possible to do it a little at a time, when it's all fresh in your memory.

So I took the time to update my three resumes, and the four web pages as well as my WordPress 'About me' page. I had updated LinkedIn a few weeks ago as I was still getting a lot of interest on that front, and it was only fair to shut that down by posting the new position.

It's really quite nice using the tools I have for this: Pages and Coda, as I really only need to write the text once and then I can just move it from document to page to document and simply reformat it. Sweet tools. But hey… it's a Mac - it's gotta be sweet!

All done and it only took me a few minutes for everything. Love to stay up to date.

Well… we just got out of the important demo that was coming up and it was a wonderful success! Everything went off according to plan, all the data was there, and it was just bad enough to keep people on their toes in the audience. I mean with figures like a potential deal value of $3.2mil for a single merchant - well, it's clear that we need to work on limits as well.

Still, it all went well, and while we may not have had the very top brass in the room, the point was that it's all running in production, end-to-end on production data, and from this point on we'll just be able to tweak the process and verify that it's still running. Very nice.

Good way to end a week, that's for sure.

We're in the final hours for a big demo with the Top Brass, and I'm trying to get things done, but I go to check on a run being done on the UAT box, and I find out that someone has started another copy! Now I know he didn't mean to mess me over, but he did. And I know he didn't mean to trash 30 mins of my work, but he did.

It's all that:

The road to Hell is paved with good intentions - Proverb

I know what it's like to be doing the best you can. I really do. I remember being on a Class A softball team, and I was lucky to get hits and play catcher. I was clearly outclassed. But this is a job, not a recreation. This is where you're supposed to be good - not just want to be good.

When I asked him if he'd started running things, he was honest and upfront, but said "Doesn't it check against that?" Of course not! Why would it? Well… now I have the answer to that question - I needed to write it for guys like him.

There was another exchange I had with someone… I had someone show me the following javascript:

  function(doc) {
    var filter = function(doc) {
      return doc.meta.label == "QuantumLead.results" &&
             doc.otcs.length > 0 &&
             doc.merchant.category != null &&
             doc.merchant.sales_value > 0;
    };
 
    var key = function(doc) {
      return [
        doc.meta.execution_tag,
        doc.meta.division,
        doc.merchant.category,
        doc.merchant.sales_value
      ];
    };
 
    var value = function(doc) {
      return {
        name:        doc.merchant.name,
        sf_id:       doc.merchant.sf_id,
        sales_value: doc.merchant.sales_value
      };
    };
 
    if (filter(doc)) {
      emit(key(doc), value(doc));
    }
  }

I asked him why he chose to write it that way. Just what the motivation was for the specific structure. His response was that this was about clarity and maintenance. To me, it seems awfully complex for something that I'd have written as:

  function(doc) {
    if (doc.meta.label == "QuantumLead.results" &&
        doc.otcs.length > 0 &&
        doc.merchant.category != null &&
        doc.merchant.sales_value > 0) {
      var key = [ doc.meta.execution_tag,
                  doc.meta.division,
                  doc.merchant.category,
                  doc.merchant.sales_value ];
      var value = { name:        doc.merchant.name,
                    sf_id:       doc.merchant.sf_id,
                    sales_value: doc.merchant.sales_value };
      emit(key, value);
    };
  }

When I asked him if he really thought that his was clearer than mine, he said "Yup", and so I let it drop. After all, there's no reason to make a big deal over this. But again, this is not what I'd call a good format, but hey… I'm trying to be more flexible and I'm no code enforcer here.

I know they mean well… they really do. But it's stuff like this that is exactly why, in the past, I've stepped up and simply pushed folks like this aside.

I'm trying to be better.

Because we have several data centers - including boxes at EC2, the standard set-up at The Shop is to have several proxies - forwarding all traffic through a port to a datacenter. So all traffic to the eastern EC2 center leaves my laptop on port 1234 (for instance). This means that when I want to use the CouchRest ruby client for CouchDB, I need to do something like this:

  require 'couchrest'
 
  CouchRest.proxy('http://localhost:1234/') if use_proxy?
  @db = CouchRest.database('http://whackamole.east:5984/hammer')

to connect to the hammer database on the whackamole.east server in the east EC2 datacenter. Not hard, but there seems to be a catch in there somewhere.

For when I try this, I get about 2000 of 4000 documents saved and then I get this problem. A nasty stack trace that appears to be about some timeout, and then an attempted reconnection. What's very interesting is that it has to be in the proxy handling code of CouchRest.

Why?

Because when I do a simple port forwarding on my box to the database server on the other box, and use that port - thus bypassing the need for the proxy setting, everything works. Also, if I run it on a box in the east EC2 datacenter so that I don't need a proxy, then everything works.

I need to dig into the proxy code. I'm not exactly sure what I'll find, but it's gotta be there.