What's it all about, Alfie?

Well... it's that time of year again, and I'm going to spend it quietly at home. Tomorrow I'm traveling to Indy to spend some time with my siblings as we're all gathering at my brother's place. It should be a lot of fun - it always is, and the food will be great, and my family makes me feel "normal".

Another year on the horizon, and in many ways, it'll be a lot better than the last, but there's so much pain in the future, I hesitate to think about it. Just take 5 mins at a time, and don't think too far into the future.

I'm on a little break, and I am watching the Netflix Original Marco Polo, and it's an excellent show. I'm a sucker for historical dramas, as I love to understand why and what were the causes for the turns and decisions in history. The writing is very good, and the acting and sets are just amazing. Well worth watching, if you have the time.

I would not have thought it, but just now a co-worker came by to say they were going to really miss working with me. She was brought to tears. I was shocked. When I asked why, she said she came to work at Groupon because of our interview, and it really stuck out to her the passion and commitment. She wanted to do that.

I remember that interview, and I wanted that too. She could be really great. But she let management push her around, and that was a mistake. She's young, and there will be more times to work together.

It's surprising to me how we touch each other's lives without knowing it. I thought I was just a pain to the group - at least in a large part, but I wasn't - not to her. And she was touched enough to express it.

I have had a good run here. Learned ruby, clojure, a lot of good things, and I've clearly touched at least two people here. That's a good run. I have nothing to complain about.

While at The Shop, I've been given an interesting list of name signs for my desk. I was really fortunate to start in a good group - where there was a nice balance of work and play, and they all took part in giving people names on their name signs. While I could just save the paper copies, I decided that I'd spend a little time, do the CSS, and put them all - with a nice commentary, as a post.

Without further ado, let's get right to it.

I'm a little fuzzy about the first one, but I remember distinctly asking about the name signs, and where I could get the metal stands they all sat on. I didn't say that I was hoping to get a great nematic from someone in the group, but within a few minutes of me asking about it, Mike had gotten a new stand out of the office supply cupboard, and Gary had printed out a name tag for me:

It was so nice because it's one of my favorite movies, and it was a wonderfully kind thing for the guys to do.

The next one, like all the others, were placed without my knowledge - which was the way the group worked. And I'm not sure exactly what I did to be awarded this name tag, but I'm sure it had to do with never giving up.

This next name tag was clearly from a time when everyone in the group was sold on the idea of moving forward, and I was quoting National Treasure about what we still had to do before we got there.

This one came along after someone had doubted me on something, and I answered back the the childhood response that I used for many a year.

The ruby gem collection is impressive - some are good, others are great, but there is the tendency to go with Convention over configuration, and in that sense, there was a new feature the group needed to add - and were hoping that it already existed in the wild. In the end, I quipped that if it existed, that it would be...

There was likely a time when I brought in a cold. The group was tight, and it was a mistake on my part. I should have stayed home. But it earned me this name tag.

It was customary to send out an email to the group when you were planning on working from home in the coming days. I was typing this email way too fast, and the title was a typo, but one of epic proportions in the group. I had people using this as the tag for working from home for months to come. It was my one, real contribution to the group's culture.

My consumption of Diet Coke was certainly noted and talked about around the group. At the same time, I was working on a project called Dark Magic, and so they gave me the name tag:

During the Bitcoin boom, a few of the guys in the group and I talked about doing some interesting things with Bitcoin. I'm still interested in it, but the lack of a solid financial backing means that it's just too volatile for me. One of the guys joked that if that were solved, how would I feel then, and my response made several of them laugh. It was almost immediately my new name tag.

While in a meeting with a group trying to internationalize the data on the cheap, the person presenting was very upbeat (to say the least), and while I wanted to temper the ideas with the reality, I wanted to also let this person know that I recognized their positive attitude. It was then my next name tag.

This next one came from Gary, but I wasn't really sure why. Just Gary being Gary, I suppose.

And my final name tag came when a few in the group realized that we all had unique initials, and so made name tags for everyone in the group:

And now it's my last day at The Shop. I start a new job just after the holidays, and that's going to be very nice, but I'll always remember my first group here, and all these fantastic name tags.

A while back, just after switching to OS X 10.10 Yosemite, I saw that Yahoo! wasn't at all happy with Adium. It simply wasn't connecting. Thankfully, there was a known fix in the works, and it didn't take long for someone to build a version and post it online. Soon after that, the Adium developers posted a similar update to their Nightly Builds, and I picked that up to be a little safer.

I have to admit that I'm a little surprised that it took so long, but I was willing to let it go - Adium has been a really great Open Source tool for me over the years, and it's certainly possible that they just got caught up in other things, and didn't have time to make the release.

Now we're in a slightly different situation, and there's a problem with MSN Live:

The story is that MSN has changed the accpetable API ids and the one that Adium and libpurple is using is not on the "approved" list. There is another in libpurple now, but the Adium team hasn't picked it up, or hasn't integrated it, or hasn't released it. In any case, it's stalled. Badly.

Now I'm not a big MSN IM user, but it's there, and it's simple to fix, and it's ready to go, so let's get to it, guys. Or let's make it clear that there are no developers to support this, in which case, I'll pick up the mantle, and just do the things that I think need to be done to keep it moving forward.

My fear about this is that the codebase is in horrible shape, and it's hours and hours to figure out anything, and then it's horrible to build, etc. There's also the Colloquy concern - it stopped updating, and then Textual picked up the banner, and I bought that app off the App Store. If someone looks at the sad state of Adium, I see them coming to the same conclusion and making a paid multi-protocol IM client.

I got another email from my friend on the updates I'd made to the solver, and he pointed out that the pattern function we were using:

  (defn pattern
    "Function to take a word (as a string) and return a vector that is the
    pattern of that word where the values are the index of the character.
 
      => (pattern \"see\")
        (0 1 1)
      => (pattern \"rabbit\")
        (0 1 2 2 3 4)
    "
    [word]
    (let [dw (distinct word)]
      (map #(.indexOf dw %) word)))

does something that's not strictly necessary to the proper functioning of the algorithm - the distinct call on the input word.

If we have a function that maps these words into some kind pattern, then it really doesn't matter what the pattern is - as long as it's just the pattern, and is not able to be confused with another pattern. So if we choose to change the code to look something like this:

  (defn pattern
    "Function to take a word (as a string) and return a vector that is the
    pattern of that word where the values are the index of the character.
 
      => (pattern \"see\")
        (0 1 1)
      => (pattern \"rabbit\")
        (0 1 2 2 4 5)
    "
    [word]
    (map #(.indexOf word (int %)) word))

where we are ignoring the cost of the repeated characters in the word in terms of the index they represent, then we have a similar pattern that's deterministic and comparable, but it lacks the call to distinct, and so it's going to be a little faster.

In the REPL, we can see the old way:

  cq.main=> (time (dotimes [_ 100000] (dpat "bivteclnbklzn")))
  "Elapsed time: 8.748 msecs"
  nil
  cq.main=> (time (dotimes [_ 100000] (dpat "bivteclnbklzn")))
  "Elapsed time: 8.158 msecs"
  nil

or about 87 nsec/call. That's fast. But when we test the new non-distinct mapper, we get:

  cq.main=> (time (dotimes [_ 100000] (pattern "bivteclnbklzn")))
  "Elapsed time: 4.859 msecs"
  nil
  cq.main=> (time (dotimes [_ 100000] (pattern "bivteclnbklzn")))
  "Elapsed time: 4.972 msecs"
  nil

or down to about 49 nsec/call. That's another significant percentage on that call, but I doubt it'll convert to something really significant in the overall solution. It's down to about 5 to 6 msec per benchmark solution. Not bad, considering it's all on a laptop, and all on the JVM. I do love this game!

Today I got an email from my friend about the times we were getting on the pattern function in the code - that function that turns the word into a sequence of ints that corresponds to the character pattern of the word. So we might have:

  => (pattern "see")
  [0 1 1]
  => (pattern "rabbit")
  [0 1 2 2 3 4]

At the time, my best code looked like this:

  (defn pattern
    [word]
    (mapv (into {} (map vector (distinct word) (range))) word))

and the times we were getting just for this function were not as good as his version:

  (defn pattern
    [word]
    (loop [p [] d {} n 0 f (first word) r (rest word)]
      (if f
        (if (contains? d f)
          (recur (conj p (d f)) d n (first r) (rest r))
          (recur (conj p n) (assoc d f n) (inc n) (first r) (rest r)))
        p)))

On 1000 calls he was beating me by a couple of milliseconds - and it makes sense... he's building the map as he goes, and I took the time to build it up-front, and I'm paying the cost for it. The big cost is that I then have to run map again to use the map I just created. His code builds and uses the map at the same time.

So I was trying to think of a way to speed up the process, and I couldn't think of a simple way, and then I decided just to try a simple test - use the Java indexOf method to get the index on the distinct string and then return that. The code then becomes:

  (defn pattern
    [word]
    (let [dw (distinct word)]
      (map #(.indexOf dw %) word)))

The times were startling:

  cq.main=> (time (dotimes [_ 1000] (bob "bivteclnbklzn")))
  "Elapsed time: 0.342 msecs"
  nil
  cq.main=> (time (dotimes [_ 1000] (bret "bivteclnbklzn")))
  "Elapsed time: 3.605 msecs"
  nil

where bob is my new version with indexOf and bret is his version with the loop/recur.

A factor of 10x. Wow! Ten times faster. Make the string and use the inherent properly of the string and it's index to do the mapping. What a hoot! I have really enjoyed working on this with my friend.

When we put this into the web server, we get even better solve times:

[12:29:13.757:qtp1360409373-14] INFO  cq.block - Finished match-up [8 words] in 1ms.
[12:29:13.763:qtp1360409373-14] INFO  cq.block - Finished solve in 7ms.

...just amazingly fun.

This morning I thought I'd just check on the status of Apache + PHP + Postgres on OS X 10.10 as it was likely that in the update from 10.9, Apple had changed things, and sure enough, they have. But it wasn't all them - Apache has some changes in it as well, so to be fair, getting this all to run is a little more complex than you might imagine, but it's still not all that hard.

Getting PostgreSQL 9.3

As with the previous post I'm using Homebrew and that's made the entire operation a lot smoother. There's no 32-bit/64-bit issue as Homebrew does both, and it builds in on your box, so again, a lovely customized solution with a simple command:

  $ brew install postgresql

It even has a nice, simple informational blurb about how to start/stop and upgrade. Very nice. But now that it's on the box, and ready to roll, let's add in the PostgreSQL support to the PHP 5.5.14 that's installed with Mac OS X 10.10.

Activating UserDir in Apache 2.4.9

The first thing I noticed is that the UserDir extension is not enabled by default, so we need to get that going right away. This enables the code to be run from the development directories, and that's a big time-saver. First, we need to enable the UserDir module in Apache, and then make a specific config file for the user in question. Start by editing /etc/apache2/httpd.conf and line 166 needs to be uncommented to read:

  LoadModule userdir_module libexec/apache2/mod_userdir.so

and then similarly on line 493 uncomment the line to read:

  Include /private/etc/apache2/extra/httpd-userdir.conf

At this point, you need to make sure you have at least one file in the /etc/apache2/users/ directory for each user, like: drbob.conf:

  <Directory "/Users/drbob/Sites/">
      Options FollowSymLinks Indexes MultiViews
      AllowOverride None
      Order allow,deny
      Allow from all
      Require all granted
  </Directory>

where the last line - Require all granted is new as of Apache 2.4, and without it you will get errors like:

  [Thu Dec 18 10:41:32.385093 2014] [authz_core:error] [pid 55994]
    [client fe80::7a31:c1ff:fed2:ca2c:58108] AH01630: client denied by server
    configuration: /Users/drbob/Sites/info.php

Activating PHP in Apache

The mext thing to do is to activate PHP in the supplied Apache 2 with OS X 10.10. This is line 169 in the file - /etc/apache2/httpd.conf and you need to uncomment it to read:

  LoadModule php5_module libexec/apache2/libphp5.so

and then add a new file called /etc/apache2/other/php5.conf and have it contain:

  <IfModule php5_module>
    AddType application/x-httpd-php .php
    AddType application/x-httpd-php-source .phps
 
    <IfModule dir_module>
      DirectoryIndex index.html index.php
    </IfModule>
  </IfModule>

which does all the other PHP configuration in a separate file to make upgrades easy.

Building in PostgreSQL 9.3 extra for PHP 5.5.14

At this point we need to get the source code for the exact version of PHP that Apple ships. This is pretty easy by getting the PHP 5.5.14 source from a mirror and unpacking it into a directory. We then just run the following commands:

  $ cd php-5.5.14/ext/pgsql/
  $ phpize
  $ ./configure
  $ make

at the end of this process, you'll have a file: php-5.5.14/ext/pgsql/modules/pgsql.so and that needs to be placed in the right directory and referenced in the php.ini file.

For Mac OS X 10.10.1, this is accomplished with:

  $ sudo cp modules/pgsql.so /usr/lib/php/extensions/no-debug-non-zts-20121212/

and then edit the /etc/php.ini file to include this new extension by adding the line:

  extension=pgsql.so

Thankfully, the extension_dir is already set in OS X 10.10.1, and we no longer have to set that.

Finishing Up

At this point, a simple restart of apache:

  $ sudo apachectl restart

and everything should be in order. Hit a URL that's a static file with the contents:

  <?php
    phpinfo();
  ?>

and you should see all the details about the PHP install - including the PostgreSQL section with the version of Postgres indicated:

All is working once again!

I read with a giggle this article off Slashdot about the problems companies are having making Hadoop actually work in their environment. This is not in the least surprising to me. Hadoop is a nice, distributed, storage environment, but so is a very nice SAN. Both can store a lot of files, and do it very quickly, with redundancy, but one is putting entire computers on each "disk", while the other is allowing the disks to just be... well... disks.

The promise (hype) of Hadoop is that by distributing the computing power like the storage, the map/reduce jobs can be done fast and easily and you get old-style SQL performance with as much online storage as you can muster.

But the truth of the matter is far different than this, as this article in the WSJ attests.

Hadoop is nice, but it's not fast. It's good for lots of storage space, but so is a SAN. It's nice to do small map/reduce jobs, but so is CouchDB. But you can't scale CouchDB to any size you want, either. There will always be limits.

I know folks that are looking at Cassandra, and really like it. It's not as general-purpose as Hadoop, but it's targeted at the problem of massive storage with SQL access. Advocates of Hadoop will say Use Spark - get SQL that way! and that's possible, but then that's not Hadoop, is it?

You can use many different caching schemes to make any storage scheme work faster, but the promise of Hadoop was that you wouldn't need them. That's where it fell over. I have no doubt that Hadoop can do a lot of really good things in a lot of very specialized environments, but it's not the silver bullet worth $2 billion. It's a nice open source map/reduce like CouchDB, but on as many boxes as you want. It's nice... but it's not what people hoped it would be.

Too many distractions in systems development these days. Just too many.

Over break I'm going to be working at home on a few little fun projects, and with my two Thunderbolt displays, I was thinking I might need a hub of some kind, so I went to look for one on the Apple Store web site - just to see what was out there, and what they might cost - if I shopped in town.

I honestly didn't find anything, and then I wondered What's on the back of the monitor? to see if I could chain these together? Turns out, you can. The guy working across from me as two, and they are chained, and I've seen them work just fine. There's even a YouTube video on the set-up.

Very nice!

This will be just what I have when I get started on Saturday morning - if I wait that long. Apple stuff is pricey, yes, but you do really get what you pay for, and this is no exception: one cable to the laptop, driving two 27" displays. Sweet.