What's it all about, Alfie?

I was scanning iPhoto this morning looking for the picture of the last family Christmas Tree I had, and I came across this picture that was taken of my family... my siblings and me. We have not always agreed with one another, but we have always been there for each other. Without question. Without hesitation. Without judgement.

I am blessed beyond words.

I just updated my post about the conversion of the CryptoQuip Solver from Obj-C to ruby, and then for a piece of that to clojure, after having built that code in LightTable, and it just brought a smile to my face. The code was fun because closure allows me to do things that most languages simply don't, and it's always a simplification when that happens. In ruby I had to say:

  (pairs.count == ctc) && (ctc == ptc)

but in clojure, it's simpler:

  (= (count p) ctc ptc)

the = operator allows for any number of arguments - as does the other inequalities. Say I needed to have four numbers that all were ascending. That would be a hassle in ruby - maybe a sort, followed by the check that no two were the same, but in clojure it's as simple as:

  (< a b c d)

where it's only going to return true is d is greater than c is greater than b is greater than a. It's simple, and it's elegant.

But I also really like writing to friends about this stuff, and writing here as well. I've been recovering for almost two years now, and it's about time I work a little harder at just forgiving and moving on.

Because there's so much fun stuff to do, and if this little code block is all that exciting and fun to me, I've got a lot of fun ahead of me in a place that will feed my desire to solve problems.

And if I'm lucky enough to once again work for myself, then this blog is going to once again be open, and I'll be putting a lot more about solving problems up here as I think that's just amazingly fun stuff to share.

I was chatting with a friend today, and the movie Die Hard came up, and I realized that I haven't seen that moving in many years. Just as importantly, it was a really good movie. And the second one as well. So I was thinking... maybe I'll have a little Die Hard Christmas Film Festival? That might be nice.

So I looked at iTunes, and I can get the complete set of movies for $40. Not bad. I then looked at BestBuy and the Blu-Ray set is only like $27. I have no Blu-Ray player, but then I looked those up at BestBuy, and they aren't more than $60. Hey... we might have a winner here.

I've got my nice TV in the living room... and a $60 Blu-Ray player that does DVDs as well, and then the Die Hard set and for less than $100 I can have a really nice time on Christmas Day. I'm starting to look forward to this... Nice!

This morning I finished the little project I started yesterday on porting my old Obj-C CryptoQuip Solver from Obj-C to ruby. I wanted to brush up on my ruby skills as it's been nearly 18 months since I've done any real solid ruby development, and I've also wanted to get this codebase ported to Swift, and there are a lot of similarities between ruby and Swift. So this morning I finally got it all working as I wanted. Nothing really major, but I had made a few changes to try and capitalize on the strengths of ruby, and those made a few things that needed to be fixed up.

Nothing major.

But Wow... the codebase is significantly smaller. The original Obj-C code was well documented, and comes in at 3370 lines for all the headers and implementation files. Ruby is not as extensively documented, but it's a lot smaller so I didn't feel the need. It's a total of 316 lines. That's a factor of ten. Wow.

One of the nicest parts of the port was to change how I determined if a cyphertext word could potentially match a plaintext word. In the Obj-C code it looked like this:

  /*!
   One of the initial tests of a plaintext word is to see if the pattern of
   characters matches the cyphertext. If the pattern doesn't match, then the
   decoded text can't possibly match, either. This method will look at the
   pattern of characters in the cyphertext and compare it to the pattern in
   the argument and if they match, will return YES.
   */
  - (BOOL) matchesPattern:(NSString*)plaintext
  {
    BOOL   match = YES;
 
    // make sure that we have something to work with
    if (match && (([self getCypherText] == nil) || (plaintext == nil))) {
      match = NO;
    }
 
    // check the lengths - gotta be the same here for sure
    if (match && ([[self getCypherText] length] != [plaintext length])) {
      match = NO;
    }
 
    /*
     * Assume that each pair of characters is a new map, and then test that
     * mapping against all other cyphertext/plaintext pairs that SHOULD match
     * in the word. If we get a miss on any one of them, then we need to fail.
     */
    if (match) {
      unichar      cypher, plain, c, p;
      NSUInteger   len = [plaintext length];
      for (NSUInteger i = 0; (i < len) && match; ++i) {
        // get the next possible pair in the two words
        cypher = [[self getCypherText] characterAtIndex:i];
        plain = [plaintext characterAtIndex:i];
        // check all the remaining character pairs
        for (NSUInteger j = (i+1); (j < len) && match; ++j) {
          c = [[self getCypherText] characterAtIndex:j];
          p = [plaintext characterAtIndex:j];
          if (((cypher == c) && (plain != p)) ||
              ((cypher != c) && (plain == p))){
            match = NO;
            break;
          }
        }
      }
    }
 
    return match;
  }

where it's a pretty simple double-loop on the words. Nothing horribly hard here, but for ruby I went with a different idea to start with:

  # One of the initial tests of a plaintext word is to see if the pattern of
  # characters matches the cyphertext. If the pattern doesn't match, then the
  # decoded text can't possibly match, either. This method will look at the
  # pattern of characters in the cyphertext and compare it to the pattern in
  # the argument and if they match, will return true.
  def matches_pattern?(plaintext)
    return false unless @cyphertext && plaintext &&
                        @cyphertext.length == plaintext.length
    pairs = @cyphertext.chars.zip(plaintext.chars).uniq
    ctc = pairs.map { |p| p.first }.uniq.count
    ptc = pairs.map { |p| p.last }.uniq.count
    (pairs.count == ctc) && (ctc == ptc)
  end

I like the compactness of the ruby - very expressive, but the question is At what cost? The ruby version will check all the paris in the words, even if there is a mismatch in the second letter. This cost may - or may not - be a big deal. But it does simplify the code quite a bit.

The downside of the ruby implementation is that it's pretty slow. Where the Obj-C version was in the 60 msec range the ruby version (1.9.3 MRI) is more like 650 msec. This isn't a total shock, and I could have tried JRuby which will be considerably faster once the JIT gets warmed, but that wasn't the real point. It was just a fun little project to get this into ruby so that it might make it a little easier to get it into Swift.

And I got to brush up on my ruby too.

UPDATE: I tried it with JRuby 1.7.0, and yes, it's old, but it's a point of reference, and the times were worse. I did not expect that. The run times were in the 7 sec range. About an order of magnitude over the RMI version. Same code. Very odd, but I wanted to see what it'd be, and now I know.

UPDATE: I couldn't resist the urge to convert the new ruby scheme to clojure simply because I think clojure is going to be a lot more performant, and so I wanted to give that a try. The key pattern matching function is now:

  (defn matches?
    "Function to see if the cyphertext and plaintext have the same pattern of
    characters such that they could possibly match - given the right legend."
    [ct pt]
    (if (and (string? ct) (string? pt) (= (count ct) (count pt)))
      (let [p (distinct (map vector ct pt))
            ctc (count (distinct (map first p)))
            ptc (count (distinct (map last p)))]
        (= (count p) ctc ptc))))

and in a REPL, it's very easy to see that it's working:

  (matches? "wdllpy" "rabbit")
    => true
  (matches? "wdllpd" "rabbit")
    => false 

I have to admit that clojure is about the most fun language that I've used in a very long time. Ruby is nice, and it's very close - but clojure is just functional - and performant - all the way.

[12/12] UPDATE: my friend wrote me back with a significant simplification to the code:

  (defn matches?
    "Function to see if the cyphertext and plaintext have the same pattern of
    characters such that they could possibly match - given the right legend."
    [ct pt]
    (if (and (string? ct) (string? pt) (= (count ct) (count pt)))
      (let [pc (count (distinct (map str ct pt)))
            ctc (count (distinct ct))
            ptc (count (distinct pt))]
        (= pc ctc ptc))))

I was worried about doing this, initially, because I was thinking that it would be possible to re-arrange the letters while keeping the number of distinct characters the same, and therefore make additional possible matchings that would make it impossible to match.

What I realized with his help is that the key is that we have three checks - the pairs, and the distinct chars in the words. With all three, it's going to catch all the cases I was worried about.

Also, I really like how he went to using str as opposed to vector. Very nice. Just as effective, but far cleaner and easier to deal with in the distinct.

I can then go back and look at the ruby code and update the constructor for the CypherWord:

  def initialize(cyphertext)
    @cyphertext = cyphertext
    @cyphertext_chars = cyphertext.chars
    @cyphertext_uniq_count = cyphertext.chars.to_a.uniq.count
  end

and then the matches_pattern? becomes the simpler:

  # One of the initial tests of a plaintext word is to see if the pattern of
  # characters matches the cyphertext. If the pattern doesn't match, then the
  # decoded text can't possibly match, either. This method will look at the
  # pattern of characters in the cyphertext and compare it to the pattern in
  # the argument and if they match, will return true.
  def matches_pattern?(plaintext)
    return false unless @cyphertext && plaintext &&
                        @cyphertext.length == plaintext.length
    pc = @cyphertext_chars.zip(plaintext.chars).uniq.count
    ptc = plaintext.chars.to_a.uniq.count
    (pc == @cyphertext_uniq_count) && (@cyphertext_uniq_count == ptc)
  end

Yeah... we're beating this Bad Boy into the ground, but it's a lot of fun to be working with my friend on something - even if it's just a 30-year olg problem.

This morning I got in and fired up my favorite Christmas song playlist and plan to just listen to it over and over all day long. This is my favorite time of the year. I love the music, it makes me feel peaceful, and thankful, and that's a really good place to be.

Every day is not going to be perfect. There are times things still get me a little down... OK, there are times that things get me a lot down, but as long as I can stay focused on the meaning of Christmas, and why we are here, then it doesn't last long.

I don't have a big christmas tree like I used to. But the little tree I have now I leave up all year around because I don't want to let go of this feeling. It makes me feel better to see it each day, and hopefully very soon I'll be at peace much more of the time. But life is what it is, and I need to just keep moving.

But I am blessed. Very blessed.

This was the last family Christmas Tree I had... and it was very beautiful.

I don't remember who told me this story, but it's an interesting tale that has really hit home this morning as I've been reading some of the posts I wrote in the last month. The story is about The Tree of Problems, and it goes something like this:

At the end of each person's life, they walk up to The Tree of Problems where every person takes all their problems - as a group, and hangs them on the tree. Then they walk around to see what set of problems they would like to have in their next life, and after looking at all the problems on the tree, they pick theirs back up.

Because they realize they can deal with these problems best.

And it's really just a statement that when we look at people, and think they have no problems, that's because we aren't really seeing what their life is like. We are only seeing this tiny slice of their life that they choose to expose to us. And the real issues are not something that we really almost never see, because most of us really try to hide those bad problems from others - even friends and family.

I'm currently in the middle of this job search. It's going well. I'm hoping to have something all lined up before the first of the year, but I'm trying to manage my expectations as well. This all relates to The Tree of Problems in that I've not been happy at work for a while. A long while. I can look at posts going back a while, and while there are times when the work was fun and exciting, the environment I'm working in is really not something I like.

Yet I held onto it far too long.

I kept thinking "It'll get better"... or "This is the system I know"... when in reality it's not going to get better, and in fact, it's getting worse. Not because the people are mean or nasty - just because they share different values that I do, and there isn't a group in Chicago that appears to share my values.

This also reminds me of Liza. Loving and marrying the wrong person is a recipe for the same kind of grief. You think "I'll work harder"... or "I'll stop doing this for them"... or any of a million other things, all of which don't matter because ultimately, they aren't interested in being with you. It's a bad fit.

So I need to Let Go... as Richard Bach said in Illusions... let go of the rocks I'm clinging to at the bottom of this river and trust that the river knows where it's going, and will carry me along. It's not all that easy, but nothing of value is. Letting go and having a little more faith is what I need, and this morning it was just so clear that people hold on all the time.

So I just need to cut myself a little slack and try and relax as I see where my path is headed.

Last week I was in an interview where they wanted me to code in Ruby - nothing else. I wasn't really prepared to write Ruby, as it's been a good 18 months since I've written a line of it - maybe more. But I excused my poor memory of the syntax and dug in. But it got me thinking, and I decided to brush up on my Ruby skills, and have a little fun at the same time.

So I made a new directory, and started grinding the gears on my memory to get things back in line for working on Ruby. This means remembering rvm, and how to even find the version of Ruby for the .ruby-version file in the root of the project. Then there were the .rspec lines for getting nice looking test output - and running the tests in random order every time. And I still hadn't gotten to the code.

So I started working on the programming task I got in the interview, and it was really amazing to me that some of the hints I'd gotten in the interview were really not at all how I wrote it when I had the chance to do it on my own. For example, I needed to take a string, and get the first character, and then the rest - very much like clojure would do it. I was given the hint that "hello".chars would give me an Array of the characters, and on their box it did - but in Ruby 1.9.2, it returns an Enumerator, and that's an entirely different thing. Also, it requires that we allocate new storage.

What I really wanted was: "hello"[0,1] - that's a string of the first character in the string. And then the rest is just: "hello"[1,10], or another long number to get the rest. This is what was sitting in the back of my memory, and when I saw it, a lot of other little things started flooding back in.

Things started accelerating, and in no time I was looking at a far cleaner version of the test code I'd written just a few days ago. Far more idiomatic Ruby, and the tests were super simple to write, and took advantage of the contextual nature of the tests.

I'm not going to claim that I'm even a decent Ruby coder. I'm just fair. The class library is just far too expansive to know it all without dealing with it on a daily basis, and even then, on a large codebase that takes advantage of the different classes as well. It's vast. Which is nice, don't get me wrong, but it's why I know it'd take me a very long time to master even where things are - let alone their syntax and usage. But it's something I may have to do, so it's nice to get back in it.

I have been using VisualVM for quite a while now... it's got the monitoring tools that work pretty nicely if you have the network connectivity you need to get to the box. What I learned this morning on the #clojure channel on IRC is that it's now included in the JVM.

That's sweet! SO I had to check:

  $ which jvisualvm
  /usr/bin/jvisualvm

Really nice! It's the same tool, but now it's integrated, and that's even better.

Now I'll be able to use it on any platform and not have to hassle with downloading it as a separate install. Very nice!

Clojure is forced to use reflection in a lot of cases simply because there is no type system in place, and when you're iterating on a sequence of elements, you have to ask each one Hey... what are you? and then see if you can do something with that. It's understandable, and it's the point of adding type hinting in the function signatures, but it hurts when you have Java auto-boxing int to Integer, and then trying to do math on those values.

It slows simple math way down - and that's not typically something you want to slow down.

Thankfully, in Clojure 1.7, there's a new dynamic variable that will check all the math to make sure that it's not boxed:

  (set! *unchecked-math* :warn-on-boxed)

and then for the scope of that definition, it'll check to see that all the math is not boxed. This can help identify the source of a performance issue without the painful profiling.

Great to see... now if Storm could just be on 1.7...

I've been trying to get some data validation code going, and one of the best looking libraries is Prismatic's Schema. It's got both clojure and clojurescript, as well as a very well thought out scheme. It's got capabilities to run tests on code, in-line, or as data to filter like tests. It's very nice.

Problem is, it's meant to use clojure 1.5.1 or better.

But I wanted to see if we could get it going in clojure 1.4.0 for the Storm topologies we have. Turns out, it wasn't all that easy. But it appears possible.

First, let's change a few things in the project.clj file. First, make the version something that's clear it's not the normal version:

  (defproject prismatic/schema "0.3.4G"

and then, because clojure 1.4.0 doesn't have the EDN reader, we needed to add:

    :dependencies [[potemkin "0.3.2"]
                   [org.clojure/tools.reader "0.8.12"]]

and then changed the main version of clojure:

    :profiles [{:dev {:dependencies [[org.clojure/clojure "1.4.0"]

And then in src/cljx/schema/coerce.cljx we need to reference the correct EDN reader:

    #+clj [clojure.tools.reader.end :as end]

and then in test/cljx/schema/core_test.cljx we need to use a proper defprotocol for 1.4.0:

  (defprotocol ATestProtocol (test-fn [x]))

And at this point, we can run:

  $ lein do clean, reps, test

And it will work just fine - for the clojure part. The clojure script requires 1.5.1, but that's OK, as we don't need it.