Archive for the ‘Coding’ Category

Postgres 9.3 and Mac OS X 10.10 Yosemite Upgrade

Thursday, December 18th, 2014

Yosemite

This morning I was looking into yet another Adium problem with MSN Live, and I opened up the Console.app and noticed a lot of messages like this:

  12/18/14 4:08:22.886 AM com.apple.xpc.launchd[1]:
    (homebrew.mxcl.postgresql[52026]) Service exited with abnormal code: 1

and when I tried to check if Postgres was running:

  $ psql -l
  psql: could not connect to server: No such file or directory
          Is the server running locally and accepting
          connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

so it was clear that something was wrong with postgres on my laptop.

To be fair, I haven't really messed with this since going to Yosemite, so I hit DuckDuckGo and it reported this post on StackOverflow, which says that in the Yosemite upgrade, several empty directories were deleted, and all I needed to do was to re-create them and restart postgres.

So I:

  $ launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
  $ cd /usr/local/var/postgres
  $ mkdir pg_tblspc
  $ mkdir pg_twophase
  $ mkdir pg_stat_tmp
  $ chmod 700 pg_tblspc pg_twophase pg_stat_tmp
  $ launchctl load ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist

and then I could see all the databases on the laptop, and the Console.app showed that things were running just fine.

While I understand that postgres is inefficient as it's running all the time, I still love this database, and it's worth the cycles to have it on my development laptop because it scales to the server room so bloody well.

Problem solved. Whew!

Bitbucket is Pretty Decent

Wednesday, December 17th, 2014

Bitbucket

While it's no GitHub, I've been working with Bitbucket for a few projects with a friend, and I have to say it's not bad at all. I happen to like the helpful hints GitHub offers on making a new repo, but the help Bitbucket has is very fair, and reasonable. It's got a little different UI, which is fine, and it works pretty much the same so that you don't have to come up to speed with a new workflow if you're switching back and forth from Bitbucket to GitHub.

I looked at Bitbucket a long time ago, and it wasn't nearly as similar to GitHub as it is now, and that makes a lot of sense. It's got Pull Requests, and it's got a very nice source view:

Bitbucket Source

Many will say It's just like GitHub's - which is a good thing. Face it - you copy what's a great design and works well. This is one of those cases.

The thing I like about Bitbucket is the free repos for small teams - up to 5 people. In short, they have a different approach than GitHub - repos are free, people are charged. It's a nice approach because it means that non-Open Source projects that you work on with a friend can be private without a cost. GitHub charges for the first private repo, and I don't begrudge them that - I'm a paying GitHub user. But I like the different approach because it gives small groups a leg up.

I'm not going to stop using GitHub, but it's nice to see that Bitbucket exists and seems every bit as good as GitHub for the task. It's something I'll remember.

XQuartz 2.7.7 on OS X 10.10.1 Yosemite

Friday, December 12th, 2014

X11.jpg

As I didn't have a lot to do this afternoon, I decided to check on the state of X11 on OS X 10.10.1 - and it's actually not bad at all. I was able to get Quartz 2.7.7 from the download site, and it installed easily enough. All the normal things run, and it's a complete X11 we just need the apps - or in the case of SSH access, getting to a linux machine for the X11 apps on the remote host.

Because I got rid of my home machines with X11 a while back, I'm not sure how much this is going to get used any more, but it's nice to know that it works before I really need it, so that I know I can go to it in a time of need.

Amazing Documentation Viewer – Dash

Friday, December 12th, 2014

Dash

I've been using Dash for a while now. It started out with a really odd looking cat as the icon, and it was a little iffy in the beginning, but it got better - a lot better. Today, it's the best documentation viewer I have ever used. It's just stunning. And it's fast. I can't believe how fast it is, but maybe it caches everything in a docket in memory - who knows? But Wow... it's easy to use, expansive on all the docs it covers, and the speed and ease of use are simply hard to beat.

I have a set for Clojure coding which includes clojure, postgres, and Java, and I have one set for ruby - with ruby, postgres and Java (for JRuby). All told, it's amazing how access to the right docs immediately is so powerful.

The Joy of Writing

Thursday, December 11th, 2014

smiley.jpg

I just updated my post about the conversion of the CryptoQuip Solver from Obj-C to ruby, and then for a piece of that to clojure, after having built that code in LightTable, and it just brought a smile to my face. The code was fun because closure allows me to do things that most languages simply don't, and it's always a simplification when that happens. In ruby I had to say:

  (pairs.count == ctc) && (ctc == ptc)

but in clojure, it's simpler:

  (= (count p) ctc ptc)

the = operator allows for any number of arguments - as does the other inequalities. Say I needed to have four numbers that all were ascending. That would be a hassle in ruby - maybe a sort, followed by the check that no two were the same, but in clojure it's as simple as:

  (< a b c d)

where it's only going to return true is d is greater than c is greater than b is greater than a. It's simple, and it's elegant.

But I also really like writing to friends about this stuff, and writing here as well. I've been recovering for almost two years now, and it's about time I work a little harder at just forgiving and moving on.

Because there's so much fun stuff to do, and if this little code block is all that exciting and fun to me, I've got a lot of fun ahead of me in a place that will feed my desire to solve problems.

And if I'm lucky enough to once again work for myself, then this blog is going to once again be open, and I'll be putting a lot more about solving problems up here as I think that's just amazingly fun stuff to share.

Ported the CryptoQuip Solver to Ruby

Wednesday, December 10th, 2014

Ruby

This morning I finished the little project I started yesterday on porting my old Obj-C CryptoQuip Solver from Obj-C to ruby. I wanted to brush up on my ruby skills as it's been nearly 18 months since I've done any real solid ruby development, and I've also wanted to get this codebase ported to Swift, and there are a lot of similarities between ruby and Swift. So this morning I finally got it all working as I wanted. Nothing really major, but I had made a few changes to try and capitalize on the strengths of ruby, and those made a few things that needed to be fixed up.

Nothing major.

But Wow... the codebase is significantly smaller. The original Obj-C code was well documented, and comes in at 3370 lines for all the headers and implementation files. Ruby is not as extensively documented, but it's a lot smaller so I didn't feel the need. It's a total of 316 lines. That's a factor of ten. Wow.

One of the nicest parts of the port was to change how I determined if a cyphertext word could potentially match a plaintext word. In the Obj-C code it looked like this:

  /*!
   One of the initial tests of a plaintext word is to see if the pattern of
   characters matches the cyphertext. If the pattern doesn't match, then the
   decoded text can't possibly match, either. This method will look at the
   pattern of characters in the cyphertext and compare it to the pattern in
   the argument and if they match, will return YES.
   */
  - (BOOL) matchesPattern:(NSString*)plaintext
  {
    BOOL   match = YES;
 
    // make sure that we have something to work with
    if (match && (([self getCypherText] == nil) || (plaintext == nil))) {
      match = NO;
    }
 
    // check the lengths - gotta be the same here for sure
    if (match && ([[self getCypherText] length] != [plaintext length])) {
      match = NO;
    }
 
    /*
     * Assume that each pair of characters is a new map, and then test that
     * mapping against all other cyphertext/plaintext pairs that SHOULD match
     * in the word. If we get a miss on any one of them, then we need to fail.
     */
    if (match) {
      unichar      cypher, plain, c, p;
      NSUInteger   len = [plaintext length];
      for (NSUInteger i = 0; (i < len) && match; ++i) {
        // get the next possible pair in the two words
        cypher = [[self getCypherText] characterAtIndex:i];
        plain = [plaintext characterAtIndex:i];
        // check all the remaining character pairs
        for (NSUInteger j = (i+1); (j < len) && match; ++j) {
          c = [[self getCypherText] characterAtIndex:j];
          p = [plaintext characterAtIndex:j];
          if (((cypher == c) && (plain != p)) ||
              ((cypher != c) && (plain == p))){
            match = NO;
            break;
          }
        }
      }
    }
 
    return match;
  }

where it's a pretty simple double-loop on the words. Nothing horribly hard here, but for ruby I went with a different idea to start with:

  # One of the initial tests of a plaintext word is to see if the pattern of
  # characters matches the cyphertext. If the pattern doesn't match, then the
  # decoded text can't possibly match, either. This method will look at the
  # pattern of characters in the cyphertext and compare it to the pattern in
  # the argument and if they match, will return true.
  def matches_pattern?(plaintext)
    return false unless @cyphertext && plaintext &&
                        @cyphertext.length == plaintext.length
    pairs = @cyphertext.chars.zip(plaintext.chars).uniq
    ctc = pairs.map { |p| p.first }.uniq.count
    ptc = pairs.map { |p| p.last }.uniq.count
    (pairs.count == ctc) && (ctc == ptc)
  end

I like the compactness of the ruby - very expressive, but the question is At what cost? The ruby version will check all the paris in the words, even if there is a mismatch in the second letter. This cost may - or may not - be a big deal. But it does simplify the code quite a bit.

The downside of the ruby implementation is that it's pretty slow. Where the Obj-C version was in the 60 msec range the ruby version (1.9.3 MRI) is more like 650 msec. This isn't a total shock, and I could have tried JRuby which will be considerably faster once the JIT gets warmed, but that wasn't the real point. It was just a fun little project to get this into ruby so that it might make it a little easier to get it into Swift.

And I got to brush up on my ruby too.

UPDATE: I tried it with JRuby 1.7.0, and yes, it's old, but it's a point of reference, and the times were worse. I did not expect that. The run times were in the 7 sec range. About an order of magnitude over the RMI version. Same code. Very odd, but I wanted to see what it'd be, and now I know.

UPDATE: I couldn't resist the urge to convert the new ruby scheme to clojure simply because I think clojure is going to be a lot more performant, and so I wanted to give that a try. The key pattern matching function is now:

  (defn matches?
    "Function to see if the cyphertext and plaintext have the same pattern of
    characters such that they could possibly match - given the right legend."
    [ct pt]
    (if (and (string? ct) (string? pt) (= (count ct) (count pt)))
      (let [p (distinct (map vector ct pt))
            ctc (count (distinct (map first p)))
            ptc (count (distinct (map last p)))]
        (= (count p) ctc ptc))))

and in a REPL, it's very easy to see that it's working:

  (matches? "wdllpy" "rabbit")
    => true
  (matches? "wdllpd" "rabbit")
    => false 

I have to admit that clojure is about the most fun language that I've used in a very long time. Ruby is nice, and it's very close - but clojure is just functional - and performant - all the way.

[12/12] UPDATE: my friend wrote me back with a significant simplification to the code:

  (defn matches?
    "Function to see if the cyphertext and plaintext have the same pattern of
    characters such that they could possibly match - given the right legend."
    [ct pt]
    (if (and (string? ct) (string? pt) (= (count ct) (count pt)))
      (let [pc (count (distinct (map str ct pt)))
            ctc (count (distinct ct))
            ptc (count (distinct pt))]
        (= pc ctc ptc))))

I was worried about doing this, initially, because I was thinking that it would be possible to re-arrange the letters while keeping the number of distinct characters the same, and therefore make additional possible matchings that would make it impossible to match.

What I realized with his help is that the key is that we have three checks - the pairs, and the distinct chars in the words. With all three, it's going to catch all the cases I was worried about.

Also, I really like how he went to using str as opposed to vector. Very nice. Just as effective, but far cleaner and easier to deal with in the distinct.

I can then go back and look at the ruby code and update the constructor for the CypherWord:

  def initialize(cyphertext)
    @cyphertext = cyphertext
    @cyphertext_chars = cyphertext.chars
    @cyphertext_uniq_count = cyphertext.chars.to_a.uniq.count
  end

and then the matches_pattern? becomes the simpler:

  # One of the initial tests of a plaintext word is to see if the pattern of
  # characters matches the cyphertext. If the pattern doesn't match, then the
  # decoded text can't possibly match, either. This method will look at the
  # pattern of characters in the cyphertext and compare it to the pattern in
  # the argument and if they match, will return true.
  def matches_pattern?(plaintext)
    return false unless @cyphertext && plaintext &&
                        @cyphertext.length == plaintext.length
    pc = @cyphertext_chars.zip(plaintext.chars).uniq.count
    ptc = plaintext.chars.to_a.uniq.count
    (pc == @cyphertext_uniq_count) && (@cyphertext_uniq_count == ptc)
  end

Yeah... we're beating this Bad Boy into the ground, but it's a lot of fun to be working with my friend on something - even if it's just a 30-year olg problem.

Refreshing my Ruby Skills

Monday, December 8th, 2014

Ruby

Last week I was in an interview where they wanted me to code in Ruby - nothing else. I wasn't really prepared to write Ruby, as it's been a good 18 months since I've written a line of it - maybe more. But I excused my poor memory of the syntax and dug in. But it got me thinking, and I decided to brush up on my Ruby skills, and have a little fun at the same time.

So I made a new directory, and started grinding the gears on my memory to get things back in line for working on Ruby. This means remembering rvm, and how to even find the version of Ruby for the .ruby-version file in the root of the project. Then there were the .rspec lines for getting nice looking test output - and running the tests in random order every time. And I still hadn't gotten to the code.

So I started working on the programming task I got in the interview, and it was really amazing to me that some of the hints I'd gotten in the interview were really not at all how I wrote it when I had the chance to do it on my own. For example, I needed to take a string, and get the first character, and then the rest - very much like clojure would do it. I was given the hint that "hello".chars would give me an Array of the characters, and on their box it did - but in Ruby 1.9.2, it returns an Enumerator, and that's an entirely different thing. Also, it requires that we allocate new storage.

What I really wanted was: "hello"[0,1] - that's a string of the first character in the string. And then the rest is just: "hello"[1,10], or another long number to get the rest. This is what was sitting in the back of my memory, and when I saw it, a lot of other little things started flooding back in.

Things started accelerating, and in no time I was looking at a far cleaner version of the test code I'd written just a few days ago. Far more idiomatic Ruby, and the tests were super simple to write, and took advantage of the contextual nature of the tests.

I'm not going to claim that I'm even a decent Ruby coder. I'm just fair. The class library is just far too expansive to know it all without dealing with it on a daily basis, and even then, on a large codebase that takes advantage of the different classes as well. It's vast. Which is nice, don't get me wrong, but it's why I know it'd take me a very long time to master even where things are - let alone their syntax and usage. But it's something I may have to do, so it's nice to get back in it.

VisualVM is Bundled with the JVM

Wednesday, December 3rd, 2014

java-logo-thumb.png

I have been using VisualVM for quite a while now... it's got the monitoring tools that work pretty nicely if you have the network connectivity you need to get to the box. What I learned this morning on the #clojure channel on IRC is that it's now included in the JVM.

That's sweet! SO I had to check:

  $ which jvisualvm
  /usr/bin/jvisualvm

Really nice! It's the same tool, but now it's integrated, and that's even better.

Now I'll be able to use it on any platform and not have to hassle with downloading it as a separate install. Very nice!

Finding the Joy in Life Again

Wednesday, October 22nd, 2014

Great News

I honestly would have put money on the fact that this would not have happened today. Big money.

I'm sitting on the bus riding to work, and I realize that I'm pretty happy without a pain-causing personal relationship in my life. That was a wow! moment. I've been separated for about 2 years, and the divorce is in the works, but I would have bet real money I'd feel horrible for the rest of my natural life. But today... on the bus... for a few minutes... I didn't.

That was huge for me. Huge.

Then I'm in work, updating a few postings with the results of the tests I'd done overnight, and I'm back into the swing of posting like I used to. It's been a long two years, but I'm back to writing about what I'm doing, and it's really helping. I'm feeling like I'm enjoying myself again.

This, too, was huge for me.

I don't expect this to last all day... but the fact that I have felt this way tells me that I need to keep doing what I'm doing - keep moving forward, and then maybe this will come again. And maybe when it comes again, it'll last longer. Maybe.

Fixed Log Configs for SIngle-File Logging

Thursday, October 16th, 2014

Storm Logo

This morning I wanted to take some time to make sure that I got all the log messages into one file, and that not being a redirection of stdout or stderr. This is something I've done a few times, and it just took the time to set up the Logback config file. The reason we're using Logback is that this is what Storm uses, and since this is a Storm jar, we needed to use this style of logging.

Interestingly, the config wasn't all that hard to get a nice, daily-rotating, compressing, log file for everything I needed:

<configuration scan="true">
  <appender name="FILE"
            class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>/home/${USER}/log/experiments.log</file>
    <rollingPolicy
        class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
      <fileNamePattern>
        /home/${USER}/log/experiments_%d{yyyy-MM-dd}.log.gz
      </fileNamePattern>
      <maxHistory>30</maxHistory>
    </rollingPolicy>
    <encoder>
      <pattern>
        [%d{yyyy-MM-dd HH:mm:ss.SSS}:%thread] %-5level %logger{36} - %msg%n
      </pattern>
    </encoder>
  </appender>
 
  <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
      <pattern>
        [%d{yyyy-MM-dd HH:mm:ss.SSS}:%thread] %-5level %logger{36} - %msg%n
      </pattern>
    </encoder>
  </appender>
 
  <root level="INFO">
    <appender-ref ref="FILE" />
  </root>
</configuration>

I have to admit that this is a decent tool - if you know how to configure it properly. But I guess that's true for a lot of the Apache projects.