What's it all about, Alfie?

Over the course of the last day and a half, I've been putting together a nice little test app using clojure and compojure on the back-end, and Bootstrap with Handsontable as the UI. It's simple, and not all that complicated, but it's nice in that it demonstrates a lot of the things that I often want to do in a simple application.

Then today one of the guys I was putting this together for suggested OAuth2 for authentication and then something internal for authorization. This turned out to be not all that bad with Google's Identity Service. They have a nice little JavaScript client that gets the auth token that you then feed to the calls to the service, and it, in turn, hits Google to make sure that this is a person that they say they are. You end up with an email address, and that can be used as the "username" for any authorization scheme you want.

In short, it's very slick. Works like a dream. I'm having a pretty good time with this.

In Postgres 9.4, there are several JSONB operators that include a ? as part of the operator. This is nasty for the JDBC usage because the JDBC driver uses the question-mark as the substitute parameter place-holder for arguments. And there's no escaping of values in a JDBC PreparedStatement so what's a guy to do?

Sure, you can make a simple custom function to do this, but that's kinda wasteful, and it's not very transportable. Thankfully, I was reading the release notes for the 9.4 JDBC driver and saw this.

Using the latest, released JDBC driver in your project.clj file:

  [org.postgres/postgres "9.4-1201-jdbc41"]

you can pick up this pull request, and feature.

By simply doubling the question mark, it'll now reduce this to one ? and the operator will work. Very slick. So now in a clojure SQL statement, just put ?? when you need ? and you're good to go!

Fantastic! I've checked it and it works perfectly.

The default JDBC behavior in clojure.java.jdbc is not to handle the additional Postgres data types like Array and JSON. But you do have the protocols that you can use to override the behavior of how to store and read things - and this article looks to set this up for both these data types.

The idea is to look at the data type on both the write and read, and do the JSON mapping and we're done. The code is very simple:

  (ns our-app.jdbc.json
    "Inspired by http://www.niwi.be/2014/04/13/
     postgresql-json-field-with-clojure-and-jdbc/"
    (:require [clojure.java.jdbc :as jdbc]
              [cheshire.core :as json])
    (:import org.postgresql.util.PGobject))
 
  (defn value-to-json-pgobject [value]
    (doto (PGobject.)
      (.setType "json")
      (.setValue (json/generate-string value))))
 
  (extend-protocol jdbc/ISQLValue
    clojure.lang.IPersistentMap
    (sql-value [value] (value-to-json-pgobject value))
 
    clojure.lang.IPersistentVector
    (sql-value [value] (value-to-json-pgobject value)))
 
    clojure.lang.IPersistentList
    (sql-value [value] (value-to-json-pgobject value)))
 
    clojure.lang.LazySeq
    (sql-value [value] (value-to-json-pgobject value)))
 
  (extend-protocol jdbc/IResultSetReadColumn
    PGobject
    (result-set-read-column [pgobj metadata idx]
      (let [type  (.getType pgobj)
            value (.getValue pgobj)]
        (case type
          "json" (json/parse-string value true)
          :else value))))

Where I have chosen to use Cheshire, as opposed to clojure.data.json. It's a matter of taste, I'll agree, but we're using Cheshire all the time anyway, so the cost is nothing, and it's consistently parsed in the same way.

Very cool.

Yesterday I got a quick note from an ex-co-worker about how he might go about writing a micro-service in Clojure. Seems his CTO is a big functional programmer, and as such, he wanted my friend to write something in Clojure. So he reached out to the only person he knew that had actually done that, and so I got the call.

I was happy to help him along. I sent him links to Gists about how a server.clj file might look, as well as how to handle JSONP - assuming that the calls were coming from jQuery, and basically, just make sure that things got off on the right foot.

I also gave him some pointers about the standard libraries that I've been using for clojure work - cli-time, composure, cheshire, carmine, etc. They make it a lot easier, and a lot of folks are using them - so it's good to get started using the standard stuff right off the bat.

We talked a lot about getting started, and how the RESTful stuff was going to look, and then he said he'd dig into it when he had time. But the most impressive part was the recognition that by simply saying "Clojure Developers Wanted" - the CTO knew he'd get all kinds of hits, and all kinds of street cred to make the most simpler.

Smart man... deserves to be in that position.

This afternoon I saw a tweet from Heroku about them adding Redis to the add-ons for their service. This just a few days after their announcement that Postgres was available for the free tier, and the new "Free" tier for apps. They are getting aggressive with what services they are providing. This makes a ton of good sense to me, as I'm a huge fan of redis from clojure, and this makes all the experience I've got in building apps directly transferable.

While I know the Free tier isn't all that great, the idea that there is a Free tier is amazing, and it means that I can write something and throw it up there, and as it's needed, I can scale it up. Very cool. They also have a hobbyist tier that's only something like $8/mo. - similar to GitHub.

If I needed to be firing up a web service, it'd be clojure, redis, and postgres - all on Heroku. What an amazing service.

A really nice co-worker of mine has been doing some interesting work with random number generators for the test.check clojure library, and I decided this morning to have a look and see what it's all about. After all, I can't imagine something a lot better than the standard clojure.test as it's got everything I've ever needed. But Bret has used it, and Gary has to have started using it for a reason, so I wanted to find out what that was.

I'm very glad I did.

The problem that I've run into with simple unit test frameworks is that they allow you to test specific inputs and outputs. If you want something to run 100 times with random inputs, you could do it, but it wasn't easy. Certainly, the question as to Why? would arise unless you had a testing framework that thought not in terms of specific inputs and outputs - but in the structure of the data.

If we wanted to test a sorting function, then we could check to see if the act of operating on a sorted sequence was the same as the input sequence, then we'd need to be able to make arbitrary sequences, and then verify the structure of the data as well. This is then more like generators and properties - which is what test.check calls them.

In this regard, it's easy to see how you'd use this to test the limits of a function without worrying about making sure you got all the edge conditions in the unit tests. It's the shotgun approach, and in some cases, it can really make short work of finding the edge cases if it's not immediately easy to see them from the code.

There's even a simple macro to allow these tests to run alongside the clojure.test unit tests - which is exactly how I can see using it. There are some things that the unit testing system really works well for, and there are others that a shotgun approach would really help for as well. So a hybrid approach seems to be the best move.

Certainly interesting, and I'll put it into a project soon.

OK, since I was on a roll with the work I was doing at The Shop, it made sense to keep going and apply it to putting a web front-end on the CryptoQuip solver a friend and I have been working on. The solver is all Clojure, and so it took just a few tweaks to get all the static files in-place and ready, and then getting the POST working was really the challenge.

Turns out, the way to get it to work was:

  function solveThePuzzle() {
    console.log("attempting to solve the puzzle");
    // get what we need for the call...
    var inp = {};
    inp.cyphertext = $("#plaintext").val();
    inp.clue = {};
    inp.clue[$("#key1").val()] = $("#key2").val();
    // make the call to solve the puzzle
    $.ajax({type: "POST",
            url: "/solve",
            processData: false,
            contentType: 'application/json',
            data: JSON.stringify(inp),
            success: function(resp) {
              var cont = '<div class="alert alert-success" role="alert">';
              cont += '<strong>Solved:</strong> ' + resp.plaintext;
              cont += '</div>';
              $("#status").replaceWith(cont);
            }
    })
  }

and the results are very nice:

It had to happen... JDK 7 is end-of-life by Oracle, and while I imagine that this is not what is really going to happen to all the JDK 7 code out in the wild, it's enough of a pressure that it's time to move off it and onto JDK 8 for real.

I've got a nice switcher tool for the active version of the JDK for each shell, and I just switched it to 1.8, and restarted the REPL and web server to pick up the changes. It's not a ton of work, but it needed to be done, and I need to stay on JDK 8 moving forward.

It's funny... in writing this post, I realized that I can remember when JDK 1.5 was "new" and untested... and 1.6... and 1.7... it's just how things go. Marching onward.

This morning I didn't have a lot going on, and I decided to upgrade my laptop from Posgres 9.3.4 to 9.4.1 as there are a few little things in 9.4 that are nice, and I've got 9.4 on my work laptop, and I figured this would be an easy upgrade - like super easy... I was mistaken.

The rules about automatic upgrades for Postgres is a bug release version change. I thought it was a minor release. So I was expecting to simply shut down the server, upgrade the packages with Homebrew, and then start it back up. The code would detect that it was the next minor version, and automatically update the data. Sadly, that's not the case. It's a big upgrade, and that means that I might as well do a complete dump/load.

Sadly, I didn't do a dump, so I'd have to live with an older version. Not a tragedy, but annoying when I'm in the middle of the upgrade process only to learn that it's not going to work. So it goes...

So here's what I had to do - in the right order to get things working. Not bad, but it's basically the instructions for a dump/load, so I'll assume we know this going in.

First, create a complete dump of the database. Assuming that all these things are installed on Mac OS X, and using Homebrew, the paths are not important - they are all fixed with Homebrew, anyway.

  $ pg_dumpall > dump_file

Next, shut down the running server, update Homebrew, and then upgrade postgres within Homebrew. Just to be safe, let's re-link the launchctl file because in this case, it has changed, and better safe than sorry.

  $ launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
 
  $ brew update
 
  $ brew upgrade postgresql
  $ ln -sfv /usr/local/opt/postgresql/*.plist ~/Library/LaunchAgents

At this point I need to move the old database data to the side, and then initialize the database with the new codebase. Once that's done, we can then restart it with the re-linked launchctl file.

  $ cd /usr/local/var
  $ mv postgres postgres.old
 
  $ initdb -D /usr/local/var/postgres
 
  $ launchctl load ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist

Finally, we need to load up the entire database from the dump file we made in the first step.

  $ psql -d postgres -f /path/to/dump_file

Check and make sure that everything looks OK and then you can easily remove the old database directory:

  $ rm -rf /usr/local/var/postgres.old

That's it.

I was working on stripping out a little bit of code from a project at The Shop today, and when I stripped out the library I had made, I got the following error when trying to start the REPL:

  $ lein repl
  Exception in thread "main" java.lang.NoClassDefFoundError:
  clojure/tools/logging/impl/LoggerFactory, compiling:
  (/private/var/folders/ct/jhkds06j26v1lq2t40jx4cndl_629q/T/
  form-init4416651774354867948.clj:1:124)
      at clojure.lang.Compiler.load(Compiler.java:7142)
      at clojure.lang.Compiler.loadFile(Compiler.java:7086)
      at clojure.main$load_script.invoke(main.clj:274)
      at clojure.main$init_opt.invoke(main.clj:279)
      at clojure.main$initialize.invoke(main.clj:307)
      at clojure.main$null_opt.invoke(main.clj:342)
      at clojure.main$main.doInvoke(main.clj:420)

And if I put the library in the project.clj, I don't get this error, but if I take it out, I get this error. And I needed to take it out.

When I did a lien reps :tree to see what overlaps there might be in the libraries, I found a few in the Google AdWords libraries. Easily enough, I took them out, and forced the right version with a simple:

  ;; AdWords Java API
  [commons-lang "2.6"]
  [com.google.api-ads/ads-lib "1.38.0" :exclusions [commons-lang
                                                    org.slf4j/slf4j-api]]
  [com.google.api-ads/adwords-axis "1.38.0" :exclusions [commons-lang
                                                         org.slf4j/slf4j-api]]

because the conflict was in the Google jars and both 2.5 and 2.6 of commons-lang were being used. Simply exclude them both, reference 2.6 first, and that should take care of it.

But it didn't.

So I took to the Google and found that someone had solved this by putting the class in the :apt section of the project.clj file. So with that, I tried:

  :aot [clojure.tools.logging.impl bartender.main]

and then things started working just fine again.

I'm guessing that by building the uberjar for the other library, it did this compilation, so that it wasn't necessary for this project. Take it out, and we have a problem. Force the compile first, and it's all good.

Glad I figured that out.