What's it all about, Alfie?

A couple of days ago, I found out about the licensing rules for JDK 11 from Oracle, and wanted to install OpenJDK 11 with Homebrew. Sadly, Homebrew doesn't really like the idea of installing older versions of packages, so I'm going to live with JDK 1.8, and OpenJDK 11. But that's not what this is about. 🙂 This is about Clojure, and Leiningen's REPL, and the fact that in that post I found a known problem with the current version of Ultra, and Leiningen versions greater than 2.7.1

Well... a little digging, and I find that the core component of Ultra is another project, and that same author has a similar REPL formatter and colorizer - called whidbey. Now it turns out that whidbey also has issues with Leiningen 2.9.0 - but at least it works with 2.8.3, so I can upgrade to 2.8.3, and then let them figure out the issue with nREPL 0.6.0 - as it's already an issue on whidbey.

So until then, I have both in my ~/.lein/profiles.clj, and I can flip between the two as needed:

  {
   :user {:plugins [[lein-exec "0.3.7"]
  ;                  [venantius/ultra "0.5.4"]
                    [mvxcvi/whidbey "2.0.0"]
                    ]
          :middleware [whidbey.plugin/repl-pprint]
          :whidbey {:color-scheme {:delimiter [:bold :yellow]
                                   :tag [:bold :yellow]
                                   :nil [:cyan]
                                   :boolean [:bold :cyan]
                                   :number [:bold :green]
                                   :string [:bold :red]
                                   :character [:cyan]
                                   :keyword [:yellow]
                                   :symbol [:bold :magenta]
                                   :function-symbol [:bold :blue]
                                   :class-delimiter [:blue]
                                   :class-name [:green]
                                   :exception [:bold :red]}}
          :ultra {:color-scheme ;; :solarized_dark
                   {:delimiter [:bold :yellow]
                    :tag [:bold :yellow]
                    :nil [:cyan]
                    :boolean [:bold :cyan]
                    :number [:bold :green]
                    :string [:bold :red]
                    :character [:cyan]
                    :keyword [:yellow]
                    :symbol [:bold :magenta]
                    :function-symbol [:bold :blue]
                    :class-delimiter [:blue]
                    :class-name [:green]
                    :exception [:bold :red]}}
          }
  }

At this point, I can switch to 2.8.3 with:

  $ brew switch leiningen 2.8.3
  Cleaning /usr/local/Cellar/leiningen/2.7.1
  Cleaning /usr/local/Cellar/leiningen/2.9.0
  Cleaning /usr/local/Cellar/leiningen/2.8.3
  3 links created for /usr/local/Cellar/leiningen/2.8.3

and I'm good to go with Leiningen 2.8.3, and as soon as whidbey is updated, I'll be able to move up to 2.9.0. Really glad I found this project.

[3/6] UPDATE: it looks like ultra is getting an update that will allow it to work with Leiningen 2.9.0 - which is great! I just need to wait for ultra 0.6.0 to hit Clojars, and I can update and use it. That would be very nice!

Well... today has been a very eventful day. I had noticed that Oracle dropped support for JDK 10, and mentioned it to some friends in Slack. I didn't know the details, so I decided to dig in and see if I could see anything on the net about it. I did. Lots. Changed my day.

Turns out, Oracle is changing it's licensing for JDK 11 - it's free for development and testing, but if you use it in production, it requires a license. A paid license. Yikes!

OpenJDK is also built by Oracle, and IBM, and RedHat - and it is Open Source - free for any use-case. So that's where people appear to be going. I can see that, but I've stayed away from OpenJDK because it's just the official JDK from Oracle. But now it seems Oracle is looking to cash in on the JDK, and this is just a lovely predicament.

But there is hope - kinda. Homebrew is supporting OpenJDK as a cask so that it can be versioned, and supported within the Homebrew ecosystem. That's good news. To install the latest, you simply need to:

  $ brew cask install java

and it will install it into:

  $ brew cask info java
  java: 11.0.2,9
  https://jdk.java.net/
  /usr/local/Caskroom/java/11.0.2,9 (64B)
  From: https://github.com/Homebrew/homebrew-cask/blob/master/Casks/java.rb
  ==> Name
  OpenJDK Java Development Kit
  ==> Artifacts
  jdk-11.0.2.jdk -> /Library/Java/JavaVirtualMachines/openjdk-11.0.2.jdk
  (Generic Artifact)

and the tools I built for switching JDK versions work just fine:

  $ setjdk 1.8; echo $JAVA_HOME
  /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home
  $ setjdk 11; echo $JAVA_HOME
  /Library/Java/JavaVirtualMachines/openjdk-11.0.2.jdk/Contents/Home

That's the good news. What isn't really good is that there is an issue with the Java Collections where the toArray method added an additional arity and that breaks Clojure - badly. So it really doesn't help to have the OpenJDK 11 on the box - as Clojure can't make use of it.

However, that's not all... it appears that when I upgraded Leiningen, I broke one of the really nice plugins that I like to use with it - Ultra. The error is nasty as well:

  $ lein clean
  $ lein deps
  $ lein repl
  [WARNING] No nREPL middleware descriptor in metadata of
    #'clojure.tools.nrepl.middleware.render-values/render-values,
    see nrepl.middleware/set-descriptor!
  [WARNING] No nREPL middleware descriptor in metadata of
    #'clojure.tools.nrepl.middleware.render-values/render-values,
    see nrepl.middleware/set-descriptor!
  nREPL server started on port 54967 on host 127.0.0.1 - nrepl://127.0.0.1:54967
  ERROR: Unhandled REPL handler exception processing message
    {:id ad3092dd-8491-4d60-a1a9-5092d4d090a9, :op clone}
    java.lang.IllegalArgumentException: No implementation of method:
    :send of protocol: #'nrepl.transport/Transport

and if I removed Ultra from my ~/.lein/profiles.clj then everything worked fine again. But this was with Leiningen 2.9.0 - and that was an issue. In fact, it was an issue that's already been reported to Ultra. I added my notes, as the suggestion didn't work for me.

So now I'm left with downgrading Leiningen to get Clojure working again. This is not as easy as I'd hoped, but it's what needs to be done. First, get into where the Homebrew formulae are stored, :

  $ cd $(brew --prefix)/Homebrew/Library/Taps/homebrew/homebrew-core/Formula
  $ git log --follow leiningen.rb
  commit 2be19f3787d811247095e704765fc33a8815d639
  Author: BrewTestBot <homebrew-test-bot@lists.sfconservancy.org>
  Date:   Tue Feb 12 13:07:29 2019 +0000
 
      leiningen: update 2.9.0 bottle.
 
  commit 9ca7899818eb02cc8252d47cc32b5a42554a7655
  Author: Rahul De <rahul080327@gmail.com>
  Date:   Tue Feb 12 10:43:59 2019 +0100
 
      leiningen 2.9.0
 
      Closes #36927.
 
      Signed-off-by: Chongyu Zhu <i@lembacon.com>
 
  commit fd1c0ba8f162a3c41debef6143dca8ba19227f90
  Author: BrewTestBot <homebrew-test-bot@lists.sfconservancy.org>
  Date:   Sat Dec 15 10:45:41 2018 +0000
 
      leiningen: update 2.8.3 bottle.
 
  commit 40553be0b01710db9e16cff978f593b83ecdfe3b
  Author: Rahul De <rahul080327@gmail.com>
  Date:   Sat Dec 15 11:27:08 2018 +0100
 
      Leiningen 2.8.3
 
      Closes #35151.

at this point we heed the git SHA for the version we want to move to. In the case of Leiningen, the SHA for 2.8.3 is fd1c0ba8f162a3c41debef6143dca8ba19227f90 and the SHA for 2.7.1 is cdddd1094b26d0092e0030051dac97a286bb3fc4. I started with 2.8.3, and that didn't work, so I had to go back to 2.7.1. This is a big difference.

Then make a branch for that version, and reinstall Leininden from the Ruby file:

  $ git checkout -b leiningen-2.8.3 fd1c0ba8
  $ brew reinstall ./leiningen.rb
  $ brew switch leiningen 2.8.3
  $ git checkout master

for 2.8.3 and, for 2.7.1:

  $ git checkout -b leiningen-2.7.1 cdddd109
  $ brew reinstall ./leiningen.rb
  $ brew switch leiningen 2.7.1
  $ git checkout master

At this point, we can see that both versions are installed:

  $ brew info leiningen
  leiningen: stable 2.7.1 (bottled), HEAD
  Build tool for Clojure
  https://github.com/technomancy/leiningen
  /usr/local/Cellar/leiningen/2.7.1 (9 files, 14.7MB) *
    Poured from bottle on 2019-02-18 at 13:29:34
  /usr/local/Cellar/leiningen/2.8.3 (9 files, 13MB)
    Poured from bottle on 2019-02-18 at 13:22:14

Now it's possible to run Leiningen and Ultra - again. It has to be JDK 8, and Leiningen 2.7.1... but until Ultra is fixed for the change in Leiningen, this is as far as I can go.

What was even more annoying was that my new work laptop has a very recent version of Homebrew on it, and the git repo has been trimmed, and so there is no git SHAs for the previous versions. However, with tar and copying the tarball over, I was able to get the 2.7.1 and 2.8.3 versions there, and things are now working OK.

Wow! What a mess to get back to where I thought I started the day!

It's time to make sure that my laptop has the latest version of PostgreSQL, as I was reading that they were making big strides in the windowing functions in postgreSQL 11. And with Homebrew 2.0 out, it made sense to just write it all down again, just so that I have it for reference.

The process isn't bad... dump all the databases into one file, stop the server, update Homebrew, update postgres. This gets us to the point that we are ready to rebuild the new database:

  $ pg_dumpall > dump.sql
  $ brew services stop postgresql
  $ brew update
  $ brew upgrade postgres

Now we need to move out the old database data, create a new structure, and restart the service:

  $ cd /usr/local/var
  $ mv postgres postgres.old
  $ initdb -D /usr/local/var/postgres
  $ brew services start postgresql

You then need to go back to the directory of the first command - the one where you dumped the databases, and reload them all:

  $ psql -d postgres -f dump.sql

and at this point, everything should be back and running:

  $ psql --version
  psql (PostgreSQL) 11.1
  $ psql -l
                                      List of databases
      Name     | Owner | Encoding  |    Collate     |     Ctype      |  Access privileges
  -------------+-------+-----------+----------------+----------------+---------------------
   health      | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII |
   inventory   | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII |
   northhollow | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII |
   postgres    | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII |
   template0   | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII | =c/drbob           +
               |       |           |                |                | drbob=CTc/drbob
   template1   | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII | drbob=CTc/drbob    +
               |       |           |                |                | =c/drbob           +
               |       |           |                |                | _postgres=CTc/drbob+
               |       |           |                |                | postgres=CTc/drbob
   test        | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII |
  (7 rows)

At this point you can remove the old data:

  $ rm -rf /usr/local/var/postgres.old
  $ rm dump.sql

and everything is updated. You can then use any of the normal tools, including the Apache/PHP/Postgres that Apple makes available, and Postico - a great GUI client.

This morning I saw a tweet that announced that the Cognitect AWS Client library 0.8.243 was released. This was news to me, because I've been using Amazonica for years now, and it works, but has plenty of little things that always made me write a wrapper around all the things I really needed to do.

But hey... Cognitect are the guys that own Clojure - so if there's a "preferred way" to integrate with AWS, these guys would be the ones that would do it in the best way possible. I haven't looked at the docs much, but I wanted to put all this in a post so I could come back to it the next time I needed to talk to AWS from Clojure, and give this a spin.

Quite interesting...

UPDATE: I've taken a look at it, and I find it very interesting. Rather than use the AWS Java SDK, and just wrap the Java objects in Clojure code - which isn't bad but it doesn't really make it discoverable, and you end up wondering if there isn't a simpler way to get the data. With the Cognitect scheme, you send data, and get data, and it's up to you to then read what you need. Much nicer.

I saw this on Twitter this morning:

Adam McCrea - @adamlogic: I see a lot of confusion around @heroku's free dynos, so I wrote up a guide. Let me know if you find this helpful, and please share!

https://railsautoscale.com/heroku-free-dynos

and I have to say, it's packed with a lot of really useful information about the free, hobby, and paid dynes and database plans. What I really liked was that the alternatives, and work-arounds that he presented makes it clear there are some decent alternatives to the Heroku plans, but that Heroku is still in a good place.

I still want to put some Clojure service up on Heroku someday... I just need to know what the app is, and maybe it's tied into a macOS or iOS app as well. Who knows... but I really like to see someone look into the good, and the not-so-great parts of a service provider - just to keep them honest.

A friend asked me today for the trick I've used in the past to make sure that the data in a database is immutable and versioned. This is nice because it matches what Clojure acts like, internally, and it makes it easy to see who's doing what, when. Assuming we start with a table - for the sake of argument, let's say it's customers, and looks something like:

  CREATE TABLE IF NOT EXISTS customer (
    id             uuid NOT NULL,
    version        INTEGER NOT NULL,
    as_of          TIMESTAMP WITH TIME zone NOT NULL,
    last_name      VARCHAR,
    first_name     VARCHAR,
    PRIMARY KEY (id, version, as_of)
  );

where the first three fields are really the key ones for any table to work with this scheme. You need to have a unique key, and in this case, the easiest I've found is a UUID, and then you need a version and a timestamp when that change was made. What's left in the table is really not important here, but it's something.

You then need to make an Audit table that has the same table structure, but has the string _audit added to the name:

  CREATE TABLE IF NOT EXISTS customer_audit LIKE customer including ALL;

What we then need to create is the following trigger that intercepts the INSERT and UPDATE commands on the customers table, and places the historical data into the Audit table, and the most recent version of the customer data is always kept in the customer table.

The trigger looks like this:

  --
  -- create the trigger on INSERT and UPDATE to update the version if it's
  -- not provided, and to maintain all versions in the audit table, but have
  -- the current version in the non-audit table. Importantly, NOTHING is
  -- deleted.
  --
  CREATE OR REPLACE FUNCTION audit_customer()
  RETURNS TRIGGER AS $body$
  DECLARE
    ver INTEGER;
  BEGIN
    -- get the advisory lock on this id
    perform pg_advisory_xact_lock(('x' || translate(LEFT(NEW.id::text, 18),
                                  '-', ''))::bit(64)::BIGINT);
 
    -- get the max of the existing version for the data now
    SELECT MAX(version) INTO ver
      FROM customer_audit
     WHERE id = NEW.id;
    -- and bump it up one and use that
    IF ver IS NULL THEN
      NEW.version := 1;
    ELSE
      NEW.version := ver + 1;
    END IF;
 
    -- if an update, then we need to insert the new
    IF tg_op = 'UPDATE' THEN
      -- now let's insert the old row into the audit table
      INSERT INTO customer_audit
        VALUES (NEW.*);
    elsif tg_op = 'INSERT' THEN
      -- now let's insert the new row into the audit table
      INSERT INTO customer_audit
        VALUES (NEW.*);
      -- and delete the old one in the customer table
      DELETE FROM customer
        WHERE id = NEW.id
          AND version <= ver;
    END IF;
 
    -- finally, return the row to be inserted to customer
    RETURN NEW;
  END
  $body$ LANGUAGE plpgsql;
 
  CREATE TRIGGER set_version BEFORE INSERT OR UPDATE ON customer
    FOR each ROW EXECUTE PROCEDURE audit_customer();

At this point, we can INSERT or UPDATE on customers and the previous version of that customer will be mmoved to the Audit table, and the the most recent version will be held in the customers table.

I have found this very useful, and I've put it in a gist for easy access.

The point of:

    -- get the advisory lock on this id
    perform pg_advisory_xact_lock(('x' || translate(LEFT(NEW.id::text, 18),
                                  '-', ''))::bit(64)::BIGINT);

is to get an exclusive lock on the data for a given id. This is necessary to make sure that updates from multiple services get serialized on the same data. This scheme can't ensure that there are merged changes - only that there is a sequence of changes to the table, and each one is entirely correct for the time it was entered.

So... what happens if you have a string as the primary key, and not a UUID? Well, use can use the MD5 checksum of the string as the lock indicator:

    -- get the advisory lock on a general string
    perform pg_advisory_xact_lock(('x' || md5(NEW.wiggle::VARCHAR))::bit(64)::BIGINT);

where the field wiggle is a varchar, and here, we are just computing the MD5, and using that as the basis of the lock. Yes, there could be some hash collisions, but that's likely not a huge performance problem, and it's conservative in that we'll over-lock, and not under-lock.

UPDATE: a friend asked about using an int as the primary key, and in that case, the line would be:

    -- get the advisory lock on a general string
    perform pg_advisory_xact_lock(NEW.id::BIGINT);

where the column id is an int. Again, we just need to get it to the bigint for the advisory lock call. After that, Postgres does the rest.

It's Day 11 of Advent of Code, and I have to say that I'm having a great time this year. It helps to have a little spare time as they are working on the rearrangement of the groups with the acquisition, so I've got a little spare time most days.

I honestly expected to be using Swift, Clojure, ObjC - all mixed in there. This was the advantage of getting CodeRunner a little bit ago - that I could mix-n-match language to the problem based on the needs of each problem. But I have to say that I'm enjoying using Clojure 100% so far this year. It's just so good at what I need it to do... it's hard to find a need for another language.

I've also really enjoyed the times when the solution for the first part of the day is fine with my original implementation, but then the second part requires that I really re-think the solution and come up with a much more performant solution. Not always easy, but when I get something that really changes the scope of what can be done, it's a lot of fun.

One of the guys at The Shop pointed out that I'm doing the problems like Programming Golf - where the minimal solution is the best. And that's exactly what I enjoy about these - it's about a minimalist approach to the problems. What fun. 🙂

This morning, I wanted to get the first version of Cap'n Hook published on Clojars, but I needed to get a few things cleaned up. First, I needed to get the latest changes to durable-queue up to Clojars, as there were some dependencies I need to clear up - most notably in the nippy and byte-streams packages, so that was fixed up and a simple publish, and I'm up on Clojars.

At this point, I'm a little surprised that I have to type in my GPG passphrase each time I sign the jar for publishing. I can see the logic that I've read online - basically, signing is a fairly infrequent task, and needs to be done with care, so ask for the passphrase on those occasions. OK... I get it, but I'd still like to have it in my Keychain, and then just lock it up there. No need to keep typing it in.

I need to pay closer attention to the passphrase dialog on the next publish, and see if I can't get it to save the passphrase in the Keychain. But it's not horrible at this time - I don't publish all that often.

After I got durable-queue up, I could then clean up the capn-hook code a little and then publish it. Again, not bad - cleared out a few dependencies to streamline things just a touch, and then update the README.md for the folks coming to the GitHub page, and away it goes.

Not a bad morning - got a little more familiar with GPG, and published a few things. Good enough.

This morning I've finally put the docs and a few tests on my Clojure library called Cap'n Hook - after the cereal, and the fact that it's all about making web hooks easier to implement in services. When I was looking for my current position, I was thinking what could I write that would be fun and useful at the same time. One of the things that I knew were really useful were the web hooks that allowed one-way, asynchronous, message passing from a service to a set of registered clients.

While it's not really a good replacement for a real messaging system like any of the messaging offerings from AWS, or Tibco, or even Java MQ - it is very easy to implement, and if you're in the world of RESTful services, it's a really handy thing to have, as it's super simple to implement.

Also, if you put in retries, as I have, then it's pretty close to reliable - but only if your receiver is online and taking POST messages on that URL. Sure... it's not perfect, but for dealing with simple messaging between data centers, it's hard to beat. There are a ton of services (GitHub, etc.) that are using this to fire-off events on the client-side of things.

So it's pretty simple to use - the key is that for most cases, you are going to need to have some persistent set of registered URLs, These are the "endpoints" for the HTTP POST calls that this library will be sending in response to the main application saying "Here you go!".

I was a little blocked for several months on this - because I wanted to be able to implement a shared, persistent, reliable storage for the registrations - and then yesterday I realized that was just a bad idea. Each application that uses this will want to implement their own registration their way - and while I implemented a simplified registration that wasn't shared or persistent, the ideas are there, and it's easy to slot in a different registration scheme at any time.

I can see using redis to store the URLs - it has all the primitives to do what's needed. And there's Postgres - or any database, really, and I even cover that in one of the examples in the README.md on the GitHub repo. So there's no need to really solve the registration problem for everyone. Just give the users a simple one to get going, and then they can get as fancy as they need to.

I spent a little time on writing some unit tests, but it's going to be hard to really test the sending as it really needs to hit something, but I'll spend a little time today seeing if there isn't an "echo" test site out there for just this kind of testing, and then I can test sending the POST messages to see if they work.

But it's been a lot of fun to get this done and into testing... The documentation was great fun, and it just was a great little thing to finish.

This morning, I really wanted to see if I could get Clojure running in CodeRunner. It's a pretty nice lightweight development environment which is really targeted at scripting or prototyping ideas. The docs say it can do a full IDE for web development, and that's certainly possible - but I'm not sure that I'd use it for that, as there are likely far more dependencies - like the RESTful service, that need to be included, and that's just too much.

While CodeRunner has a lot of language support built-in, it doesn't handle Clojure, and that's the one language that I really wish they would have. So much so that I emailed the author to ask about adding it. When I didn't get a response, I decided to see if I couldn't do it on my own.

Turns out, it's not that hard.

Start by having Leiningen installed, and that you can get from Homebrew - which is a good idea to have installed on a Mac anyway. Assuming you have Homebrew installed, you simply need to say:

  $ brew install leiningen

and it'll be downloaded and ready to go.

Then you can look at the lein-exec GitHub page for the rest of the instructions as to how to get it going. You need to update your ~/.lein/profiles.clj file to include:

  {:user {:plugins [[lein-exec "0.3.7"]]}}

and then in your ~/bin directory, or really any place in your PATH you need to create the following two files - copied from the GitHub repo. First is lein-exec containing:

  #!/bin/bash
 
  if [[ $1 =~ ^[~\/] ]]
  then
    # its already an absolute path
    lein exec "$@"
  else
    # This is a relative path, so make it absolute,
    # using the current directory as the base.
    lein exec "`pwd`/$@"
  fi

and the second is lein-exec-p, which contains:

  #!/bin/bash
 
  if [[ $1 =~ ^[~\/] ]]
  then
    # its already an absolute path
    lein exec -p "$@"
  else
    # This is a relative path, so make it absolute,
    # using the current directory as the base.
    lein exec -p "`pwd`/$@"
  fi

At this point, you can make a script that looks like:

  #!/bin/bash lein-exec

  (prn (* 4 6))

and when you run it, you will get 24 on the console. Not bad.

In order to get this running in CodeRunner, we need to add a new language to the preferences. I chose to duplicate Perl as that's very close to what I needed. I changed the settings to look like:

and the Templates I deleted, and the Docs I added the Dash docs, as I have that installed on my mac, and it is very good:

Once I had that, I could create a script and run it:

While not perfect, it's exactly what I wanted to be able to do, and it does it perfectly.