What's it all about, Alfie?

Today I've been having an email exchange with an old co-worker about an Open Source project that he's been working on - actually, it's pretty much done. It's all in Python, so it's not something that I'm really keen to use, but the design is good, and I can remember using something like it back at a previous job where we had to solve complex data modeling issues in as near real-time as possible.

The project is called dataflow and is really a compute/modeling system where you define Nodes and these nodes can be simple calculations on inputs, but they can be quite advanced - say rolling time averages, etc. and the key is that these Nodes can relate to one another in a simple Graph, that the user can describe, and then when any Node is updated, it fires off the updates of the dependent Nodes.

The nice thing is that these Nodes have a very simple API, and so they can do just about anything - averages of inputs, calculations, mean-square calls... it's really what the user wants to see. Tie this with a UI that displays the data, and you get something that is very powerful indeed. It can be a a Risk system where each Node can aggregate the child Nodes so that you can look across account, symbol, etc. and all updates as the input data updates.

It's been a nice reminder of some of the work I'd done at that job - but the work I did was in Java, and it was a bit more limited, but it still enabled some pretty nice apps to be built. Good stuff.

Well... today has been a very eventful day. I had noticed that Oracle dropped support for JDK 10, and mentioned it to some friends in Slack. I didn't know the details, so I decided to dig in and see if I could see anything on the net about it. I did. Lots. Changed my day.

Turns out, Oracle is changing it's licensing for JDK 11 - it's free for development and testing, but if you use it in production, it requires a license. A paid license. Yikes!

OpenJDK is also built by Oracle, and IBM, and RedHat - and it is Open Source - free for any use-case. So that's where people appear to be going. I can see that, but I've stayed away from OpenJDK because it's just the official JDK from Oracle. But now it seems Oracle is looking to cash in on the JDK, and this is just a lovely predicament.

But there is hope - kinda. Homebrew is supporting OpenJDK as a cask so that it can be versioned, and supported within the Homebrew ecosystem. That's good news. To install the latest, you simply need to:

  $ brew cask install java

and it will install it into:

  $ brew cask info java
  java: 11.0.2,9
  https://jdk.java.net/
  /usr/local/Caskroom/java/11.0.2,9 (64B)
  From: https://github.com/Homebrew/homebrew-cask/blob/master/Casks/java.rb
  ==> Name
  OpenJDK Java Development Kit
  ==> Artifacts
  jdk-11.0.2.jdk -> /Library/Java/JavaVirtualMachines/openjdk-11.0.2.jdk
  (Generic Artifact)

and the tools I built for switching JDK versions work just fine:

  $ setjdk 1.8; echo $JAVA_HOME
  /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home
  $ setjdk 11; echo $JAVA_HOME
  /Library/Java/JavaVirtualMachines/openjdk-11.0.2.jdk/Contents/Home

That's the good news. What isn't really good is that there is an issue with the Java Collections where the toArray method added an additional arity and that breaks Clojure - badly. So it really doesn't help to have the OpenJDK 11 on the box - as Clojure can't make use of it.

However, that's not all... it appears that when I upgraded Leiningen, I broke one of the really nice plugins that I like to use with it - Ultra. The error is nasty as well:

  $ lein clean
  $ lein deps
  $ lein repl
  [WARNING] No nREPL middleware descriptor in metadata of
    #'clojure.tools.nrepl.middleware.render-values/render-values,
    see nrepl.middleware/set-descriptor!
  [WARNING] No nREPL middleware descriptor in metadata of
    #'clojure.tools.nrepl.middleware.render-values/render-values,
    see nrepl.middleware/set-descriptor!
  nREPL server started on port 54967 on host 127.0.0.1 - nrepl://127.0.0.1:54967
  ERROR: Unhandled REPL handler exception processing message
    {:id ad3092dd-8491-4d60-a1a9-5092d4d090a9, :op clone}
    java.lang.IllegalArgumentException: No implementation of method:
    :send of protocol: #'nrepl.transport/Transport

and if I removed Ultra from my ~/.lein/profiles.clj then everything worked fine again. But this was with Leiningen 2.9.0 - and that was an issue. In fact, it was an issue that's already been reported to Ultra. I added my notes, as the suggestion didn't work for me.

So now I'm left with downgrading Leiningen to get Clojure working again. This is not as easy as I'd hoped, but it's what needs to be done. First, get into where the Homebrew formulae are stored, :

  $ cd $(brew --prefix)/Homebrew/Library/Taps/homebrew/homebrew-core/Formula
  $ git log --follow leiningen.rb
  commit 2be19f3787d811247095e704765fc33a8815d639
  Author: BrewTestBot <homebrew-test-bot@lists.sfconservancy.org>
  Date:   Tue Feb 12 13:07:29 2019 +0000
 
      leiningen: update 2.9.0 bottle.
 
  commit 9ca7899818eb02cc8252d47cc32b5a42554a7655
  Author: Rahul De <rahul080327@gmail.com>
  Date:   Tue Feb 12 10:43:59 2019 +0100
 
      leiningen 2.9.0
 
      Closes #36927.
 
      Signed-off-by: Chongyu Zhu <i@lembacon.com>
 
  commit fd1c0ba8f162a3c41debef6143dca8ba19227f90
  Author: BrewTestBot <homebrew-test-bot@lists.sfconservancy.org>
  Date:   Sat Dec 15 10:45:41 2018 +0000
 
      leiningen: update 2.8.3 bottle.
 
  commit 40553be0b01710db9e16cff978f593b83ecdfe3b
  Author: Rahul De <rahul080327@gmail.com>
  Date:   Sat Dec 15 11:27:08 2018 +0100
 
      Leiningen 2.8.3
 
      Closes #35151.

at this point we heed the git SHA for the version we want to move to. In the case of Leiningen, the SHA for 2.8.3 is fd1c0ba8f162a3c41debef6143dca8ba19227f90 and the SHA for 2.7.1 is cdddd1094b26d0092e0030051dac97a286bb3fc4. I started with 2.8.3, and that didn't work, so I had to go back to 2.7.1. This is a big difference.

Then make a branch for that version, and reinstall Leininden from the Ruby file:

  $ git checkout -b leiningen-2.8.3 fd1c0ba8
  $ brew reinstall ./leiningen.rb
  $ brew switch leiningen 2.8.3
  $ git checkout master

for 2.8.3 and, for 2.7.1:

  $ git checkout -b leiningen-2.7.1 cdddd109
  $ brew reinstall ./leiningen.rb
  $ brew switch leiningen 2.7.1
  $ git checkout master

At this point, we can see that both versions are installed:

  $ brew info leiningen
  leiningen: stable 2.7.1 (bottled), HEAD
  Build tool for Clojure
  https://github.com/technomancy/leiningen
  /usr/local/Cellar/leiningen/2.7.1 (9 files, 14.7MB) *
    Poured from bottle on 2019-02-18 at 13:29:34
  /usr/local/Cellar/leiningen/2.8.3 (9 files, 13MB)
    Poured from bottle on 2019-02-18 at 13:22:14

Now it's possible to run Leiningen and Ultra - again. It has to be JDK 8, and Leiningen 2.7.1... but until Ultra is fixed for the change in Leiningen, this is as far as I can go.

What was even more annoying was that my new work laptop has a very recent version of Homebrew on it, and the git repo has been trimmed, and so there is no git SHAs for the previous versions. However, with tar and copying the tarball over, I was able to get the 2.7.1 and 2.8.3 versions there, and things are now working OK.

Wow! What a mess to get back to where I thought I started the day!

This morning I was working to upgrade my laptop from Postgres 10.3 to 11.1, and I did the following, as part of the upgrade:

  $ brew update
  fatal: could not read Username for 'https://github.com': terminal
  prompts disabled
  Error: Fetching /usr/local/Homebrew/Library/Taps/homebrew
  /homebrew-boneyard failed!

I did a lot of googling, and came up with a couple of ideas. This was from GitHub:

  $ git config --global credential.helper osxkeychain

but that didn't work. Then I found another reference saying that the 2FA on GitHub was the problem, and that you needed to bypass that with:

  $ git config --global --add url."git@github.com:".insteadOf "https://github.com/"

and still that didn't work. So I took a different approach, I started looking at the Homebrew files on my laptop - specifically, the boneyard mentioned in the error. Turns out... if you looked at the .git/config file in that directory, it points to a directory in GitHub that no longer exists! OK... this might be something.

So I removed the nearly empty directory, and tried again:

  $ brew update
  Already up-to-date.

Success! It wasn't the config of git - it was that Homebrew was trying to update something that no longer existed. Lesson learned.

It's time to make sure that my laptop has the latest version of PostgreSQL, as I was reading that they were making big strides in the windowing functions in postgreSQL 11. And with Homebrew 2.0 out, it made sense to just write it all down again, just so that I have it for reference.

The process isn't bad... dump all the databases into one file, stop the server, update Homebrew, update postgres. This gets us to the point that we are ready to rebuild the new database:

  $ pg_dumpall > dump.sql
  $ brew services stop postgresql
  $ brew update
  $ brew upgrade postgres

Now we need to move out the old database data, create a new structure, and restart the service:

  $ cd /usr/local/var
  $ mv postgres postgres.old
  $ initdb -D /usr/local/var/postgres
  $ brew services start postgresql

You then need to go back to the directory of the first command - the one where you dumped the databases, and reload them all:

  $ psql -d postgres -f dump.sql

and at this point, everything should be back and running:

  $ psql --version
  psql (PostgreSQL) 11.1
  $ psql -l
                                      List of databases
      Name     | Owner | Encoding  |    Collate     |     Ctype      |  Access privileges
  -------------+-------+-----------+----------------+----------------+---------------------
   health      | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII |
   inventory   | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII |
   northhollow | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII |
   postgres    | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII |
   template0   | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII | =c/drbob           +
               |       |           |                |                | drbob=CTc/drbob
   template1   | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII | drbob=CTc/drbob    +
               |       |           |                |                | =c/drbob           +
               |       |           |                |                | _postgres=CTc/drbob+
               |       |           |                |                | postgres=CTc/drbob
   test        | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII |
  (7 rows)

At this point you can remove the old data:

  $ rm -rf /usr/local/var/postgres.old
  $ rm dump.sql

and everything is updated. You can then use any of the normal tools, including the Apache/PHP/Postgres that Apple makes available, and Postico - a great GUI client.

I've been rebuilding a new MacBook Pro laptop from work because the security tools we use at The Shop don't allow Migration Assistant to work - which is sad, because it works so well under normal circumstances, but water under the bridge, and I have to reinstall a lot of things on the box - as if from scratch.

Because of Clojure, one of the things I have to make sure to install is the latest JDK. Actually, I need to install all the latest JDKs - 1.8, 9, 10, and 11. And then I have a simple little tool for changing the JDK version that I want to be using. Simple. Easy.

When I last went to upgrade the versions of the JDKs, I found that Oracle has dropped support for JDK 9. I had heard a lot of talk about how JDK 9 wasn't really doing well because of some of the design decisions... and so this wasn't really a surprise, but it was unexpected to see that Oracle had pulled it from the Downloads. But OK... they didn't want people using it, so they pulled it.

Today, I saw that Oracle had pulled JDK 10 as well. Now that was a bit of a surprise. I know that it wasn't as popular with the Clojure community as JDK 1.8, but to see a second abandoned JDK version... well... that's a heck of a realization from Oracle. It's not good... it's causing us too much to support it... drop it.

So I have JDK 1.8 and 11 on the new laptop, even though on my personal machine I still have the last available JDK 10. But at some point, that will have to be thinned out because it just won't be supported. Wild.

Hats off to the JDK 1.8 guys - You've made a heck of a product.

This morning I saw a tweet that announced that the Cognitect AWS Client library 0.8.243 was released. This was news to me, because I've been using Amazonica for years now, and it works, but has plenty of little things that always made me write a wrapper around all the things I really needed to do.

But hey... Cognitect are the guys that own Clojure - so if there's a "preferred way" to integrate with AWS, these guys would be the ones that would do it in the best way possible. I haven't looked at the docs much, but I wanted to put all this in a post so I could come back to it the next time I needed to talk to AWS from Clojure, and give this a spin.

Quite interesting...

UPDATE: I've taken a look at it, and I find it very interesting. Rather than use the AWS Java SDK, and just wrap the Java objects in Clojure code - which isn't bad but it doesn't really make it discoverable, and you end up wondering if there isn't a simpler way to get the data. With the Cognitect scheme, you send data, and get data, and it's up to you to then read what you need. Much nicer.

A friend asked me today for the trick I've used in the past to make sure that the data in a database is immutable and versioned. This is nice because it matches what Clojure acts like, internally, and it makes it easy to see who's doing what, when. Assuming we start with a table - for the sake of argument, let's say it's customers, and looks something like:

  CREATE TABLE IF NOT EXISTS customer (
    id             uuid NOT NULL,
    version        INTEGER NOT NULL,
    as_of          TIMESTAMP WITH TIME zone NOT NULL,
    last_name      VARCHAR,
    first_name     VARCHAR,
    PRIMARY KEY (id, version, as_of)
  );

where the first three fields are really the key ones for any table to work with this scheme. You need to have a unique key, and in this case, the easiest I've found is a UUID, and then you need a version and a timestamp when that change was made. What's left in the table is really not important here, but it's something.

You then need to make an Audit table that has the same table structure, but has the string _audit added to the name:

  CREATE TABLE IF NOT EXISTS customer_audit LIKE customer including ALL;

What we then need to create is the following trigger that intercepts the INSERT and UPDATE commands on the customers table, and places the historical data into the Audit table, and the most recent version of the customer data is always kept in the customer table.

The trigger looks like this:

  --
  -- create the trigger on INSERT and UPDATE to update the version if it's
  -- not provided, and to maintain all versions in the audit table, but have
  -- the current version in the non-audit table. Importantly, NOTHING is
  -- deleted.
  --
  CREATE OR REPLACE FUNCTION audit_customer()
  RETURNS TRIGGER AS $body$
  DECLARE
    ver INTEGER;
  BEGIN
    -- get the advisory lock on this id
    perform pg_advisory_xact_lock(('x' || translate(LEFT(NEW.id::text, 18),
                                  '-', ''))::bit(64)::BIGINT);
 
    -- get the max of the existing version for the data now
    SELECT MAX(version) INTO ver
      FROM customer_audit
     WHERE id = NEW.id;
    -- and bump it up one and use that
    IF ver IS NULL THEN
      NEW.version := 1;
    ELSE
      NEW.version := ver + 1;
    END IF;
 
    -- if an update, then we need to insert the new
    IF tg_op = 'UPDATE' THEN
      -- now let's insert the old row into the audit table
      INSERT INTO customer_audit
        VALUES (NEW.*);
    elsif tg_op = 'INSERT' THEN
      -- now let's insert the new row into the audit table
      INSERT INTO customer_audit
        VALUES (NEW.*);
      -- and delete the old one in the customer table
      DELETE FROM customer
        WHERE id = NEW.id
          AND version <= ver;
    END IF;
 
    -- finally, return the row to be inserted to customer
    RETURN NEW;
  END
  $body$ LANGUAGE plpgsql;
 
  CREATE TRIGGER set_version BEFORE INSERT OR UPDATE ON customer
    FOR each ROW EXECUTE PROCEDURE audit_customer();

At this point, we can INSERT or UPDATE on customers and the previous version of that customer will be mmoved to the Audit table, and the the most recent version will be held in the customers table.

I have found this very useful, and I've put it in a gist for easy access.

The point of:

    -- get the advisory lock on this id
    perform pg_advisory_xact_lock(('x' || translate(LEFT(NEW.id::text, 18),
                                  '-', ''))::bit(64)::BIGINT);

is to get an exclusive lock on the data for a given id. This is necessary to make sure that updates from multiple services get serialized on the same data. This scheme can't ensure that there are merged changes - only that there is a sequence of changes to the table, and each one is entirely correct for the time it was entered.

So... what happens if you have a string as the primary key, and not a UUID? Well, use can use the MD5 checksum of the string as the lock indicator:

    -- get the advisory lock on a general string
    perform pg_advisory_xact_lock(('x' || md5(NEW.wiggle::VARCHAR))::bit(64)::BIGINT);

where the field wiggle is a varchar, and here, we are just computing the MD5, and using that as the basis of the lock. Yes, there could be some hash collisions, but that's likely not a huge performance problem, and it's conservative in that we'll over-lock, and not under-lock.

UPDATE: a friend asked about using an int as the primary key, and in that case, the line would be:

    -- get the advisory lock on a general string
    perform pg_advisory_xact_lock(NEW.id::BIGINT);

where the column id is an int. Again, we just need to get it to the bigint for the advisory lock call. After that, Postgres does the rest.

This morning I saw a tweet from SubEthaEdit that they were Open Sourcing the editor - and the current version, SubEthaEdit 5, was still on the Mac App Store, and would be free. This was a real surprise to me. I've paid for several of the versions of this collaborative editor on the Mac - heck, I've written syntax highlighting definition files for Make and Fortran for this editor. It's a big part of my tools in the past.

I have worked with my good friend on Macs for many years, and when this first appeared, as Hydra, I thought that this would be a great tool for working on code with him. But it was commercial, and we were in different states, and we hadn't even started using Git - and GitHub wasn't even an idea at the time. So it just fizzled out.

But at several times in the last 5 years we've both talked about getting something like this going for remote pair coding. It's just an editor, and he's now using Cursive for his Clojure coding, so again, maybe it's not such a great fit... and there are other services that are going for an add-in mode for existing editors, so maybe it needs to be updated to really find it's market. If so, I think that would be great.

I hope it finds a great group of developers now that it's Open Source. I'd love to have a good tool that's really written to handle the collaborative editing from the jump. Then again, I'm not all that sure what we'd need above GitHub... but it's an admirable goal.

Today I was looking for a solution to a problem I saw in the syntax highlighting of Sublime Text 3's Markdown files when there were Clojure code blocks in the file. All the other coding blocks I had been using were highlighting with a different background, and the text in the block was highlighted according to that language's rules. But not so with Clojure.

So I asked on the Sublime Text Forums about the issue and to my amazement, I got a response! The response was clear about what I needed to do, and while there was a slight issue with the installation of a package, I solved that with a git clone in the directory, and I was in business.

At the time, I also submitted an issue with the Sublime Text GitHub group, and let them know I was having a problem. Very much like the issue on the Forum post. When I got an answer on the Forums, I updated the GitHub issue, and included the answer I got from the Forum guy. He didn't sound like he was tied into the GitHub group.

Mistake #1.

I then got a response on the Issue kinda poking me to make a PR because I have an answer that worked for me. And me, trying to be helpful to the folks that helped me, wanted to respond. It was just the tone of the request that was a little off. I should have listened to that voice.

Mistake #2.

So I made the Pull Request on the project and put in the comments, etc. and submitted it. At this point, I really want to thank GitHub for making a tool that has the most amazing workflow I've ever used. The ability to fork, and make a PR on a GitHub repo, and have the fork linked to the PR so that updating one updates the other is just amazing. I know it's not impossible - but it is very nice, and that's so nice to see.

The PR, as you can see, is really a very specific YAML file section, and there were, of course, no comments in it. I had no idea that the elements have changed, so when the reviewer chastised me for not looking at the rest of the file, and detecting the changes, I got a little prickly. But I said to myself "Take it easy... this is a simple mistake", and I apologized on the PR, and made the changes.

At the same time, the guy who helped me on the Forum chimed in and threw me under the bus - saying "Yeah, he didn't do the right thing - I gave him something that worked on the release version, but not master". I love it when people do this. It's such a comforting thought to be in an industry of people with such high integrity.

So that was fixed, and I thought "OK... enough of this, moving on..."

Mistake #3.

I then got a note saying that I hadn't read the rules of PR submissions, and that I needed to write rendering tests for this new code block. And believe me, these are not easy, and they are not trivial, and all this for something I should have just said "Sorry, I'm too busy, you can do the PR on your own."

But I read up on the tests. How to write them - and they are stupid. But I did it. And then I wrote the rendering tests and added those to the PR. And they all passed. Which was nice. So now I'm thinking "OK... this is finally over." But when will I learn?

Then a maintainer comes along and says this will have to be put on hold because they don't understand something about what's happening in the code, and that is causing an issue that is unrelated to the PR, but the PR would make things worse, or something.

So... I try to be a nice guy after someone has been nice to me... you know... passing it on... and for that, I get to deal with the Slashdot Kids living in their parent's basement and holding some power trip over poor saps like me that try to help folks out.

But I have a fix. I know what to do for subsequent releases, and I'll never do this again with these folks. Lesson learned.

This morning, I wanted to get the first version of Cap'n Hook published on Clojars, but I needed to get a few things cleaned up. First, I needed to get the latest changes to durable-queue up to Clojars, as there were some dependencies I need to clear up - most notably in the nippy and byte-streams packages, so that was fixed up and a simple publish, and I'm up on Clojars.

At this point, I'm a little surprised that I have to type in my GPG passphrase each time I sign the jar for publishing. I can see the logic that I've read online - basically, signing is a fairly infrequent task, and needs to be done with care, so ask for the passphrase on those occasions. OK... I get it, but I'd still like to have it in my Keychain, and then just lock it up there. No need to keep typing it in.

I need to pay closer attention to the passphrase dialog on the next publish, and see if I can't get it to save the passphrase in the Keychain. But it's not horrible at this time - I don't publish all that often.

After I got durable-queue up, I could then clean up the capn-hook code a little and then publish it. Again, not bad - cleared out a few dependencies to streamline things just a touch, and then update the README.md for the folks coming to the GitHub page, and away it goes.

Not a bad morning - got a little more familiar with GPG, and published a few things. Good enough.