What's it all about, Alfie?

This morning I read a tweet that was re-tweeted by a guy I used to work with. The tweet said:

If anyone needed to hear it: I don't care about .dev domains or mechanical keyboards, I don't read Hacker News, I don't have a customized terminal, I don't spend all weekend coding, I sometimes use the Git GUI in my IDE, and I am valid in tech and so are you.

The person that wrote this isn't important - they are right - they are every bit as valid in this industry as anyone else. There isn't a "test" of "worthiness" to get, and keep, a job in this industry. And those that treat someone as less of a person, or less of a value to the organization are just missing the point. Badly.

Not everyone has to want to do these things. That's perfectly fine. Take another industry - doctors, lawyers, professional sports - there are all kinds of doctors that don't read up on the medical journals in their weekends... and plenty of lawyers that don't read briefs to stay up on things... and all kinds of different levels of pro sports.

But we can't then also insist that the very best surgeon work on our loved ones. Or that if we're in trouble, and need a lawyer, that you get the very best one you can get to get out of trouble. Right? And we certainly can't be disappointed in our favorite team for not winning - Right? They are just as valid as anyone else in that industry. Right?

Or is 'valid' the only metric we want to use here? I agree that the old Doc Hollywood story is about what you value - and sure, you're not doing amazing surgery saving lives, but that's not what you value - relationships are. But no one would call a Doc Hollywood to do heart surgery - and when you need a heart surgeon, you don't want to think that Doc Hollywood is your best alternative.

And when you feel you've been wrongly accused - you probably don't want to have to rely on the Public Defender. They are just as valid in the legal profession... and no one should make them feel like "less" of a lawyer because they are in the Public Defender's office... right?

As I read this tweet two things struck me:

• This person works with some serious jerks - it's clear that this person has been working with, or working around, some major jerks that feel there is some litmus test to be a "good developer" - or even a "real developer". That's just wrong, and you can't change other people's attitudes, so my advice would be to find another job. There are better ones out there.
• 'Valid' isn't the same thing as 'Value' - no one should be so marginalized that they feel that their fundamental worth is in question - but that's not to say that everyone is equal, either. Skills matter. Experience matters. Value isn't the same thing as validity. I can play basketball... but I have no business playing on the Lakers. And I shouldn't expect to.

I would like to tell this person: "Listen, you're right - you work with jerks - ignore them. Get a job that you like, and do your best." but at the same time, I can't say that this person should also expect to see the same paycheck as those that work 7 days a week. Or those that have devoted decades to their craft. Just recognize that choices matter.

Technology really has become the new workplace where classic blue-collar and white-collar jobs are mixed into the same pot. There's nothing wrong with picking what you want to do, and what's important to you - and you should not be made to feel your choices somehow make you less of a person. But they do have consequences.

This morning I was chatting with my Boss, and we were talking about the progress that was being made - and I had to admit, that we are starting to at least work together - as opposed to hold meetings where we disagree, but make no decisions, and just table the discussion for the next meeting. This works to simply make no progress, and it can be quite effective. But we're not there, and that's a really solid move in the right direction.

But that is not to say that yesterday wasn't a challenge. I understand how oppressive a job can be where you are not valued, where you are not listened to - I've worked in them. But the flip-side is no better - where every person in the company feels entitled to reject an assignment because they are self-empowered, and the management team does nothing to dissuade them of that opinion.

So I find myself in the position of, once again, patiently explaining requirements for a project to members of a team that simply will not accept it, and so won't be on the implementation team, and we'll simply have to get new folks. These guys can support the legacy stuff - it's all their stuff, and it'll be around for at least a year or so, so they have jobs, and comfortable ones, but they won't be on the new stuff - because they don't believe in it.

And that got me to make the statement:

I feel like an Old Knight - in a Rusted Suit of Armor. Way past his time.

and it struck me that this was a turning point for me. To see that Honor, Integrity, and Commitment are so vital to me, and so completely foreign to them. For example, I may not like what I'm doing, but it's the Job. It's the social contract on employment - I agree to do the job they ask me, and they agree to pay me the promised wage.

That's it.

No arguments.

My choice, and my power is to change jobs. That's perfectly acceptable. I can change jobs as often as I want, and for whatever reasons I wish. That's my choice and power. But if I'm here, and taking the wage, then by golly, if they ask you to do the job then you do it. Period.

This isn't about being hired as a Data Scientist, and being asked to pick up an AK-47 and defend the CEO's mansion from zombies... this is about accepting a job to develop software, and being asked do just that. It's simple.

But clearly, in today's environment, it isn't. Which is why I feel so completely out of time with many of my co-workers. They can't understand me because they don't have the same sense of what's important to me, and while I can understand why they feel entitled (too many Participation Trophies) it doesn't excuse the behavior.

Just like an Old Knight... in a rusted suit of armor.

Today I've been having an email exchange with an old co-worker about an Open Source project that he's been working on - actually, it's pretty much done. It's all in Python, so it's not something that I'm really keen to use, but the design is good, and I can remember using something like it back at a previous job where we had to solve complex data modeling issues in as near real-time as possible.

The project is called dataflow and is really a compute/modeling system where you define Nodes and these nodes can be simple calculations on inputs, but they can be quite advanced - say rolling time averages, etc. and the key is that these Nodes can relate to one another in a simple Graph, that the user can describe, and then when any Node is updated, it fires off the updates of the dependent Nodes.

The nice thing is that these Nodes have a very simple API, and so they can do just about anything - averages of inputs, calculations, mean-square calls... it's really what the user wants to see. Tie this with a UI that displays the data, and you get something that is very powerful indeed. It can be a a Risk system where each Node can aggregate the child Nodes so that you can look across account, symbol, etc. and all updates as the input data updates.

It's been a nice reminder of some of the work I'd done at that job - but the work I did was in Java, and it was a bit more limited, but it still enabled some pretty nice apps to be built. Good stuff.

Well... today has been a very eventful day. I had noticed that Oracle dropped support for JDK 10, and mentioned it to some friends in Slack. I didn't know the details, so I decided to dig in and see if I could see anything on the net about it. I did. Lots. Changed my day.

Turns out, Oracle is changing it's licensing for JDK 11 - it's free for development and testing, but if you use it in production, it requires a license. A paid license. Yikes!

OpenJDK is also built by Oracle, and IBM, and RedHat - and it is Open Source - free for any use-case. So that's where people appear to be going. I can see that, but I've stayed away from OpenJDK because it's just the official JDK from Oracle. But now it seems Oracle is looking to cash in on the JDK, and this is just a lovely predicament.

But there is hope - kinda. Homebrew is supporting OpenJDK as a cask so that it can be versioned, and supported within the Homebrew ecosystem. That's good news. To install the latest, you simply need to:

  $ brew cask install java

and it will install it into:

  $ brew cask info java
  java: 11.0.2,9
  https://jdk.java.net/
  /usr/local/Caskroom/java/11.0.2,9 (64B)
  From: https://github.com/Homebrew/homebrew-cask/blob/master/Casks/java.rb
  ==> Name
  OpenJDK Java Development Kit
  ==> Artifacts
  jdk-11.0.2.jdk -> /Library/Java/JavaVirtualMachines/openjdk-11.0.2.jdk
  (Generic Artifact)

and the tools I built for switching JDK versions work just fine:

  $ setjdk 1.8; echo $JAVA_HOME
  /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home
  $ setjdk 11; echo $JAVA_HOME
  /Library/Java/JavaVirtualMachines/openjdk-11.0.2.jdk/Contents/Home

That's the good news. What isn't really good is that there is an issue with the Java Collections where the toArray method added an additional arity and that breaks Clojure - badly. So it really doesn't help to have the OpenJDK 11 on the box - as Clojure can't make use of it.

However, that's not all... it appears that when I upgraded Leiningen, I broke one of the really nice plugins that I like to use with it - Ultra. The error is nasty as well:

  $ lein clean
  $ lein deps
  $ lein repl
  [WARNING] No nREPL middleware descriptor in metadata of
    #'clojure.tools.nrepl.middleware.render-values/render-values,
    see nrepl.middleware/set-descriptor!
  [WARNING] No nREPL middleware descriptor in metadata of
    #'clojure.tools.nrepl.middleware.render-values/render-values,
    see nrepl.middleware/set-descriptor!
  nREPL server started on port 54967 on host 127.0.0.1 - nrepl://127.0.0.1:54967
  ERROR: Unhandled REPL handler exception processing message
    {:id ad3092dd-8491-4d60-a1a9-5092d4d090a9, :op clone}
    java.lang.IllegalArgumentException: No implementation of method:
    :send of protocol: #'nrepl.transport/Transport

and if I removed Ultra from my ~/.lein/profiles.clj then everything worked fine again. But this was with Leiningen 2.9.0 - and that was an issue. In fact, it was an issue that's already been reported to Ultra. I added my notes, as the suggestion didn't work for me.

So now I'm left with downgrading Leiningen to get Clojure working again. This is not as easy as I'd hoped, but it's what needs to be done. First, get into where the Homebrew formulae are stored, :

  $ cd $(brew --prefix)/Homebrew/Library/Taps/homebrew/homebrew-core/Formula
  $ git log --follow leiningen.rb
  commit 2be19f3787d811247095e704765fc33a8815d639
  Author: BrewTestBot <homebrew-test-bot@lists.sfconservancy.org>
  Date:   Tue Feb 12 13:07:29 2019 +0000
 
      leiningen: update 2.9.0 bottle.
 
  commit 9ca7899818eb02cc8252d47cc32b5a42554a7655
  Author: Rahul De <rahul080327@gmail.com>
  Date:   Tue Feb 12 10:43:59 2019 +0100
 
      leiningen 2.9.0
 
      Closes #36927.
 
      Signed-off-by: Chongyu Zhu <i@lembacon.com>
 
  commit fd1c0ba8f162a3c41debef6143dca8ba19227f90
  Author: BrewTestBot <homebrew-test-bot@lists.sfconservancy.org>
  Date:   Sat Dec 15 10:45:41 2018 +0000
 
      leiningen: update 2.8.3 bottle.
 
  commit 40553be0b01710db9e16cff978f593b83ecdfe3b
  Author: Rahul De <rahul080327@gmail.com>
  Date:   Sat Dec 15 11:27:08 2018 +0100
 
      Leiningen 2.8.3
 
      Closes #35151.

at this point we heed the git SHA for the version we want to move to. In the case of Leiningen, the SHA for 2.8.3 is fd1c0ba8f162a3c41debef6143dca8ba19227f90 and the SHA for 2.7.1 is cdddd1094b26d0092e0030051dac97a286bb3fc4. I started with 2.8.3, and that didn't work, so I had to go back to 2.7.1. This is a big difference.

Then make a branch for that version, and reinstall Leininden from the Ruby file:

  $ git checkout -b leiningen-2.8.3 fd1c0ba8
  $ brew reinstall ./leiningen.rb
  $ brew switch leiningen 2.8.3
  $ git checkout master

for 2.8.3 and, for 2.7.1:

  $ git checkout -b leiningen-2.7.1 cdddd109
  $ brew reinstall ./leiningen.rb
  $ brew switch leiningen 2.7.1
  $ git checkout master

At this point, we can see that both versions are installed:

  $ brew info leiningen
  leiningen: stable 2.7.1 (bottled), HEAD
  Build tool for Clojure
  https://github.com/technomancy/leiningen
  /usr/local/Cellar/leiningen/2.7.1 (9 files, 14.7MB) *
    Poured from bottle on 2019-02-18 at 13:29:34
  /usr/local/Cellar/leiningen/2.8.3 (9 files, 13MB)
    Poured from bottle on 2019-02-18 at 13:22:14

Now it's possible to run Leiningen and Ultra - again. It has to be JDK 8, and Leiningen 2.7.1... but until Ultra is fixed for the change in Leiningen, this is as far as I can go.

What was even more annoying was that my new work laptop has a very recent version of Homebrew on it, and the git repo has been trimmed, and so there is no git SHAs for the previous versions. However, with tar and copying the tarball over, I was able to get the 2.7.1 and 2.8.3 versions there, and things are now working OK.

Wow! What a mess to get back to where I thought I started the day!

This morning I was working to upgrade my laptop from Postgres 10.3 to 11.1, and I did the following, as part of the upgrade:

  $ brew update
  fatal: could not read Username for 'https://github.com': terminal
  prompts disabled
  Error: Fetching /usr/local/Homebrew/Library/Taps/homebrew
  /homebrew-boneyard failed!

I did a lot of googling, and came up with a couple of ideas. This was from GitHub:

  $ git config --global credential.helper osxkeychain

but that didn't work. Then I found another reference saying that the 2FA on GitHub was the problem, and that you needed to bypass that with:

  $ git config --global --add url."git@github.com:".insteadOf "https://github.com/"

and still that didn't work. So I took a different approach, I started looking at the Homebrew files on my laptop - specifically, the boneyard mentioned in the error. Turns out... if you looked at the .git/config file in that directory, it points to a directory in GitHub that no longer exists! OK... this might be something.

So I removed the nearly empty directory, and tried again:

  $ brew update
  Already up-to-date.

Success! It wasn't the config of git - it was that Homebrew was trying to update something that no longer existed. Lesson learned.

It's time to make sure that my laptop has the latest version of PostgreSQL, as I was reading that they were making big strides in the windowing functions in postgreSQL 11. And with Homebrew 2.0 out, it made sense to just write it all down again, just so that I have it for reference.

The process isn't bad... dump all the databases into one file, stop the server, update Homebrew, update postgres. This gets us to the point that we are ready to rebuild the new database:

  $ pg_dumpall > dump.sql
  $ brew services stop postgresql
  $ brew update
  $ brew upgrade postgres

Now we need to move out the old database data, create a new structure, and restart the service:

  $ cd /usr/local/var
  $ mv postgres postgres.old
  $ initdb -D /usr/local/var/postgres
  $ brew services start postgresql

You then need to go back to the directory of the first command - the one where you dumped the databases, and reload them all:

  $ psql -d postgres -f dump.sql

and at this point, everything should be back and running:

  $ psql --version
  psql (PostgreSQL) 11.1
  $ psql -l
                                      List of databases
      Name     | Owner | Encoding  |    Collate     |     Ctype      |  Access privileges
  -------------+-------+-----------+----------------+----------------+---------------------
   health      | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII |
   inventory   | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII |
   northhollow | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII |
   postgres    | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII |
   template0   | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII | =c/drbob           +
               |       |           |                |                | drbob=CTc/drbob
   template1   | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII | drbob=CTc/drbob    +
               |       |           |                |                | =c/drbob           +
               |       |           |                |                | _postgres=CTc/drbob+
               |       |           |                |                | postgres=CTc/drbob
   test        | drbob | SQL_ASCII | en_US.US-ASCII | en_US.US-ASCII |
  (7 rows)

At this point you can remove the old data:

  $ rm -rf /usr/local/var/postgres.old
  $ rm dump.sql

and everything is updated. You can then use any of the normal tools, including the Apache/PHP/Postgres that Apple makes available, and Postico - a great GUI client.

I've been rebuilding a new MacBook Pro laptop from work because the security tools we use at The Shop don't allow Migration Assistant to work - which is sad, because it works so well under normal circumstances, but water under the bridge, and I have to reinstall a lot of things on the box - as if from scratch.

Because of Clojure, one of the things I have to make sure to install is the latest JDK. Actually, I need to install all the latest JDKs - 1.8, 9, 10, and 11. And then I have a simple little tool for changing the JDK version that I want to be using. Simple. Easy.

When I last went to upgrade the versions of the JDKs, I found that Oracle has dropped support for JDK 9. I had heard a lot of talk about how JDK 9 wasn't really doing well because of some of the design decisions... and so this wasn't really a surprise, but it was unexpected to see that Oracle had pulled it from the Downloads. But OK... they didn't want people using it, so they pulled it.

Today, I saw that Oracle had pulled JDK 10 as well. Now that was a bit of a surprise. I know that it wasn't as popular with the Clojure community as JDK 1.8, but to see a second abandoned JDK version... well... that's a heck of a realization from Oracle. It's not good... it's causing us too much to support it... drop it.

So I have JDK 1.8 and 11 on the new laptop, even though on my personal machine I still have the last available JDK 10. But at some point, that will have to be thinned out because it just won't be supported. Wild.

Hats off to the JDK 1.8 guys - You've made a heck of a product.

I saw this on Twitter this morning:

Adam McCrea - @adamlogic: I see a lot of confusion around @heroku's free dynos, so I wrote up a guide. Let me know if you find this helpful, and please share!

https://railsautoscale.com/heroku-free-dynos

and I have to say, it's packed with a lot of really useful information about the free, hobby, and paid dynes and database plans. What I really liked was that the alternatives, and work-arounds that he presented makes it clear there are some decent alternatives to the Heroku plans, but that Heroku is still in a good place.

I still want to put some Clojure service up on Heroku someday... I just need to know what the app is, and maybe it's tied into a macOS or iOS app as well. Who knows... but I really like to see someone look into the good, and the not-so-great parts of a service provider - just to keep them honest.

This morning, actually, last night, Apple released macOS 10.14.3, iOS 12.1.3, and tvOS 12.1.2 - and so I had a good morning updating my devices. It always puts me in a good mood to update machines... it's a fresh start... a new beginning, and it makes me smile.

I am glad to have updated to the latest Apple TV because the TV app is really a great thing for keeping track of the shows that I'm watching on Netflix and Hulu - all in one place. And it goes to my iPhone just in case I'd like to watch on the train home.

All updated and ready to take on the day!

A friend asked me today for the trick I've used in the past to make sure that the data in a database is immutable and versioned. This is nice because it matches what Clojure acts like, internally, and it makes it easy to see who's doing what, when. Assuming we start with a table - for the sake of argument, let's say it's customers, and looks something like:

  CREATE TABLE IF NOT EXISTS customer (
    id             uuid NOT NULL,
    version        INTEGER NOT NULL,
    as_of          TIMESTAMP WITH TIME zone NOT NULL,
    last_name      VARCHAR,
    first_name     VARCHAR,
    PRIMARY KEY (id, version, as_of)
  );

where the first three fields are really the key ones for any table to work with this scheme. You need to have a unique key, and in this case, the easiest I've found is a UUID, and then you need a version and a timestamp when that change was made. What's left in the table is really not important here, but it's something.

You then need to make an Audit table that has the same table structure, but has the string _audit added to the name:

  CREATE TABLE IF NOT EXISTS customer_audit LIKE customer including ALL;

What we then need to create is the following trigger that intercepts the INSERT and UPDATE commands on the customers table, and places the historical data into the Audit table, and the most recent version of the customer data is always kept in the customer table.

The trigger looks like this:

  --
  -- create the trigger on INSERT and UPDATE to update the version if it's
  -- not provided, and to maintain all versions in the audit table, but have
  -- the current version in the non-audit table. Importantly, NOTHING is
  -- deleted.
  --
  CREATE OR REPLACE FUNCTION audit_customer()
  RETURNS TRIGGER AS $body$
  DECLARE
    ver INTEGER;
  BEGIN
    -- get the advisory lock on this id
    perform pg_advisory_xact_lock(('x' || translate(LEFT(NEW.id::text, 18),
                                  '-', ''))::bit(64)::BIGINT);
 
    -- get the max of the existing version for the data now
    SELECT MAX(version) INTO ver
      FROM customer_audit
     WHERE id = NEW.id;
    -- and bump it up one and use that
    IF ver IS NULL THEN
      NEW.version := 1;
    ELSE
      NEW.version := ver + 1;
    END IF;
 
    -- if an update, then we need to insert the new
    IF tg_op = 'UPDATE' THEN
      -- now let's insert the old row into the audit table
      INSERT INTO customer_audit
        VALUES (NEW.*);
    elsif tg_op = 'INSERT' THEN
      -- now let's insert the new row into the audit table
      INSERT INTO customer_audit
        VALUES (NEW.*);
      -- and delete the old one in the customer table
      DELETE FROM customer
        WHERE id = NEW.id
          AND version <= ver;
    END IF;
 
    -- finally, return the row to be inserted to customer
    RETURN NEW;
  END
  $body$ LANGUAGE plpgsql;
 
  CREATE TRIGGER set_version BEFORE INSERT OR UPDATE ON customer
    FOR each ROW EXECUTE PROCEDURE audit_customer();

At this point, we can INSERT or UPDATE on customers and the previous version of that customer will be mmoved to the Audit table, and the the most recent version will be held in the customers table.

I have found this very useful, and I've put it in a gist for easy access.

The point of:

    -- get the advisory lock on this id
    perform pg_advisory_xact_lock(('x' || translate(LEFT(NEW.id::text, 18),
                                  '-', ''))::bit(64)::BIGINT);

is to get an exclusive lock on the data for a given id. This is necessary to make sure that updates from multiple services get serialized on the same data. This scheme can't ensure that there are merged changes - only that there is a sequence of changes to the table, and each one is entirely correct for the time it was entered.

So... what happens if you have a string as the primary key, and not a UUID? Well, use can use the MD5 checksum of the string as the lock indicator:

    -- get the advisory lock on a general string
    perform pg_advisory_xact_lock(('x' || md5(NEW.wiggle::VARCHAR))::bit(64)::BIGINT);

where the field wiggle is a varchar, and here, we are just computing the MD5, and using that as the basis of the lock. Yes, there could be some hash collisions, but that's likely not a huge performance problem, and it's conservative in that we'll over-lock, and not under-lock.

UPDATE: a friend asked about using an int as the primary key, and in that case, the line would be:

    -- get the advisory lock on a general string
    perform pg_advisory_xact_lock(NEW.id::BIGINT);

where the column id is an int. Again, we just need to get it to the bigint for the advisory lock call. After that, Postgres does the rest.