What's it all about, Alfie?

I was pulling data from a RESTful API, and the returned data was paginated, as data like it often is, and I wanted to process it all in bash because I wanted to be able to share this with a large community. But how to really do that efficiently? I'm not worried about the time it takes to process data in bash, but just to make sure that we're not fetching the same data multiple times. What I came up with was pretty interesting.

First, let's set up the body of the RESTful call. Standard JSON, with the data in the payload as well as a nextLink key:

  {
    "events": [ { "dish": "Stevie is at Poppers!", "userId": "ralph" },
                { "dish": "Ralph is at Waffle House", "userId": "dorris" }
              ],
    "nextLink": "https://yoyo.com/api/rumors?page=4"
  }

where the events key is really holding the important data from the call, but because there was too much data to get in one (efficient) call, the service gave a complete link to hit to get the next page of data. Simple.

Now some APIs return a page number, or a placeholder value to pass to the same URL that generated this data, and this saves 100 bytes, but that's just a different way of getting the "next" page to the caller.

So... how to put all this in bash? Let's start with jq, and you can get it from Homebrew, with:

  $ brew install jq

and then we can look at the basic loop over all pages:

  # this is my YoYo API Token - you can get one too
  tok="111111111111111111111"
 
  # dump all the rumors into a file
  hit="https://yoyo.com/api/rumors"
  while [ "$hit" != "null" ]; do
    # make a temp file for the details of this page of the rumor group
    rumors=$(mktemp /tmp/yoyo.rumors.XXXXXXXX)
    # get the page, and possible next page URL...
    curl -s -X GET $hit \
       -H "accept: application/json" \
       -H "X-API-Token: $tok" > $rumors
    # write out as CSV, the data from this file
    for usr in `jq '.events[] | select(has("userId")) | .userId' $rumors`; do
      echo -e "$title,$usr"
    done
    # get the next URL to load from the service
    hit=`jq '.nextLink' $rumors | sed -e 's/^"//' -e 's/"$//'`
    # clean up the file as we don't need it any more
    rm $rumors
  done

In this example, the $title variable is something that's not defined, but would presumably be defined in the script before we get to the loop. Whatever the point of the gathering the data really is.

What I enjoyed about this is that we can get the data from the service with a simple curl command, and then process it with jq and the flexibility with jq is really quite impressive. In just a few lines, I was able to make a script that fetched a lot of data from a service, extract what I needed into a CSV for processing in a spreadsheet, and it didn't take all that long. Winner. 🙂

This morning I thought I'd perform the ritual of getting the old web development tools that I've used in the past going again - this time on macOS 10.15 Catalina. Now I haven't used PHP in ages, but I've still got code and databases for Postgres to use that - so it makes sense to get this all working again, and it's always fun to see how things work out.

Getting Postgres 11.1

Loads of coverage here about Postgres, and it's just so simple to get the latest version from Homebrew:

  $ brew install postgresql

I've even posted how to upgrade from major version differences, so it's easy to get the latest Postgres running on your box, and the tools are just superb.

Activating UserDir in Apache 2.4.41

As in the previous updates, the UserDir extension is not enabled by default, so we need to get that going right away. This enables the code to be run from the development directories, and that's a big time-saver. First, we need to enable the UserDir module in Apache, and then make a specific config file for the user in question. Start by editing /etc/apache2/httpd.conf and line 183 needs to be uncommented to read:

  LoadModule userdir_module libexec/apache2/mod_userdir.so

and then similarly on line 520 uncomment the line to read:

  Include /private/etc/apache2/extra/httpd-userdir.conf

Next, make sure that the file we just included is set up right for including the user directories. Edit /etc/apache2/extra/httpd-userdir.conf and line 16 needs to be
uncommented to read:

  Include /private/etc/apache2/users/*.conf

At this point, you need to make sure you have at least one file in the /etc/apache2/users/ directory for each user, like: drbob.conf:

  <Directory "/Users/drbob/Sites/">
      Options FollowSymLinks Indexes MultiViews ExecCGI
      Require all granted
  </Directory>

where the last line - Require all granted is new as of Apache 2.4, and without it you will get errors like:

  [Thu Dec 18 10:41:32.385093 2014] [authz_core:error] [pid 55994]
    [client fe80::7a31:c1ff:fed2:ca2c:58108] AH01630: client denied by server
    configuration: /Users/drbob/Sites/info.php

Activating PHP in Apache

The mext thing to do is to activate PHP in the supplied Apache 2 with macOS 10.15. This is line 186 in the file - /etc/apache2/httpd.conf and you need to uncomment it to read:

  LoadModule php7_module libexec/apache2/libphp7.so

and then verify a file called /etc/apache2/other/php7.conf exists and contains:

  <IfModule php7_module>
    AddType application/x-httpd-php .php
    AddType application/x-httpd-php-source .phps
 
    <IfModule dir_module>
        DirectoryIndex index.html index.php
    </IfModule>
  </IfModule>

which does all the other PHP configuration in a separate file to make upgrades easy.

Finishing Up

At this point, a simple restart of apache:

  $ sudo apachectl restart

and everything should be in order. Hit a URL that's a static file with the contents:

  <?php
    phpinfo();
  ?>

and you should see all the details about the PHP install - including the PostgreSQL section with the version of Postgres indicated:

What's really great is that Apple has included lots of support in the default PHP install:

• PHP 7.3.8
• Postgres 9.3.7
• MySQL 5.0.12
• SQLite3 3.28.0

so there's no reason to do anything more to get the kind of support that I used to get. And I get the other databases for free. This is great news! I then run my little test page to make sure the database connection is working:

and everything is working exactly as expected!

Yesterday I had an interesting phone screen with someone, and the seemingly common practice of asking a candidate to code on the phone - in a shared web-based environment again came up. I recognize that any employer can have any legal criteria for employment, and the "Coding Phonescreen" is a very common one. You get to see if the person can write code in a few minutes as opposed to inviting them for a day-long interview cycle that can cost considerably more. It's decent logic.

But it really doesn't tell the story, does it?

Speed Coding has about the same relationship to real development as a college classroom has to Jeopardy!... yeah, the material is the same, but the skills to be able to do well in one do not necessarily translate to the other. And the most critical skill in the speed forms is pattern recognition of the problem.

If you've seen this problem before, and written a simple solution to it, then you're in good shape. You know the flow, you know the pitfalls, and you can talk your way through it - like you're talking your way through directions to a local restaurant. No pressure, you're showing someone something you know, and it happens to take a few steps. No biggie.

But if you're at all unsure, then you're not going to get time to think about the solution before being expected to answer it. This is the problem with Speed Coding - if you know the answer, it's fine. But then it's not really seeing if you can think on your feet... if you don't know the answer, you're likely going to make one or two edge-case mistakes, and those will be clearly visible to the person that knows the solution.

The problem I ran into was a binary tree issue, and while I had been practicing my Speed Coding in Clojure, the nature of the binary tree really leaned towards a C++ solution, and that was not horrible, but it was a lot less friendly to edge-conditions.

I ended up writing something like this:

  struct Node { int value; Node *left; Node *right };
 
  bool stored(Node *me, op) {
    bool   retval = true;
    if (retval && (me->left != NULL) && (me->left op me->value)) {
      retval = stored(me->left, op);
    }
    if (retval && (me->right != NULL) && (me->value op me->right->value)) {
      retval = stored(me->right, op);
    }
    return retval;
  }

and the missed edge-case is that once you are one, or more, steps down in the tree, it's possible to have the relative position of the values be correct, but the absolute position to be false. There are two ways to solve this, in this code:

1. Pass limits down with the call - this could be done with max and min arguments and then in the recursive calls, place the correct values there, and test them as well.
2. Scan up the tree on each check - this could be a walk-up the tree and check to see that you aren't in violation of the location you have. This would take more time because of all the walking, but it'd work.

But what I'd wanted to do was to write it in Clojure, but the data structure didn't jump out at me. Until this morning. 🙂 This morning I spent the minute or two thinking about the data structure, and then formulated the following solution:

  ;; [val left right]
  ;;       10
  ;;    5      15
  ;;  1   7  12   20
  (def good [10 [5 [1] [7]] [15 [12] [20]]])
 
  ;;       10
  ;;    5      15
  ;;  1   17  12   20    -- the 17 is out of place
  (def bad [10 [5 [1] [17]] [15 [12] [20]]])
 
  (defn sorted?
    "Function to check a binary tree to see if it's sorted for proper
     searching."
    [[v lt rt] op]
    (let [ltv (first lt)
          rtv (first rt)]
      (and (or (nil? lt) (and (every? identity (map #(op % v) (flatten lt)))
                              (check lt op)))
           (or (nil? rt) (and (every? identity (map #(op v %) (flatten rt)))
                              (check rt op))))))

What I really like about this solution is that it checks the entire subtree with the operation. This means that the effort to do one, is really checking all of them. This is what I wanted to write, and it works perfectly.

But I didn't force the issue, and pull back and take the time to think. My mistake. I won't make it again.

UPDATE: a friend and I were talking about this same problem, and he came up with a solution that was very clever - the structure can be validated by simply assuming that the structure is a sorted binary tree, and then calculating the min and max values of the tree.

The catch being that if you get to a node where the current value isn't within the min and max, then you have to fail, and return nil. It's really quite amazingly simple in that it's very fast, very easy to understand and adds the additional benefit of returning the extremum of the tree.

  (defn bst-rx*
    "returns extent [min max] of valid bst or nil if invalid"
    [t]
    (cond
      (nil? t) nil
      (vector? t) (let [[v l r] t
                        lx (if l (bst-rx* l) [v v])
                        rx (if r (bst-rx* r) [v v])]
                    (when (and lx rx (<= (lx 1) v (rx 0)))
                      [(lx 0) (rx 1)]))
      :else [t t]))
 
  (defn bst-rx?
    [t]
    (boolean (bst-rx* t)))

Yesterday morning, I thought to check my blog and see something, but to my complete surprise, it was down. Like very down, and I had no idea what was happening. I've been using WordPress and HostMonster for more than a decade, and I've never had anything even remotely like this. I didn't know where to start - so I went to their admin site, thinking I'd been hacked somehow...

First, it was clear that they had done a ton of upgrades to the host and its support platform. Massive changes. So the first thing was to get logged in. This was a little odd because it wasn't working on Safari, and it had been in the past - so I switched to Chrome, and finally got logged in. Step 1 - accomplished!

Then I looked at the installed users on each of the three WordPress sites I have, and in each case, there was a user that I couldn't explain. It certainly appeared to me that these were bad actors, and I had no idea how they got there. I stay up to date, don't allow logins, don't allow replies... it's a Journal more than anything else. But still... I could not deny these accounts. So I asked for help.

It took a long while to figure this out, but in the end, the logs for the site indicated that there was a PHP problem in one of my plugins, and one of my themes. Why this happened yesterday wasn't at all clear, but it became clear as I dug further.

HostMonster had dropped support for PHP 5.x, and the only versions available to me were 7.0, 7.1, 7.2, and 7.3, with the latter being the default. Now it seemed to be clear what had happened... nothing had changed, but in all the upgrades for the infrastructure on the hosts, they had switched to a different config for PHP, and the plugin and theme were doing something that they shouldn't. OK... now to the code.

The first one I tackled was the plugin, as the error in the logs was pretty clear. I did a quick search for =& and sure enough, it was a PHP 5-ism, and that was easy to fix. That solved the plugin problem, and it loaded and ran fine. The theme had a different problem with a deprecated function that wasn't really even needed in the theme, but I found a replacement, and used that, and the theme was fine.

All told, this took more than 5 hours... it was not quick, and I just ahead of the part where I found out that the timezone plugin I was using wasn't needed in WordPress 5, and so I didn't put that back into play, Also, when I got the site up, it was possible to see the errors on activation of the plugin (but not the theme), which made continued debugging a lot easier.

In the end, it was all cleaned up, and now it's set for PHP 7. I'm glad that there wasn't a bigger issue, but I really have to be careful of these things because there is almost no support for the plugin and theme - and I really like to have both of them working for me for this blog. 🙂

Yesterday I saw this very enjoyable tweet and it made me smile - quite a lot... because I do interviews often, and they are almost always about Architecture and Design, and scaling is one of the key use-cases to explore. It really helps see how the candidate approaches the problem:

Used to pay $5/mo on a small instance for my personal site. Then I discovered Kubernetes and realized my site didn't scale! No canary deployments! So I upgraded and pay $200/mo now. Took weeks to configure. Millions of people can now read my resume. Damn, it's never looked better
-- @malaroto

When faced with a scaling problem, AWS has been amazingly good - and amazingly bad for junior developers on the path of learning their craft. On the up-side, it's wonderfully full-featured, stable, available, on-demand in every way, and global. On the downside, it doesn't require any effort to use - so the first answer most jump to, is to build something that doesn't need to be built with unnecessary complexity that will work, but slows down the ability for someone else to understand the solution, and it's basically just wasting money.

"Not my dime" - is accurate, but not really the point. The point is to understand the problem, and then fix the problem, but if everything is able to be solved with millions of dollars of computing infrastructure, there seems to be no motivation to solve it with an hour of understanding and code refactoring.

So I giggle... they will have their day - a massive supercomputer in a grain of sand - ubiquitous and omnipresent... and then there will be no need to understand the why of issues... and that will be too bad. There will always be a need for craftsmanship.

Interesting, that a few days after I move off Oracle's JDKs for my laptop, RedHat announces that it's going to assume stewardship of OpenJDK 8 and 11 from Oracle. It seems that Oracle wants to get out of the enterprise java business as it moved J2EE support to the Eclipse Foundation, and this just makes it clear that they really don't want to be in the Java business - at all.

This could be that they can't make enough money on it - or that they are just more interested in GraalVM, which they control, and can be used to compile a large subset of Java code into machine code. With this control, what do they need with the OpenJDK spec? It's just weight and cost. Better to get someone else to take care of that, and get back to making money licensing what they can.

It's just sad, but all too familiar, with Oracle. Glad I moved off. I'll stick with AdoptOpenJDK until there's a reason to move.

This morning I decided to really dig into the Homebrew support for AdoptOpenJDK - and while I was initially a little unsure of the AdoptOpenJDK group, after reading their web site, and digging into their Homebrew cask, I was convinced to give it a try. It will keep things moving with updates on the JDKs - delivering JDK 1.8.0r212 as of today, and that's what my biggest fear was. So let's get to it.

In order to get the very latest AdoptOpenJDK (11), it was as simple as:

  $ brew cask install adoptopenjdk

which resulted in my installation directory looking like:

  $ ls /Library/Java/JavaVirtualMachines/
  1.6.0_26-b03-383.jdk/    jdk1.7.0_45.jdk/         jdk1.8.0_144.jdk/
  adoptopenjdk-11.0.2.jdk/ jdk1.7.0_51.jdk/         jdk1.8.0_181.jdk/
  jdk-10.0.2.jdk/          jdk1.7.0_75.jdk/         jdk1.8.0_40.jdk/
  jdk1.7.0_13.jdk/         jdk1.8.0_131.jdk/        openjdk-11.0.2.jdk/

At this point, I could remove the openjdk-11.0.2.jdk/ install, as that's from Oracle, proper, and I really just want to let that go due to the licensing...

  $ brew cask uninstall java

and then re-select the correct JDK 11 with:

  $ setjdk 11; echo $JAVA_HOME
  /Library/Java/JavaVirtualMachines/adoptopenjdk-11.0.2.jdk/Contents/Home

So it was installed properly, and picked up the new installation from AdoptOpenJDK. Good. Now in order to pick up the older versions, we can follow the instructions on the Homebrew cask:

  $ brew tap AdoptOpenJDK/openjdk
  $ brew cask install adoptopenjdk8

and after this, we have the following installed on my machine:

  $ ls
  1.6.0_26-b03-383.jdk/    jdk1.7.0_13.jdk/         jdk1.8.0_131.jdk/
  adoptopenjdk-11.0.2.jdk/ jdk1.7.0_45.jdk/         jdk1.8.0_144.jdk/
  adoptopenjdk-8.jdk/      jdk1.7.0_51.jdk/         jdk1.8.0_181.jdk/
  jdk-10.0.2.jdk/          jdk1.7.0_75.jdk/         jdk1.8.0_40.jdk/

and I have to admit, I'm a little concerned about the lack of a release number on the AdoptOpenJDK 8's directory - but maybe that will be handled a little differently as they version them. We will have to wait and see.

At this point, we can:

  $ setjdk 1.8; echo $JAVA_HOME
  /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home
  $ java -version
  openjdk version "1.8.0_212"
  OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_212-b03)
  OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.212-b03, mixed mode)

To make sure that we don't cleanup the old versions with Homebrew, when I install a new one, I added the following to my ~/.bashrc:

  export HOMEBREW_NO_INSTALL_CLEANUP=1

Which Homebrew says will keep from doing the automatic cleanups on the installs.

To do a final check with the AdoptOpenJDK 1.8.0r212, I checked what Leiningen was saying:

  $ lein --version
  Leiningen 2.8.3 on Java 1.8.0_212 OpenJDK 64-Bit Server VM

At this point, I've got AdoptOpenJDK 1.8.0r212, and 11.0.2, and updating each is very simple with brew. I just need to:

  $ brew update
  $ brew cask upgrade adoptopenjdk
  $ brew cask upgrade adoptopenjdk8

and then when I want, I can clean up the older versions with:

  $ brew update
  $ brew cleanup adoptopenjdk
  $ brew cleanup adoptopenjdk8

This is so much cleaner, and I don't have to worry about the crazy licensing from Oracle. It's a lot nicer place to be.

This morning I got an update notice on Java 8 on my machine - it was for JDK 8r211, and during the normal update process, the installer popped up a dialog box where it was very clear that the terms of the license for JDK 8r211 have changed.

Specifically, it stated that this JDK was only for non-commercial use, and if it was going to be used in a commercial environment, then a license would have to be obtained from Oracle.

This is the JDK 11 license, and why I went through the pain to get OpenJDK 11 on my laptop, because I just can't see even pretending to use a product that I know I won't ever use in production. Now, I have to stick with the version of JDK 8 I've got installed, because I don't see an easy path to get OpenJDK 8 from Homebrew. There's an old cask, but I don't know that it's any more current than what I have - and I don't know that I need anything more current.

Just seems so short-sighted by Oracle, but they see a revenue stream, and go after it.

This afternoon, after reading a Master's Thesis on the prediction of the Corn Spot Prices, I was talking to a friend about using Mathematica, and how much I'd really enjoy getting back into the mathematically-based work I did in my Thesis, and how if this new work was put in a service, Clojure would be the obvious choice, and plotting would be easily done with DataGraph, which is just a beautiful way to create publication-quality plots.

Here's an evolution of an electron pulse evolution in GaAs:

I've been using this for years, and now I read in the 4.4 beta release notes, they are going to be adding tools to make it easier to get data into the app, and so it might make it all a lot easier for me and intergrating it with Clojure. That would be very nice.

This morning I was thinking about the snippet manager I saw on Twitter... and then I realized that Gists on GitHub are exactly that - and why not just check out the status of the Sublime Text 3 plugin that deals with Gists... and the Vim plugin to do something similar.

I had to update the access token for the Sublime Text 3 plugin, and then it was working fine. But one thing I noticed was that the Gists showed up in the editor as modified files. While that's understandable if you realize the model for this is to read the Gist, and then paste it into the editor window, it's not really the workflow I expected to see.

Additionally, the syntax highlighting wasn't being triggered, so the Gists all had the syntax type of 'Text' - which was really annoying... given that the extension was exactly what you'd need to trigger the syntax type of the file. So that wasn't as nice as I expected. But it still worked.

Then I went to Vim and updated the Gist plugin, and that was interesting, but not as horrible as it could be. Still... it's a bunch of command-line steps, but it's OK. Again, there was another auth token from GitHub, which is nice, but that's about it.

The difference is that when you load a Gist into a Vim buffer, it is not seen as modified, and the syntax highlighting is correct. So those are nice to see in Vim, and while it's not amazing, it is nice to see it done right.

In the end, I'm glad to have updated these things - even if I really wish there was a nice macOS client like Dash that worked with Gists.