What's it all about, Alfie?

I was pulling data from a RESTful API, and the returned data was paginated, as data like it often is, and I wanted to process it all in bash because I wanted to be able to share this with a large community. But how to really do that efficiently? I'm not worried about the time it takes to process data in bash, but just to make sure that we're not fetching the same data multiple times. What I came up with was pretty interesting.

First, let's set up the body of the RESTful call. Standard JSON, with the data in the payload as well as a nextLink key:

  {
    "events": [ { "dish": "Stevie is at Poppers!", "userId": "ralph" },
                { "dish": "Ralph is at Waffle House", "userId": "dorris" }
              ],
    "nextLink": "https://yoyo.com/api/rumors?page=4"
  }

where the events key is really holding the important data from the call, but because there was too much data to get in one (efficient) call, the service gave a complete link to hit to get the next page of data. Simple.

Now some APIs return a page number, or a placeholder value to pass to the same URL that generated this data, and this saves 100 bytes, but that's just a different way of getting the "next" page to the caller.

So... how to put all this in bash? Let's start with jq, and you can get it from Homebrew, with:

  $ brew install jq

and then we can look at the basic loop over all pages:

  # this is my YoYo API Token - you can get one too
  tok="111111111111111111111"
 
  # dump all the rumors into a file
  hit="https://yoyo.com/api/rumors"
  while [ "$hit" != "null" ]; do
    # make a temp file for the details of this page of the rumor group
    rumors=$(mktemp /tmp/yoyo.rumors.XXXXXXXX)
    # get the page, and possible next page URL...
    curl -s -X GET $hit \
       -H "accept: application/json" \
       -H "X-API-Token: $tok" > $rumors
    # write out as CSV, the data from this file
    for usr in `jq '.events[] | select(has("userId")) | .userId' $rumors`; do
      echo -e "$title,$usr"
    done
    # get the next URL to load from the service
    hit=`jq '.nextLink' $rumors | sed -e 's/^"//' -e 's/"$//'`
    # clean up the file as we don't need it any more
    rm $rumors
  done

In this example, the $title variable is something that's not defined, but would presumably be defined in the script before we get to the loop. Whatever the point of the gathering the data really is.

What I enjoyed about this is that we can get the data from the service with a simple curl command, and then process it with jq and the flexibility with jq is really quite impressive. In just a few lines, I was able to make a script that fetched a lot of data from a service, extract what I needed into a CSV for processing in a spreadsheet, and it didn't take all that long. Winner. 🙂

This morning, I read about Level Lock and it is exactly what I've been hoping to see in a smart lock for my home. It's just the guts of the deadbolt. The facade - even the keys are your old keys - and the guts are smart. Very.

The basic functionality is similar to many of the others - HomeKit aware, with the ability to lock/unlock remotely, and give people "keys" for times of day, etc. It's all pretty standard. But the fact that it in no way effects your ability to use the same key as before, but adds the smart lock features - well... that's what sold me.

So I put my name on the waiting list for early 2020, and we'll see if it ships. 🙂

With the upgrade to macOS 10.15 Catalina, I noticed that Adium is not functioning as it should. Specifically, that the chat windows are not auto-scrolling to the bottom on new input to the window. I'm guessing this is something related to the way the internals of Adium, and macOS are updating the contents of the window, but it's also well known that Adium hasn't had the best support in the last few years, so it might not get fixed.

Thankfully, the Forums are still alive, and this was posted recently:

Running version 1.5.10.4 Adium on MacOs Catalina version 10.15.

Previously, in Chats when the list of text boxes was scrolled to the bottom, any additional messages, either typed by user or received from other users, was added to the bottom and the list was scrolled up to reveal it. After upgrading to Catalina yesterday, this autoscrolling behavior has stopped working. When user types a message and hits return, the new message is hidden. Likewise, messages from other users are not revealed by scrolling up. So one has to constantly manually scroll up to see the message thread.

No respondants with ideas - yet, but I have hope that the few that have access to the code can at least make the code available - if not fix the bug, and then we can get back to our lives. But if not, I found InstantBird which seems to have been somewhat abandoned several years ago, but it might offer some utility if Adium is never getting fixed.

I did look for Adium on GitHub and just googling it... but it seems to be a bit difficult to find... Hmmm...

UPDATE: At least I have a few more links... the page for Getting Adium Source is live, and describes how to get the code, and there's a page about Getting the Newest Adium Source as well. In the discussion about this, there's even a note that there exists a BitBucket repo for Adium, where PRs are welcomed.

On the bad news front, the code isn't even close to Xcode 11 ready:

I was going to take a crack at it. I'm fairly good at Objective-C. However, I can't even get Adium to compile with the newest Xcode. What a mess. File encoding issues galore.

So it seems like this may not be an easy fix, but I'll keep looking, and if I get some time, Maybe I'll dive in.

UPDATE: We have a solution... it's in the Javascript for the rendering of the window, and it's in two places:

• /Applications/Adium.app/Contents/Resources/Template.html
• ~/Library/Application Support/Adium 2.0/Message Styles/
  Colloquy.AdiumMessageStyle/template.html

where both might have to change these functions:

  // Auto-scroll to bottom.  Use nearBottom to determine if a scrollToBottom
  // is desired.
  function nearBottom() {
    //return ( document.body.scrollTop >= ( document.body.offsetHeight - 
    //                                      ( window.innerHeight * 1.2 ) ) );
    return 1;
  }
  function scrollToBottom() {
    window.scrollTo(0, document.body.scrollHeight);
    //document.body.scrollTop = document.body.offsetHeight;
  }

If you just fix the one in the app bundle, that's not enough, but you have to fix him. And then you need to fix the one in the Message Style as that's the one that's actually used. Once both are updated to working code, a restart of the app gets the scrolling back into action! 🙂

I really like my Matrix T75 treadmill with the XIR console... and today I checked the Matrix web site to see that they had updated the firmware to 1.4.8.3, so I downloaded it, unzipped it, put it on a USB drive, and then went through the update on the treadmill.

I'm hoping that this fixes the problem I've been having with the control electronics... about the only way I can be sure to run in the morning - uninterrupted, is to wake it up from it's sleep mode, and then pull the kill switch, plug it back in, and then start my run.

If I skip the pull/replace, then one morning in a week - and I don't know which one, will have an issue where the control electronics will think I pulled the kill switch, and stop the run. It's pretty jarring, and I'd like to avoid that, so I go through the ritual of the pull/replace, and I'm pretty safe.

We will see if this update solves that problem... fingers crossed. 🙂

This morning I thought I'd perform the ritual of getting the old web development tools that I've used in the past going again - this time on macOS 10.15 Catalina. Now I haven't used PHP in ages, but I've still got code and databases for Postgres to use that - so it makes sense to get this all working again, and it's always fun to see how things work out.

Getting Postgres 11.1

Loads of coverage here about Postgres, and it's just so simple to get the latest version from Homebrew:

  $ brew install postgresql

I've even posted how to upgrade from major version differences, so it's easy to get the latest Postgres running on your box, and the tools are just superb.

Activating UserDir in Apache 2.4.41

As in the previous updates, the UserDir extension is not enabled by default, so we need to get that going right away. This enables the code to be run from the development directories, and that's a big time-saver. First, we need to enable the UserDir module in Apache, and then make a specific config file for the user in question. Start by editing /etc/apache2/httpd.conf and line 183 needs to be uncommented to read:

  LoadModule userdir_module libexec/apache2/mod_userdir.so

and then similarly on line 520 uncomment the line to read:

  Include /private/etc/apache2/extra/httpd-userdir.conf

Next, make sure that the file we just included is set up right for including the user directories. Edit /etc/apache2/extra/httpd-userdir.conf and line 16 needs to be
uncommented to read:

  Include /private/etc/apache2/users/*.conf

At this point, you need to make sure you have at least one file in the /etc/apache2/users/ directory for each user, like: drbob.conf:

  <Directory "/Users/drbob/Sites/">
      Options FollowSymLinks Indexes MultiViews ExecCGI
      Require all granted
  </Directory>

where the last line - Require all granted is new as of Apache 2.4, and without it you will get errors like:

  [Thu Dec 18 10:41:32.385093 2014] [authz_core:error] [pid 55994]
    [client fe80::7a31:c1ff:fed2:ca2c:58108] AH01630: client denied by server
    configuration: /Users/drbob/Sites/info.php

Activating PHP in Apache

The mext thing to do is to activate PHP in the supplied Apache 2 with macOS 10.15. This is line 186 in the file - /etc/apache2/httpd.conf and you need to uncomment it to read:

  LoadModule php7_module libexec/apache2/libphp7.so

and then verify a file called /etc/apache2/other/php7.conf exists and contains:

  <IfModule php7_module>
    AddType application/x-httpd-php .php
    AddType application/x-httpd-php-source .phps
 
    <IfModule dir_module>
        DirectoryIndex index.html index.php
    </IfModule>
  </IfModule>

which does all the other PHP configuration in a separate file to make upgrades easy.

Finishing Up

At this point, a simple restart of apache:

  $ sudo apachectl restart

and everything should be in order. Hit a URL that's a static file with the contents:

  <?php
    phpinfo();
  ?>

and you should see all the details about the PHP install - including the PostgreSQL section with the version of Postgres indicated:

What's really great is that Apple has included lots of support in the default PHP install:

• PHP 7.3.8
• Postgres 9.3.7
• MySQL 5.0.12
• SQLite3 3.28.0

so there's no reason to do anything more to get the kind of support that I used to get. And I get the other databases for free. This is great news! I then run my little test page to make sure the database connection is working:

and everything is working exactly as expected!

This morning I decided it was time to install macOS 10.15 Catalina on my personal MacBook Pro. There are plenty of folks warning about the loss of 32-bit apps, and I've done an inventory of what I'll loose, and I'm OK with all of those. Sure, I purchased some, but I've gotten a lot of good use out of them, and nothing I'm losing is not already replaced by something that's better.

So off I go to install the upgrade...

I will say, they underreported my 32-bit apps, but that's OK, I know I'll just go through my Applications folder after the update and trash what I no longer need - or can use. But that's OK... it's nice to remind people.

There are quite a few differences in macOS 10.15 - and System Preferences is one of the biggest. I don't mind that they need to make these changes, but the need for me to log out of iCloud, and then log back in seems to be a bit annoying - as it looses all my Apple Pay cards, etc., but I can easily put them back.

One of the two real issues is that Mail.app is different on the "Old School" split view, and now you can't control the headers - at all. No selection of which ones to show, or where to put them, or hoe big to make them... it's just "Here you are" - and we live with it. Maybe they'll put some of that configurability back - I sure hope so.

The second issue is with Adium, my IM Client that I use every single day to chat with folks. In this update, the text view of the chats doesn't auto-scroll to the bottom to show new lines on the view when new lines are added. I don't know if this is meant to be this way, or if it's something that you can change with Xcode - but it's kinda annoying, and I was hoping to find a more supported alternative. I may have to get the Adium code and rebuild it myself with a fix. There just aren't any really good alternatives.

I'm sure there will be a lot more... I had plenty of issues with the Apple subscriptions, but that's just a one-time annoyance... shouldn't be ongoing... but we will see.

Yesterday I had an interesting phone screen with someone, and the seemingly common practice of asking a candidate to code on the phone - in a shared web-based environment again came up. I recognize that any employer can have any legal criteria for employment, and the "Coding Phonescreen" is a very common one. You get to see if the person can write code in a few minutes as opposed to inviting them for a day-long interview cycle that can cost considerably more. It's decent logic.

But it really doesn't tell the story, does it?

Speed Coding has about the same relationship to real development as a college classroom has to Jeopardy!... yeah, the material is the same, but the skills to be able to do well in one do not necessarily translate to the other. And the most critical skill in the speed forms is pattern recognition of the problem.

If you've seen this problem before, and written a simple solution to it, then you're in good shape. You know the flow, you know the pitfalls, and you can talk your way through it - like you're talking your way through directions to a local restaurant. No pressure, you're showing someone something you know, and it happens to take a few steps. No biggie.

But if you're at all unsure, then you're not going to get time to think about the solution before being expected to answer it. This is the problem with Speed Coding - if you know the answer, it's fine. But then it's not really seeing if you can think on your feet... if you don't know the answer, you're likely going to make one or two edge-case mistakes, and those will be clearly visible to the person that knows the solution.

The problem I ran into was a binary tree issue, and while I had been practicing my Speed Coding in Clojure, the nature of the binary tree really leaned towards a C++ solution, and that was not horrible, but it was a lot less friendly to edge-conditions.

I ended up writing something like this:

  struct Node { int value; Node *left; Node *right };
 
  bool stored(Node *me, op) {
    bool   retval = true;
    if (retval && (me->left != NULL) && (me->left op me->value)) {
      retval = stored(me->left, op);
    }
    if (retval && (me->right != NULL) && (me->value op me->right->value)) {
      retval = stored(me->right, op);
    }
    return retval;
  }

and the missed edge-case is that once you are one, or more, steps down in the tree, it's possible to have the relative position of the values be correct, but the absolute position to be false. There are two ways to solve this, in this code:

1. Pass limits down with the call - this could be done with max and min arguments and then in the recursive calls, place the correct values there, and test them as well.
2. Scan up the tree on each check - this could be a walk-up the tree and check to see that you aren't in violation of the location you have. This would take more time because of all the walking, but it'd work.

But what I'd wanted to do was to write it in Clojure, but the data structure didn't jump out at me. Until this morning. 🙂 This morning I spent the minute or two thinking about the data structure, and then formulated the following solution:

  ;; [val left right]
  ;;       10
  ;;    5      15
  ;;  1   7  12   20
  (def good [10 [5 [1] [7]] [15 [12] [20]]])
 
  ;;       10
  ;;    5      15
  ;;  1   17  12   20    -- the 17 is out of place
  (def bad [10 [5 [1] [17]] [15 [12] [20]]])
 
  (defn sorted?
    "Function to check a binary tree to see if it's sorted for proper
     searching."
    [[v lt rt] op]
    (let [ltv (first lt)
          rtv (first rt)]
      (and (or (nil? lt) (and (every? identity (map #(op % v) (flatten lt)))
                              (check lt op)))
           (or (nil? rt) (and (every? identity (map #(op v %) (flatten rt)))
                              (check rt op))))))

What I really like about this solution is that it checks the entire subtree with the operation. This means that the effort to do one, is really checking all of them. This is what I wanted to write, and it works perfectly.

But I didn't force the issue, and pull back and take the time to think. My mistake. I won't make it again.

UPDATE: a friend and I were talking about this same problem, and he came up with a solution that was very clever - the structure can be validated by simply assuming that the structure is a sorted binary tree, and then calculating the min and max values of the tree.

The catch being that if you get to a node where the current value isn't within the min and max, then you have to fail, and return nil. It's really quite amazingly simple in that it's very fast, very easy to understand and adds the additional benefit of returning the extremum of the tree.

  (defn bst-rx*
    "returns extent [min max] of valid bst or nil if invalid"
    [t]
    (cond
      (nil? t) nil
      (vector? t) (let [[v l r] t
                        lx (if l (bst-rx* l) [v v])
                        rx (if r (bst-rx* r) [v v])]
                    (when (and lx rx (<= (lx 1) v (rx 0)))
                      [(lx 0) (rx 1)]))
      :else [t t]))
 
  (defn bst-rx?
    [t]
    (boolean (bst-rx* t)))

Yesterday morning, I thought to check my blog and see something, but to my complete surprise, it was down. Like very down, and I had no idea what was happening. I've been using WordPress and HostMonster for more than a decade, and I've never had anything even remotely like this. I didn't know where to start - so I went to their admin site, thinking I'd been hacked somehow...

First, it was clear that they had done a ton of upgrades to the host and its support platform. Massive changes. So the first thing was to get logged in. This was a little odd because it wasn't working on Safari, and it had been in the past - so I switched to Chrome, and finally got logged in. Step 1 - accomplished!

Then I looked at the installed users on each of the three WordPress sites I have, and in each case, there was a user that I couldn't explain. It certainly appeared to me that these were bad actors, and I had no idea how they got there. I stay up to date, don't allow logins, don't allow replies... it's a Journal more than anything else. But still... I could not deny these accounts. So I asked for help.

It took a long while to figure this out, but in the end, the logs for the site indicated that there was a PHP problem in one of my plugins, and one of my themes. Why this happened yesterday wasn't at all clear, but it became clear as I dug further.

HostMonster had dropped support for PHP 5.x, and the only versions available to me were 7.0, 7.1, 7.2, and 7.3, with the latter being the default. Now it seemed to be clear what had happened... nothing had changed, but in all the upgrades for the infrastructure on the hosts, they had switched to a different config for PHP, and the plugin and theme were doing something that they shouldn't. OK... now to the code.

The first one I tackled was the plugin, as the error in the logs was pretty clear. I did a quick search for =& and sure enough, it was a PHP 5-ism, and that was easy to fix. That solved the plugin problem, and it loaded and ran fine. The theme had a different problem with a deprecated function that wasn't really even needed in the theme, but I found a replacement, and used that, and the theme was fine.

All told, this took more than 5 hours... it was not quick, and I just ahead of the part where I found out that the timezone plugin I was using wasn't needed in WordPress 5, and so I didn't put that back into play, Also, when I got the site up, it was possible to see the errors on activation of the plugin (but not the theme), which made continued debugging a lot easier.

In the end, it was all cleaned up, and now it's set for PHP 7. I'm glad that there wasn't a bigger issue, but I really have to be careful of these things because there is almost no support for the plugin and theme - and I really like to have both of them working for me for this blog. 🙂

This morning I finished adding in the directional vectors to the display of the electric field on the Potentials app. Previously, we had the color indicate the magnitude of the field at each point, and then the contours were for lines of equal strength. But that's not really all that helpful for an electric field map - better, that each point have a little arrow indicating the direction of the field at that point. Then, you can see the shifting direction of the field, and where a test charge would feel force in the field.

The approach was to start with the points for line segments for the graphics primitives, and then modify them to suit the needs of each point. For example, assume an arrow, pointing to the right, lying on the x-axis, centered at the origin, and fitting with the unit square. There are five endpoints needed to draw these lines in one stroke, so lay them out. This is the starting point for all arrows.

Then we need to rotate it by the angle of the electric field at each point. This is in radians, and we can do this with the CGAffineTransform:

  CGAffineTransformMakeRotation(dir)

where dir is the angle of the field at that point.

Then we can scale the arrow for the size of the box in the view with:

  CGAffineTransformMakeScale(fac, fac)

because we want to look at the minimum size of the box, and call that fac.

Finally, we can move the arrow to the correct place in the view with:

  CGAffineTransformMakeTranslation(x+dx/2.0, y+dy/2.0)

And of course, we can use CGAffineTransformConcat to put these all together into one transformation that can then be applied to each point in the line segment:

  CGAffineTransform    xfrm = CGAffineTransformConcat(
                                CGAffineTransformConcat(
                                  CGAffineTransformMakeRotation(dir), 
                                  CGAffineTransformMakeScale(fac, fac)),
                                CGAffineTransformMakeTranslation(x+dx/2.0, y+dy/2.0));

because you can't call CGAffineTransformConcat with three arguments.

The results are just fantastic:

and it's easy to track where the test charge will feel the force of the field in every point in the workspace. This is just what we were hoping to get, and it wasn't all that hard with a standard arrow, and then a set of transformations.

Nice. 🙂

Just a little update on the Apple Card, and how it's been to use, now that I've had the physical card in-hand for a week now. In short - the card is physically very nice - I like the heft and the fact that it's got no identifying marks on it but my name, and that it doesn't get deformed in my wallet. But what's really setting it apart is the Wallet app on my iPhone.

I had been using the Capital One Quicksilver card, and still carry it now - but all my recurring expenses, and online services, have moved over to the Apple Card. The Capital One app had notifications, and that's become pretty much table stakes for credit cards these days - my Bank has it, Capital One, and Apple. But it's the frictionless nature of the Apple experience that is really what this card means to me: Service.

A card is a card - I'm one of the very lucky ones that doesn't have to worry about interest rates - I pay it off every month - without fail. So the limit matters, but only a little, and so it's really the services, the help in tracking purchases, verifying that they are correct, and in a timely manner - that I really care about. And in that regard, the Apple Wallet app is just going to be so much better at it than the Capital One developers - because they only have to build for one platform (theirs), and they have the advantage of handling security in any way they want.

It's made it much easier to see what's happening, keeping track of the charges/purchases, and the easy way to see what gets the different cash-back percentages has been baked-into the UI on the list of charges. It's really quite good. But that's the point - Apple is going to be focused on the Service and experience of the user, and not on the financials that another issuer will be.

I have to say, I'm really enjoying it. 🙂

UPDATE: I followed the advice on Daring Fireball to opt-out of the arbitration clause on the Apple Card, and it could not have been easier. Everything was done in the Messages app... just go into the Wallet, hit the three-dots, and select Message, and text that you want to exercise your right to reject the arbitration clause.

They will transfer the chat conversation to a Goldman employee, and then they will see it, and mark you down as having opted-out of the arbitration clause, and that's it. No mess. No fuss. This is the way technology should empower us.