What's it all about, Alfie?

Many years ago, I built a market data server based on the Bloomberg API that was available on Solaris - back in the days when the Bloomberg Terminal was available on Solaris. In that code, I needed to solve a problem of thread starvation and in order not to confuse the Next Guy - which could have been me, I made sure to comment this code to get a reference to the "housekeeping mutex" in the object. This was all C++.

To this day, this is one of my favorite comments. It's just exactly what I want to write, tell people I think they can write:

/*
 * When any client or chat command wants to do something with the
 * server they need to get my housekeeping mutex and place a read
 * lock on it. The reason for this is that we are going to have
 * times when the controller knows that things aren't really stable
 * in the system and therefore we need to hold off on doing certain
 * things. Since this is a read-write lock, most of the time things
 * will run along swimmingly, but when there is maintenance underway
 * we will obtain a write lock and make the normal clients wait for
 * the maintenance to be done.
 */
CKFWRWMutex *BBGServerController::getHousekeepingMutex()
{
    /*
     * With so many readers (clients) hitting the server at all hours,
     * we run into the problem that the write lock is almost impossible
     * to get. This is further hanpered by the fact that the pthread
     * read-write mutex doesn't specify the writers as having a higher
     * priority than the readers. So we can get a writer starvation.
     *
     * The solution is to have another mutex "in front" of the read-write
     * mutex that controls everything. The way this works is that all
     * clients need to call this method to get the housekeeping mutex.
     * The first thing they'll need to do here is to get a lock on the
     * "out of order" mutex and then release it right away. Most of the
     * time this will happen with little to no delay. However... when the
     * major housekeeping tasks arise, they lock this guy and *leave it
     * locked* so that the write lock that comes after it can wait for
     * the pending readers to get done and *then* obtain it's write lock.
     * 
     * When the housekeeping maintenance is done, we unlock the write lock
     * and then unlock the "out of order" lock and this method can resume.
     * It's a clean process that allows me to prioritize the write lock
     * above the readers which is as it should be in this application.
     */
    mOutOfOrderMutex.lock();
    mOutOfOrderMutex.unlock();
 
    return & mHousekeepingMutex;
}

Writing comments like this just makes me smile.

This morning I decided to turn on two-factor authentication on my GitHub account using the Authy app for my iPhone. I've been able to use it for different accounts, and it's always mine - not the company I work for, and while I've been using SMS for a while on GitHub - and others, I've just decided that it's probably a good thing to get moving on this = given the privacy issues that we are all reading about these days.

The thing that I needed was a simple file encryption bash script/function so that I could store the recovery tokens in a file without having to worry about it getting snatched. So now I've got it. The code is based on openssl, and it's pretty simple:

#
# These are simple functions to encrypt and decrypt files so that I don't
# have to hassle with extreme things in order to secure one file at a time.
# They use openssl to do the work, and it's pretty simple.
#
function jack {
    openssl des3 -in "$1" -out "$1.enc"
}
 
function unjack {
    openssl des3 -d -in "$1" -out `basename "$1" .enc`
}

and this simply allows me to encrypt a file and it adds .enc on the end of the filename, and then I can decrypt it as well, stripping that addition as well. Nothing fancy, but it really works just great.

This morning, I read this article about a problem in two small towns in Massachusetts when the engineers responsible for a normally reliable, safe, natural gas distribution infrastructure don't really think about what they are doing, and the consequences of their designs. It's shameful.

The NTSB report about what happened was very clear, and very troubling:

The contracted crew was working on a tie-in project of a new plastic distribution main and the abandonment of a cast-iron distribution main. The distribution main that was abandoned still had the regulator sensing lines that were used to detect pressure in the distribution system and provide input to the regulators to control the system pressure. Once the contractor crews disconnected the distribution main that was going to be abandoned, the section containing the sensing lines began losing pressure.

As the pressure in the abandoned distribution main dropped about 0.25 inches of water column (about 0.01 psig), the regulators responded by opening further, increasing pressure in the distribution system. Since the regulators no longer sensed system pressure they fully opened allowing the full flow of high-pressure gas to be released into the distribution system supplying the neighborhood, exceeding the maximum allowable pressure.

Why the control system wasn't smart enough to detect:

• Pressure Reading of Zero - the sensor was on the abandoned line - which had no pressure. Why didn't that fact alone cause an alarm?
• Zero Change on Output based on Input Change - when the first change was done, why wasn't the change seen on the sensor - at least in a percentage measure? Even, if you look at the first-derivative of the sensor reading, a simple thing to do even with analog control systems, and see that no change could be seen should have been an error.
• Upper Limit on Input - when you see that you have increased the input some percentage in the last "...few minutes..." - then sound the alarm. There is no reason to ramp the input like this. It's a leak - or worse, and needs to be corrected.

As an engineer, I'm stunned that the folks that created this control system didn't do these, or something similar, to them to sound alarms as opposed to letting four homes blow up. I mean, it's one thing to have a leak, with a bad scent in the house - but to actually have homes blow up - that's way past bad design.

An old friend and I have talked about this many times over the years - How do you really teach this kind of design? - and it's not easy. It comes from experience, and the problem is, most engineers don't have it - any more than most people are wise. You have to accumulate experience over years - to know what to think about... what to look for... all that needs to be considered, and it takes years to really gain that perspective.

But companies will pass things like this off to the junior engineers - because that's who they believe can do all that's needed. And if you ask the folks doing the work if they are ready to do this kind of work, they are always going to say "Yes!". That's just "confidence".

Until we come to terms with this gap in the education of engineers, we're going to have these problems, and they could be a whole lot worse.

Today I decided to open up some of the fortran code I wrote for my thesis - 30 yrs ago, and see if Sublime Text 3 would handle the syntax highlighting, and therein brought up an interesting point about the problem I had in upgrading from Sublime Text 2 to 3. So it made sense to write it all down just in case it happens on another upgrade in the future.

I found a nice Fortran package Sublime Text 3 - it did all the normal syntax highlighting stuff, and lots more. Very slick that it picks out the LaTeX formatting in the comments - Who thinks of this? Wild! So it seems like a great little find. Sweet.

I go to install it, using Package Control, and I noticed that Package Control wasn't installed. That's odd... I know I had it installed - maybe it didn't come over in the upgrade? So, let's install it again... No biggie.

Well... kinda... When I following the installation instructions, I didn't see it show up in the pop-up. Very odd. I checked the directory structure - it was there... but it wasn't showing up. Very odd...

So I Googled the issue, and it seems that it's something that happened enough to warrant a tip - it might be disabled. Go to the general settings, a JSON file, and then see if it's in the list of disabled packages. Sure enough - there it was! I simply deleted it from the config, and restarted Sublime, and Bingo!

After that, everything was working just fine:

I know we all just want to have some sense of satisfaction in our lives. To feel that we matter - somewhere. To feel that we make a difference. It's universal... but so many don't have that because of circumstance or society. But that's another issue. What we, in the US Tech Industry have created is a class of people that feel entitled to things that they have no entitlement to. My sister calls these The Soccer Trophy generation.

If someone is told - at every possible turn - that they are amazing, valued, better than the rest, then their sense of reality is being formed in those moments. If all you allow a person to see is a mirror, reflecting back to them that they are the center of the world, then they have no opportunity to see what reality is like outside those mirrors. It's not surprising then, when these people get angry when the context is changed, and it's time to work together with others of differing opinions, and then all of a sudden, things go very badly.

Over the last five years, I have been working with folks that fit this profile perfectly. They are fine people - at least in comparison to most federal inmates. But they are employees that feel the company owes them a sense of purpose, a job that fulfills them... and they have every intention of getting that. But that's not how the world really works. Sometimes it's just a lot of hard work for very little short-term payoff. Life isn't fair. But telling this to our little monsters usually creates such a stink that it's avoided at all costs.

I'm all for being nice. I'm all for being generous. I think these are fantastic qualities, and I try to be as graceful as I can be. All the time. But some days I just want to say "Someone did you a disservice by making you think the world revolves around you. It doesn't." Now let's have a little moment to deal with that, and let's get back to work!

But that's not done... Go figure.

I was updating my macOS 10.14 Mojave install to get Apache + PHP + Postgres working, and I realized that I probably never wrote up this great Postgres client for macOS - Postico. When I saw it, and how complete and efficient the UI was, I knew I had to buy it - just to support more development like this.

Normally, I'm a psql guy because I'm far faster on the command line than I am in a GUI - it's just the number of hours and the fingers are always on the keyboard. Plus, let's face it - the feature set of psql is very impressive - I've got it in scripts, and you can pass in variables for insertion into your scripts... it's just pretty amazing. But still... great design needs to be supported, and this is one that really needs the support.

There are so many things I like about this UI. First, it's efficient - you have a query window, and then a table of results. I wrote something like this many years ago, and I had the same UI - but I stopped there. They didn't, and I'm glad they didn't. They then filled in the window title bar with the connection information and the status of the connection. They even let me know if it's local or remote. Lots of good stuff there at a quick glance.

Then we can look at the footer of the window - we can export this data in a variety of forms, and it lets us know the row count, if the query was successful, and the time required. This is all great stuff, and it doesn't chew up a ton of screen real estate, but it's all there - efficient

There is a lot more to this guy - as a Mac app, you'd expect multiple connections to different databases - and you get that as well. I've never really needed it - but again, I'd just open up several terminal windows and then run psql in each. But it's great to know that they spent the time to get all the details right.

Anyway... maybe I'll use it a lot - someday... but I love supporting great designs, and this is a really good one.

This morning I thought I'd perform the ritual of getting the old web development tools that I've used in the past going again - this time on macOS 10.14 Mojave. Now I haven't used PHP in ages, but I've still got code and databases for Postgres to use that - so it makes sense to get this all working again, and it's always fun to see how things work out.

Getting PostgreSQL 10.3

Loads of coverage here about Postgres, and it's just so simple to get the latest version from Homebrew:

  $ brew install postgresql

I've even posted how to upgrade from major version differences, so it's easy to get the latest Postgres running on your box, and the tools are just superb.

Activating UserDir in Apache 2.4.34

As in the previous updates, the UserDir extension is not enabled by default, so we need to get that going right away. This enables the code to be run from the development directories, and that's a big time-saver. First, we need to enable the UserDir module in Apache, and then make a specific config file for the user in question. Start by editing /etc/apache2/httpd.conf and line 174 needs to be uncommented to read:

  LoadModule userdir_module libexec/apache2/mod_userdir.so

and then similarly on line 511 uncomment the line to read:

  Include /private/etc/apache2/extra/httpd-userdir.conf

Next, make sure that the file we just included is set up right for including the user
directories. Edit /etc/apache2/extra/httpd-userdir.conf and line 16 needs to be uncommented to read:

  Include /private/etc/apache2/users/*.conf

At this point, you need to make sure you have at least one file in the /etc/apache2/users/ directory for each user, like: drbob.conf:

  <Directory "/Users/drbob/Sites/">
      Options FollowSymLinks Indexes MultiViews ExecCGI
      Require all granted
  </Directory>

where the last line - Require all granted is new as of Apache 2.4, and without it you will get errors like:

  [Thu Dec 18 10:41:32.385093 2014] [authz_core:error] [pid 55994]
    [client fe80::7a31:c1ff:fed2:ca2c:58108] AH01630: client denied by server
    configuration: /Users/drbob/Sites/info.php

Activating PHP in Apache

The mext thing to do is to activate PHP in the supplied Apache 2 with macOS 10.14. This is line 177 in the file - /etc/apache2/httpd.conf and you need to uncomment it to read:

  LoadModule php7_module libexec/apache2/libphp7.so

and then verify a file called /etc/apache2/other/php7.conf exists and contains:

  <IfModule php7_module>
    AddType application/x-httpd-php .php
    AddType application/x-httpd-php-source .phps
 
    <IfModule dir_module>
        DirectoryIndex index.html index.php
    </IfModule>
  </IfModule>

which does all the other PHP configuration in a separate file to make upgrades easy.

Finishing Up

At this point, a simple restart of apache:

  $ sudo apachectl restart

and everything should be in order. Hit a URL that's a static file with the contents:

  <?php
    phpinfo();
  ?>

and you should see all the details about the PHP install - including the PostgreSQL section with the version of Postgres indicated:

What's really great is that Apple has included lots of support in the default PHP install:

• PHP 7.1.19
• Postgres 9.3.7
• MySQL 5.0.12
• SQLite3 7.1.19

so there's no reason to do anything more to get the kind of support that I used to get. And I get the other databases for free. This is great news! I then run my little test page to make sure the database ocnnection is working:

and everything is working exactly as expected!

This morning I wanted to make sure that Homebrew was up-to-date on my laptop - now that I'd upgraded it to macOS 10.14 Mojave. In the past, changes to the OS and Xcode almost always meant that I had to update with a simple:

  $ brew update

and off it would go - updating the code for brew as well as the list of new, updated, and deprecated packages. This time, it also pointed out that in the upgrade to Mojave, I think Apple cleared out some empty directories. The update instructed me to:

  $ sudo mkdir -p /usr/local/sbin
  $ sudo chown -R $(whoami) /usr/local/sbin

so that the /usr/local/sbin directory was created, and owned by me. Sounds fair - I don't think I had anything there - and we're good to go, but it's really impressive how they have handled all the package management and versioning. I really like the tools these guys make.

I read a tweet this morning about Apache Pulsar achieving top-level project status in the Apache organization. I hadn't heard of it, so I spent a little time reading about it and getting familiar with the differences between it and Storm, Flink, etc. What I found was something that I might actually use in the future.

The important parts are that Pulsar is a message broker with reliable messaging, with storage and fail-over. This makes it way better than many other systems, but it goes a lot further than that. Because it's Apache, it's using ZooKeeper, but it's also using BookKeeper for the message storage - which is an interesting twist.

It's scalable up to a million topics... is meant to sync servers by geo-location, and all the goodies you'd expect from something like Cassandra or the like. Lost of resilience. Turns out, it was the message broker for Yahoo, and has been open sourced. It's low-latency, and meant to be server-to-server in the datacenter, even if they are separated by continents. So it's battle-tested. And all this makes it a useful project because there are lots of times I can imagine needing a better broker than SQS.

But what makes it even nicer is that it has all the Lambda capabilities that AWS has as well. So now you can add in the streaming capabilities of Storm in the messaging system - as opposed to having a streaming system talk to Kafka as a source and sink. This is a different take on the problem, but one of the problems I had with Storm was the scaling of the nodes - it was hard to get them to not back-up. If this is a message broker, then we have a lot more flexibility on the processing time, and every hand-off is to a queue/topic. That's good news.

I'm not sure if/when I'll need to have something like this, but I'm very happy to see that they have it, and it's gotten to a point that it looks pretty stable. Good news!

Withe the recent improvements in performance from Xcode 10 for my Swift code, I thought I wanted to dust off DKit and make sure it still works on macOS 10.13.6 with Xcode 10 - it's been quite a few years since I've made any changes to it, and I just wanted to make sure it all builds and runs.

One of the things I remembered from a few months ago, when I tried to dust it off for a homework project for a job interview and I had some problems with the fact that the shared library was being built as a dylib and that just makes it a lot harder. At the time I originally wrote the library - some 15 yrs ago - there wasn't the support in Mac OS X for the standard unix SO shared library - so I had to use the dylib and figure out how to link it in properly.

Today, the story is entirely different - and honestly, for several years, I've known that macOS could generate the unix SO shared library. So I just switched over to that this morning, and gave it a spin. Turns out it worked perfectly.

I was able to compile all the components of the library, and then the test apps, where the templates are really being used, and all the tests ran. I checked in the simple changes to the Makefiles and everything is solid.

I'm sure there are a lot of things that this could add - I could make a test app that read a file and sent it out on a TCP or UDP transmitter - for the testing of the TCP and UDP receiver, but I'm not worried about that right now. It's enough that it's still working just fine, and builds on the latest macOS. Can't beat that!