What's it all about, Alfie?

I have always liked redis as a wonderful example of very targeted software done very well. It's a single-threaded C app that does one thing, very well, and in the years since I started using it, it's only gotten better. As I've been building a new Clojure web app, one of the things I wanted to take advantage of, was the stored session state, specifically so that when I have multiple boxes running the code, they can all share the one session state - and quickly.

I've been getting my WebAuthN authentication going, and that is using session state as the place to store the :identity of the logged in user. After I worked out the serialization of the Authenticator for WebAuthN, I then turned my attention to persisting the session state with Ring's SessionStore.

I've started using Ring's wrap-defaults middleware. It's easily added to the options for wrap-defaults with:

  ; pull in the wrapper and base defaults
  (:require [ring.middleware.defaults :refer [wrap-defaults site-defaults]])
 
  ; augment the site defaults with the SessionStore from Carmine
  (def my-site-defaults
    (assoc site-defaults
      :session {:flash true
                :store (carmine-store)}))
 
  ; put it in the stack of the app routes...
  (-> app-routes
      (wrap-access-rules {:rules rules :on-error unauthorized-handler})
      (wrap-authorization backend)
      (wrap-authentication backend)
      wrap-user
      wrap-json-with-padding
      (wrap-defaults my-site-defaults)
      wrap-logging
      wrap-gzip)))

I've been a huge fan of carmine for many years, as it's a pure Clojure library for redis, and it's exceptionaly fast.

But first, I needed to install redis locally so that I can do laptop development and not have to have a connection to a hosted service. Simply:

  $ brew install redis

and then to make sure it's started through launchd, simply:

  $ brew services start redis

just like with Postgres. It's really very simple.

At this point, restarting the web server will automatically store the session data in the locally running redis, and for production deployments, it's easy enough to use redis cloud, or redis at AWS, Google Cloud, etc. It's super simple, and it's exceptionally reliable.

The web server can now be restarted without impact to the customers as redis has their session state, and postgres has their Authenticator for WebAuthN.

I have been trying to set Vim settings, specific for text files, in my .vimrc for the longest of time, and I really wanted to get this figured out this morning. So here we go.

It's all about the autocmd, or au, command, and you can set it up pretty easily. What I had in the past was:

  autocmd FileType md
  \    set ai sw=2 sts=2 et
  autocmd FileType txt
  \    set ai sw=2 sts=2 et

where, in these two examples, I'm trying to set the autoindent to ON, the shiftwidth to 2, the softtabstop to 2 and expandtab to ON. And it wasn't working for files like goof.txt and for the life of me I could not figure this out.

And then it hit me - I had a FileType of javascript in the file, what if Vim needed to have text for the FileType? So let's try:

  autocmd FileType markdown
  \    set ai sw=2 sts=2 et
  autocmd FileType text
  \    set ai sw=2 sts=2 et

And all of a sudden, it worked like a charm. I also tried markdown using the same reasoning. I guess that goes to show me that Vim is interpreting the file extension, and not literally using it as the FileType. Smart. Appreciated.

I noticed that Postgres 16 was updated in Homebrew, so I took the time to upgrade the installation on my MacBook Pro, and it was remarkably easy. Because it was a simple "dot" upgrade, the installer did all the heavy lifting:

 $ brew upgrade postgresql@16

and then when it was done, simply restart the service with:

 $ brew services restart postgresql@16

And that was it. Everything is up and running just fine. What a treat. 🙂

This is the second "dot" upgrade I've done with Postgres 16 and Homebrew, and I just can't get over how clean and simple it is. I've checked all the databases, and they are good, and everything is fine.

This morning I was thinking about the interrogation of the Promise in Node, and it was a little bothering to me that there was no way to easily see if it was complete. Then I thought - Maybe there is a different way? So I decided to give this a try.

The use-case is this: I want to be able to fire off a request to another Service, and maybe it'll get back to me in time, and maybe it won't. Either way, I don't want to wait. I have other work to do that must get done. So I want to fire this Promise off, and then, if it's done when I need its result - Great! If not, then Also Great!

But the normal:

  const foo = await callAsyncFunction()

isn't going to work, because that will wait until it's done. So how to work this out?

It turns out that it's not too hard.

  const { setTimeout } = require('timers/promises')
 
  const runTest = async () => {
    let done = false
    const foo = setTimeout(5000).then((val) => done = true)
    for (let i = 0; i < 10; i++) {
      console.log('FOO', foo, done)
      await setTimeout(1000)
    }
  }
 
  runTest()
    .then(() => console.log('All done running Timing test.'))
    .catch(console.error)
    .finally(() => process.exit())

and when I run this:

  $ node foo.js
  FOO Promise { <pending> } false
  FOO Promise { <pending> } false
  FOO Promise { <pending> } false
  FOO Promise { <pending> } false
  FOO Promise { <pending> } false
  FOO Promise { true } true
  FOO Promise { true } true
  FOO Promise { true } true
  FOO Promise { true } true
  FOO Promise { true } true
  All done running Timing test.

So all we need to do is to have a variable that indicates the state of the completeness, and then return that in the .then() call. Sure, it may make a lot more sense to have:

    const foo = setTimeout(5000).then((val) => {
      done = true
      return val
    })

so that we get the value back into foo, but that's easy... the point is to toggle the variable in that .then() and query that, as needed.

This way, I don't have to worry about any unsupported ways of finding out, it's simple. 🙂

I noticed that Postgres 16 was updated in Homebrew, so I took the time to upgrade the installation on my MacBook Pro, and it was remarkably easy. Because it was a simple "dot" upgrade, the installer did all the heavy lifting:

 $ brew upgrade postgresql@16

and then when it was done, simply restart the service with:

 $ brew services restart postgresql@16

And that was it. Everything is up and running just fine. What a treat. 🙂

Since it was announced, the JSONB field in Postgres has been one of the most useful fields I've encountered. It's the ability to have complex data as a single field, with very powerful querying tools into the data - as needed. But one thing that has always been a little tough for me is the manipulation of the data, in SQL, so that it stays as JSON, but it's been altered from it's as-stored value.

Let's say we have a table that has a JSONB field that has an array of Objects, each looking something like this:

  {
    "id": 101,
    "size": "large",
    "score": 91,
    "count": 1232
  }

and there can be more, but what you really want in your SELECT is just an Array of the id and score, and nothing else. It still needs to be an Array, and each element needs to be an Object, but a smaller Object, with just those two attributes.

From the docs, Postgres has jsonb_array_elements() that can turn an Array into something that looks like a table... and we can use the ->> notation to get at the value of an attribute, but how does it all fit together to disassemble, modify, and re-assemble all the elements? A sub-select.

Let's walk through one.

  SELECT jsonb_agg(
           jsonb_build_object(
             'id', t.value->>'id',
             'score', t.value->>'score'
           )
         )
    FROM jsonb_array_elements(c.targets) t

the json_build_object() will take the key/value pairs it's given, and create a new Object. The source data for this is t and is the output of the jsonb_array_elements() function in the targets field, which is the JSONB file that is holding this data.

Then to pull this together, you might have a query like:

  SELECT c.id, c.name,
         (SELECT jsonb_agg(jsonb_build_object(
                   'id', t.value->>'id',
                   'score', t.value->>'score'))
            FROM jsonb_array_elements(c.targets) t
           WHERE t.id = c.id) AS targets
    FROM company c
   WHERE ...

It's not easy... but it's powerful, and there are a lot of things that can be done with this kind of manipulation... it's just going to take a little practice. 🙂

I thought about it for a bit, but the boost in performance, and the doubling of the memory were really the key points that made it a decision worth making. So I ordered it. A new M3 Max MacBook Pro - Black, 128GB RAM, and 2TB storage (seems silly to call it a disk, or drive anymore). And the target delivery date is around the end of the month. Not bad.

After watching the Scary Fast announcement, it was clear that skipping the M2 Max was going to make this jump a much more significant one, and I was right. Also, there's nothing really wrong with my existing M1 Max MacBook Pro, but the doubling of memory, and 50% increase in speed is going to be something I will use every single day.

The new Space Black color looks good, and I'm sure it'll be just fine with regards to the fingerprints mentioned in so many reviews, and it'll be nice to see it next to my iPad Pro that's similarly dark... it should be a nice addition. 🙂

This morning I'm very happy to see that the issue I've been having with macOS Sonoma 14.0 appears to be gone. I like the upgrade, for the most part, but what I'd noticed was that the memory usage for Safari and Safari Technology Preview would rise to the point of crashing the system. This mean that I had to have Activity Monitor running all the time to make sure the web pages that were getting too big were reloaded, or dropped, when their footprint got the memory pressure into the "yellow" - before it went "red".

I had expected that this was a dot 0 issue, and I was right - with the 14.1 update earlier this week, the memory footprint has started low, and stayed there - for a few days. Now I'll probably run Activity Monitor through the weekend, just to make sure, but I have a good feeling that this is something that got cleared up, and I'm not going to see a recurrence of the problem.

I have enjoyed macOS, and the Mac System long before Cocoa and Foundation, but this is something I am glad to see I was right about. They move forward, but pay attention, and fix the little things as they go. What a great team. 🙂

This morning I noticed that not only was Postgres 14.9 out, they had released 15.x and even 16.0. It's unusual for me to be a full major version behind on my main laptop, but to be two was just something that had to be corrected.

Several months ago, it was clear that the official Postgres builds were no longer being done by the Postgres group, and so the support for 15.0 wasn't in Homebrew. I figured it'd just be a little bit, and then things would start back up again. But that was not the case. What happened, instead, was that the Homebrew volunteers took it upon themselves to build the packages for 14.x, 15, and now 16.

So let's write this all down so it's easy to do next time we change a major version of Postgres. Start by saving everything in all the databases:

  $ pg_dumpall > dump.sql
  $ brew services stop postgresql@14

Now we can wipe out the old install and it's data:

  $ brew uninstall postgresql@14
  $ rm -rf /opt/homebrew/var/postgresql@14

Now we install the new version, start it, and load back up the data:

  $ brew install postgresql@16
  $ brew services start postgresql@16
  $ psql -d postgres -f dump.sql
  $ psql -l

If the command psql doesn't show up in the path, just relink the package:

  $ brew link postgresql@16

Then it should be in the right spot.

At this point, it's all loaded up and you can ditch the dump.sql file, as it's no longer needed, and the new version is active:

  $ psql --version
  psql (PostgreSQL) 16.0 (Homebrew)

Not bad at all. 🙂

There are many times when you would like a sub-select query to be constrained on one of the values of the main query, but when you attempt to do that you get an error message about not being able to use the variable in this context. For example, this query:

  SELECT c.id, c.company_name, pb.available, pb.current,
         date_trunc('second', pb.as_of) AS as_of, pbs.*
    FROM companies c, plaid_tokens pt, plaid_balances pb,
         (SELECT SUM(available) AS all_available,
                 SUM(CURRENT) AS all_current
            FROM plaid_balances WHERE company_id=c.id) pbs
   WHERE pt.id = (c.additional_info->>'primaryPlaidAccount')::uuid
     AND pt.account_id = pb.account_id

where the goal is to have a sub-select gather the sum of the individual columns being pulled in the main query. It's a nice thing to have, but the inability to have c.id used in the sub-select really makes it difficult.

Postgres has a nice feature in LATERAL, where is allows the sub-select to reference these fields by changing the order of evaluation of the sub-select and doesn't penalize the performance too much.

  SELECT c.id, c.company_name, pb.available, pb.current,
         date_trunc('second', pb.as_of) AS as_of, pbs.*
    FROM companies c, plaid_tokens pt, plaid_balances pb,
         lateral (SELECT SUM(available) AS all_available,
                         SUM(CURRENT) AS all_current
                    FROM plaid_balances WHERE company_id=c.id) pbs
   WHERE pt.id = (c.additional_info->>'primaryPlaidAccount')::uuid
     AND pt.account_id = pb.account_id

This is still quick, and it saves the machinations of having to calculate the sums in a temp table, or write a function to do this... it's just a nice little trick that they put in the language. Very considerate. 🙂