What's it all about, Alfie?

This morning I was thinking about the interrogation of the Promise in Node, and it was a little bothering to me that there was no way to easily see if it was complete. Then I thought - Maybe there is a different way? So I decided to give this a try.

The use-case is this: I want to be able to fire off a request to another Service, and maybe it'll get back to me in time, and maybe it won't. Either way, I don't want to wait. I have other work to do that must get done. So I want to fire this Promise off, and then, if it's done when I need its result - Great! If not, then Also Great!

But the normal:

  const foo = await callAsyncFunction()

isn't going to work, because that will wait until it's done. So how to work this out?

It turns out that it's not too hard.

  const { setTimeout } = require('timers/promises')
 
  const runTest = async () => {
    let done = false
    const foo = setTimeout(5000).then((val) => done = true)
    for (let i = 0; i < 10; i++) {
      console.log('FOO', foo, done)
      await setTimeout(1000)
    }
  }
 
  runTest()
    .then(() => console.log('All done running Timing test.'))
    .catch(console.error)
    .finally(() => process.exit())

and when I run this:

  $ node foo.js
  FOO Promise { <pending> } false
  FOO Promise { <pending> } false
  FOO Promise { <pending> } false
  FOO Promise { <pending> } false
  FOO Promise { <pending> } false
  FOO Promise { true } true
  FOO Promise { true } true
  FOO Promise { true } true
  FOO Promise { true } true
  FOO Promise { true } true
  All done running Timing test.

So all we need to do is to have a variable that indicates the state of the completeness, and then return that in the .then() call. Sure, it may make a lot more sense to have:

    const foo = setTimeout(5000).then((val) => {
      done = true
      return val
    })

so that we get the value back into foo, but that's easy... the point is to toggle the variable in that .then() and query that, as needed.

This way, I don't have to worry about any unsupported ways of finding out, it's simple. 🙂

There are many times when you would like a sub-select query to be constrained on one of the values of the main query, but when you attempt to do that you get an error message about not being able to use the variable in this context. For example, this query:

  SELECT c.id, c.company_name, pb.available, pb.current,
         date_trunc('second', pb.as_of) AS as_of, pbs.*
    FROM companies c, plaid_tokens pt, plaid_balances pb,
         (SELECT SUM(available) AS all_available,
                 SUM(CURRENT) AS all_current
            FROM plaid_balances WHERE company_id=c.id) pbs
   WHERE pt.id = (c.additional_info->>'primaryPlaidAccount')::uuid
     AND pt.account_id = pb.account_id

where the goal is to have a sub-select gather the sum of the individual columns being pulled in the main query. It's a nice thing to have, but the inability to have c.id used in the sub-select really makes it difficult.

Postgres has a nice feature in LATERAL, where is allows the sub-select to reference these fields by changing the order of evaluation of the sub-select and doesn't penalize the performance too much.

  SELECT c.id, c.company_name, pb.available, pb.current,
         date_trunc('second', pb.as_of) AS as_of, pbs.*
    FROM companies c, plaid_tokens pt, plaid_balances pb,
         lateral (SELECT SUM(available) AS all_available,
                         SUM(CURRENT) AS all_current
                    FROM plaid_balances WHERE company_id=c.id) pbs
   WHERE pt.id = (c.additional_info->>'primaryPlaidAccount')::uuid
     AND pt.account_id = pb.account_id

This is still quick, and it saves the machinations of having to calculate the sums in a temp table, or write a function to do this... it's just a nice little trick that they put in the language. Very considerate. 🙂

This weekend we had an interesting issue with Clearbit's Logo Search interface - a free service they provide on their Business Information Service system. You can basically hit their endpoint with a query param of the name of a Company, and they will respond with something that looks like:

  {
    name: 'Flexbase',
    domain: 'flexbase.app',
    logo: 'https://logo.clearbit.com/flexbase.app'
  }

which is a nice little thumbnail logos of the Company. It's a very nice tool, and for the most part works flawlessly.

Until it doesn't.

The real issue was the Open Source Node client that was hitting the company's endpoint. It started with:

  topSuggestion(name){
    return new Promise((resolve, reject) => {
      resolve(getTopSuggestion(name));
    });
  }

which called:

  let getTopSuggestion = (query) => {
    return new Promise((resolve, reject) => {
      request(config.api.autocomplete + '?query=' + query, (err, response, body) => {
        resolve(JSON.parse(body)[0]);
      });
    });
  }

Now when everything is working as it should, this code is just fine. But on the weekend, the response from the endpoint in getTopSuggestion() was returning:

  Welcome to the Company API. For docs see https://clearbit.com/docs#company-api

which, of course, isn't JSON, and so the JSON.parse() was throwing an exception. But the function getTopSuggestion() was using the resolve() for the Promise, so it was creating an exception that could not be caught. This was bad news.

Now as it turned out, a coworker found that Clearbit was doing some maintenance, and that might have been the genesis of the issue, but it was made much worse because when we debugged this on our machines - several of us, the issue didn't present itself. Only in production.

Still, it was clear this wasn't the right code to use, and the library was 6 years old without an update, and the code was small. So a coworker suggested we just make the one call ourselves:

    let res = {}
    try {
      // Get the top URL Suggestion for a store name
      const url = new URL(config.api.autocomplete)
      url.searchParams.append('query', name)
      // ...now make the call, and parse the JSON payload
      const payload = await fetch(url).then(response => response.json())
      if (Array.isArray(payload) && payload.length > 0) {
        // ...pick off the top suggestion
        res = payload[0]
      }
    } catch (err) {
      log.error(`[logoFinder] ...message... Name: '${name}', Details: "${err.message}"`)
      return {
        success: false,
        error: errorMessages.badClearBitRequest,
        exception: err,
      }
    }
    return {
      success: true,
      ...res,
    }
  }

where the error message is really up to you, but the point was that this was something that would handle the simple text being returned by the endpoint and throw the exception on the JSON parsing without causing all the trouble of the library we were using.

There were a few things I liked about the new implementation we came up with:

• Explicitly setting the query param on the URL - while it's possible that 90% of all name values would not lead to an issue, it's always nice to be safe and make sure that the proper encodings are done with the query params. It's two lines of code, but it makes sure that it's all handled properly.
• The chaining of fetch() and then() - both fetch() and response.json() are async functions, so you might expect to see two await prependers on the functions, but there's only one. This is a nice feature of the then(), in that it unrolls the async nature of the fetch() so that the async nature of the .json() comes through - returning the value to the caller.

Sure, we still need to get the first element in the Array, but we also test that to make sure it's actually an array, and that there's something to get. It's just a lot more defensive coding than the original client had, and when we did this, we still got the good results on the dev machines, and at the same time, we got proper exception catching on the production instances.

Thankfully, the issues resided about the time we got the fix into the code, tested, and into production, so it wasn't long-lived, but it was a problem for a while, and we were able to recover the errors due to queues and retries, which is another saving grace that I was very thankful for.

Nothing like a little production outage to make the day exciting. 🙂

At The Shop, we're using Google Cloud Run for a containerized API written in Node, and it's a fine solution - really. But one of the issues we have run into is that of environment variables. We have a lot of them. The configuration for dev versus prod versus local development is all being held in environment variables, and the standard way for these to be passed in the cloudbuild.yaml file in the Build step:

steps:
  - name: gcr.io/cloud-builders/docker
    entrypoint: '/bin/bash'
    args:
      - '-c'
      - >-
        docker build --no-cache
        --build-arg BRANCH_NAME=$BRANCH_NAME
        --build-arg THESHOP_ENV=$_THESHOP_ENV
        --build-arg BASE_API_URL=$_BASE_API_URL
        -t $_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA
        . -f Dockerfile
    id: Build

and then in the Dockerfile, you have:

ARG BRANCH_NAME
RUN test -n "$BRANCH_NAME" || (echo 'please pass in --build-arg BRANCH_NAME' && exit 1)
ENV BRANCH_NAME=${BRANCH_NAME}
 
ARG THESHOP_ENV
RUN test -n "$THESHOP_ENV" || (echo 'please pass in --build-arg THESHOP_ENV' && exit 1)
ENV THESHOP_ENV=${THESHOP_ENV}
 
ARG BASE_API_URL
RUN test -n "$BASE_API_URL" || (echo 'please pass in --build-arg BASE_API_URL' && exit 1)
ENV BASE_API_URL=${BASE_API_URL}

While will place them in the environment of the built container. And all this is fine, until you start to hit the limits.

The cloudbuild.yaml command has a limit of 4000 characters, and if you have large, or sufficient number, of environment variables then you can exceed this, and we have. There is also a limit of 20 arguments to the docker build command, so again, we run into trouble if the number of environment variables gets more than that. So what can be done?

Well... since we are using Google Cloud Secrets, we could write something to scan those secrets, and pull them all into the running process, and stuff them into the process.env map for Node. But therein lies another problem: Node is asynchronous, so if we have top-level definitions that use these environment variables, like, say clients to Vendor services, then it's quite possible that they will need those variables before we have had the chance to load them.

So what can we do?

The solution that seems to work is to have a separate app that will be run in the Dockerfile, and will generate a .env file resides only in the container, and is built at the time the container is built, and contains all the environment variables we need. Then, the Node app can just use these with the dotenv library.

To make this file, we have the end of the Dockerfile look like:

# now copy everything over to the container to be made...
COPY . .
# run the node script to generate the .env file
RUN THESHOP_ENV=${THESHOP_ENV} \
  GCP_SECRETS_API_EMAIL=${GCP_SECRETS_API_EMAIL} \
  GCP_SECRETS_API_KEY=${GCP_SECRETS_API_KEY} \
  GCP_BUILD_PROJECT=${GCP_BUILD_PROJECT} \
  npm run create-env
# run the migrations for the database to keep things up to date
RUN npx migrate up --store='@platter/migrate-store'
EXPOSE 8080
CMD [ "node", "-r", "dotenv/config", "./bin/www" ]

So that we give the create-env script the few key environment variables it needs to read the Google Cloud Secrets, and then it generates the file. The create-env script is defined in the package.json as:

{
  "scripts": {
    "create-env": "node -r dotenv/config tools/make-env"
  }
}

and then the script itself is:

const arg = require('arg')
const { execSync } = require('child_process')
const { addSecretsToEnv } = require('../secrets')
const { log } = require('../logging')
 
const _help = `Help on command usage:
  npm run create-env -- --help         - show this message
  npm run create-env -- --file <name>  - where to write the env [.env]
  npm run create-env -- --verbose      - be noisy about it
 
  Nothing is required other than the FLEXBASE_ENV and some GCP env variables
  that can be specified on the command line.`;
 
/*
 * This is the main entry point for the script. We will simply read in all
 * the secrets for the THESHOP_ENV defined environment from the Cloud
 * Secrets, and then write them all to the '.env' file, as the default.
 * This will allow us to set up this environment nicely in a Dockerfile.
 */
(async () => {
  // only do this if we are run directly from 'npm run'...
  if (!module.parent) {
    // let's process the arguments and then do what they are asking
    const args = arg({
      '--help': Boolean,
      '--verbose': Boolean,
      '--file': String,
    })
    // break it into what we need
    const verbose = args['--verbose']
    const where = args['--file'] ?? '.env'
 
    // ... now let's pull in all the appropriate Secrets to the local env...
    log.info(`[makeEnv] loading the Secrets for ${process.env.THESHOP_ENV} into
        this environment...`)
    const resp = await addSecretsToEnv()
    if (verbose) {
      console.log(resp)
    }
    // ...and now we can write them out to a suitable file
    log.info(`[makeEnv] writing the environment to ${where}...`)
    const ans = execSync(`printenv > ${where}`).toString()
    if (verbose) {
      console.log(ans)
    }
    return
  }
})()

The addSecretsToEnv() is where we use the Google Secrets Node Client to read all the Secrets in our account, and one by one, pull them down and put them into process.env. The fact that this runs before the app starts is how we get around the asynchronous nature of Node, and by having it be an .env variable, we can use all the normal tools to read and process it, and we no longer need to worry about the top-level Vendor clients trying to define themselves with environment variables that haven't been defined.

Now if Node had a way to force an async function to finish before moving on, then this wouldn't be necessary, as we'd simply call the addSecretsToEnv() in the Node start-up script, well ahead of the loading of the other files. But alas... that's not how it works.

This has turned out to be a very workable solution, and we get past the limitations of the cloudbuild.yaml file, which is a great relief.

Today I had a reason to look at some Node issues with async processing, and ran across these two little functions that are interesting, but quite deadly. They pause the Node Runtime for the specified number of milliseconds, or seconds, and this is nice - when you have to have a delay, but all Node processing stops. This means all the async calls won't get processed, either.

  function msleep(n) {
    Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, n);
  }
 
  function sleep(n) {
    msleep(n*1000);
  }

Simple. Easy. But not really what I was looking for. 🙂

Yesterday, I was doing a little work, and noticed that I was getting a lot of connection resets on a service that has been flawless for more than 18 months, but to be fair, the load has been rising, and after digging into the cause, it appeared that the issue was overloading the Client with so many requests, it just failed.

Typically, a client will apply back-pressure on the caller to make sure that things don't get to this point, or they will queue the requests in memory so that they will be processed, in turn, as they arrived. I'm not exactly sure what's happening, the developers of the Client are looking at this, but I needed to find something to ease the load, and so I found asyncBatch().

Let's say I had the following code:

  const balances = (await Promise.all(companies
    .map(async c => {
      const bal = await minimumDueForCompany(user, c)
      if (bal?.success && !isNil(bal?.interestDue) && bal.billDate === today) {
        bal.company = c
        return bal
      }
      return undefined
    })))
    .filter(bal => bal !== undefined)

we're running through all the items in the companies array, and for each, we are calling minimumDueForCompany() and then checking a few things, and then filtering on those that we want to see. Simple.

But if we have more than 200 elements in the companies array, and the minimumDueForCompany() employs several database queries, we could get to the point of launching more than a thousand hits at nearly the same time. If this is a background task, this might be able to starve some more important tasks with all the database aork.

A batching solution was needed. And so I went looking.

asyncBatch() follows much the same style as the Promise.all(), it just takes the values as arguments: the array, the function, and the batch size:

  const asyncBatch = require('async-batch').default
 
  const balances = (await asyncBatch(companies,
    async c => {
      const bal = await minimumDueForCompany(user, c)
      if (bal?.success && !isNil(bal?.interestDue) && bal.billDate === today) {
        bal.company = c
        return bal
      }
      return undefined
    }, 2))
    .filter(bal => bal !== undefined)

With a batch size of 2, we'll start simply, and let the background task take a little longer, while preserving the more immediate user-facing calls can have priority access.

Put this in and things are working better. It's not a perfect solution, and we still need to have the Client improved, but it gets around the two problems: Flooding the database when the use-case doesn't require it... and Failures on the Client to handle the flood. We can fine-tune the batch size later.

UPDATE: it turned out that the library launched all the work in an initial Promise.all() so it really wasn't batching the work as I'd expected. So I wrote my own using the chunk library:

  const chunk = require('chunk')
 
  /*
   * We need a function that will batch the equivalent of:
   *
   *   const resp = await Promise.all(arr.map(itm => fcn(itm)))
   *
   * but do it in batches, so that when we get a large workload, we don't
   * overwhelm the system. This is that function. The first argument is the
   * array to process, the second is the async function, that takes one
   * argument, and the last is the batch size that defaults to a reasonable
   * value.
   */
  const asyncBatch = async (arr, fcn, batchSize = 4) => {
    const ans = []
    for (const b of chunk(arr, batchSize)) {
      const blk = await Promise.all(b.map(async itm => await fcn.apply(null, [itm])))
      ans.push(...blk)
    }
    return ans
  }

This works exactly as expected, working on n of the elements at a time, and then moving to the next batch. Much cleaner.

This morning I had some time and wanted to finish up the work of getting my Cloud VM running Ubuntu 22.04 working just fine as a development box - including inbound webhooks from vendors, and calls from apps like HTTPbot on my iPad Pro. The key was that I needed to be able to install and configure nginx to forward all port 443 traffic to port 6543, and that also meant getting the nginx server to be listening on port 443 with a legit certificate.

Turns out, it wasn't as bad as I thought it might be. 🙂

Starting with my Ubuntu 22.04 install, I added the packages I was going to need, based on this blog post on the nginx site.

  $ sudo apt-get -y install --no-install-recommends nginx certbot python3-certbot-nginx

Once these are installed, we could set the server_name in the nginx config:

  $ sudo /etc/nginx/sites-enabled/default

and update the server_name line to be:

  server_name mybox.mydomain.com;

and then we can get the initial certificate from Let's Encrypt and register a new email account with them with:

  $ sudo certbot --nginx -d mybox.mydomain.com -d mydomain.com

and the second -d argument is for an additional domain for the certificate. I didn't need it, so I just had the one -d pair on my certbot command.

After this, we edit the config file again, updating the port 443 section's location specification with:

  location / {
    # forward all HTTPS traffic to port 6543
    proxy_set_header  X-Forward-For $remote_addr;
    proxy_set_header  Host $http_host;
    proxy_pass        "http://127.0.0.1:6543";
  }

and then verify the nginx config with:

  $ sudo nginx -t

and then tell nginx to reload the config with:

  $ sudo nginx -s reload

At this point, the box is answering HTTPS traffic, and forwarding it on to the Node service at port 6543. Excellent. 🙂

In order to refresh the Let's Encrypt Certificate on time, let's add a simple crontab entry:

  $ crontab -e

and then have the entries:

  # run all the commands on Bash not Bourne Shell
  SHELL=/bin/bash
  # send all the mail to my main account
  MAILTO=bob@mydomain.com
 
  # check the Let's Encrypt certificate each dat at noon UTC
  0 12 * * *   sudo /usr/bin/certbot renew --quiet

And that should do it.

I have been doing some manual tasks that I decided today I really wanted to automate. The problem was, it was updating a few rows in a database, and so I didn't really want to expose an endpoint that achieve this, just on the very slim chance that the service might get hacked. I know, we protect these things carefully with Google Cloud Tasks, but I just felt these tasks needed to be done outside the scope of the running service, and it was possible, but tiring, to do it manually. First, with psql, and then with a bash script which took command-line options, and then called psql, and made the updates.

In all, it wasn't a bad idea. And it was working just fine - except for the tired part. 🙂 So while I didn't want to set up a crontab job on my laptop, I could set up one on a Cloud VM that is only accessible by me based on an RSA key. That is secure, and it's out on the network, so I don't have to worry about my internet service going down.

So I needed to have some way to replicate my entire development environment onto my iPad Pro, and then from there use the tools I've pulled together to try and make this happen, and decided to push it that last little bit.

Starting with Working Copy on the iPad, I have a complete, stand-alone Git client that has created local directories on my iPad so that an editor like Textastic on the iPad can access these files, save to them, and Working Copy will detect the changes. Additionally, Textastic can upload the files to a hope using scp, so I can edit and save, upload and test, and then make a PR, and push up to GitHub.

I needed to be able to run Node, and a few more watching commands on the box, so I had to get everything up on the Ubuntu 22.04 box in the Cloud, and then wrangle everything to get it all going. The most troublesome thing is that mosh, the Mobile Shell that I use with Blink on my iPad, doesn't allow for SSH Key Forwarding. It's something about mosh, and I understand the reason, but I also know the mosh folks are working to fix this - but it is annoying not to be able to use git within mosh because the SSH keys aren't carried up in the communication stack.

Still, with conventional ssh, they are carried, and I can use that, as needed. Someday soon, there will be a way to use something like Guardian Angel for the SSH key forwarding in mosh, but for now, this works.

At this point, I can edit locally... push to the cloud, have nodemon restart the server on any change... and hit the service on the standard Node port 6543. But then I needed to get nginx forwarding HTTPS to port 6543, and that's another challenge for another day.

What I can do is to run the bash scripts on the Cloud VM, and then crontab the runs so that I can never again have to worry about being so tired from running these commands at odd hours. 🙂

I was doing a little Node/JS coding on play.js on my iPad Pro this week, and ran into a few issues that I wrote to the developers about. They weren't all that big a deal, save one:

• Logging is iffy - using Node/Express, their default is the debug logging package, and yet you can't see any of the log messages in the console in the app.
• Running nodemon doesn't reload on changes - it would be nice to have a way to auto-reload changes in the files - specifically because the editor is saving them.
• Exceptions aren't logged - if there's an uncaught exception, it's not logged/printed in the console at all. Just silence.

And it's really the last one that's the kicker... no way to see if there has been any exceptions... that's something that would make it very hard to find errors in the code.

They wrote back that there would be updates that were coming soon that would fix most of these, and that I should sign up for the CodeSandbox Beta program and look for the updates. Well... this morning, I saw that they had an update, and changed the name of the app on iPadOS to CodeSandbox. So I fired it up to see how things had changed.

There were lots of changes, but the key problems I was having haven't changed, and they have really moved it away from what I liked about it, and towards another style, with different goals. It's OK... it's their app, but it's not the direction I was hoping they were going.

So... it looks like I'll have to wait a little longer to see what comes up as a development platform for the iPad...

This morning I did a little searching about how to disable any Auto-Complete in Sublime Text, as there are a lot of times that I really don't want to have any autocomplete happening, as it just gets in my way. So I was very happy to find that all I needed to do was to add:

  "auto_complete": false

to the Settings file, and that will turn it off.

Blissful silence. 🙂

Thankfully, Ctrl-Space will bring it back, and that's exactly the thing I was hoping to find. Then it's at my command, as opposed to simply appearing. Once again, less is more... coding is what I do, and I know what to type... so Sublime Text is again my favorite editor. Wonderful. 🙂