What's it all about, Alfie?

Today I've been working on some code for The Shop, and one of the things I've come to learn is that for about every feature, or service, of AWS, Google Cloud has a mirror image. It's not a perfect mirror, but it's pretty complete. Cloud Storage vs S3... Tasks vs. SQS... it's all there, and in fact, today, I really saw the beauty of Google Cloud Tasks over AWS SNS/SQS in getting asynchronous processing going smoothly on this project.

The problem is simple - a service like Stripe has webhooks, or callbacks, and we need to accept them, and return as quickly as possible, but we have significant processing we'd like to do on that event. There's just no time or Stripe will think we're down, and that's no good. So we need to make a note of the event, and start a series of other events that will to the more costly work.

This is now a simple design problem. How to partition the follow-on tasks to amke use of an efficient loadbalancer, and at the same time, make sure that everything is done in as atomic way as possible. For this project, it wasn't too hard, and it turned out to actually be quite fun.

The idea with Cloud Tasks is that you essientially give it a payload and a URL, and it will call that URL with that payload, until it gets a successful response (status of 200). It will back-off a bit each time, so if there is a contention issue, it'll automatically handle that, and it won't flood your service, so it's really doing all the hard work... the user just needs to implement the endpoints that are called.

What turned out to be interesting was that the docs for Cloud Tasks didn't say how to set the content-type of the POST. It assumes that the content-type is applciation/octet-stream, which is a fine default, but given the Node library, it's certainly possible to imagine that they could see that the body being passed in was an Object, and then make the content-type applciation/json. But they don't.

Instead, they leave an undocumented feature on the creation of the task:

  // build up the argument for Cloud Task creation
  const task = {
    httpRequest: {
      httpMethod: method || 'POST',
      url,
    },
  }
  // ...add in the body if we have been given it - based on the type
  if (body) {
    if (Buffer.isBuffer(body)) {
      task.httpRequest.body = body.toString('base64')
    } else if (typeof body === 'string') {
      task.httpRequest.body = Buffer.from(body).toString('base64')
    } else if (typeof body === 'object') {
      task.httpRequest.body = Buffer.from(JSON.stringify(body)).toString('base64')
      task.httpRequest.headers = { 'content-type': 'application/json' }
    } else {
      // we don't know how to handle whatever it is they passed us
      log.error(errorMessages.badTaskBodyType)
      return { success: false, error: errorMessages.badTaskBodyType, body }
    }
  }
  // ...add in the delay, in sec, if we have been given it
  if (delaySec) {
    task.scheduleTime = {
      seconds: Number(delaySec) + Date.now() / 1000,
    }
  }

The ability to set the headers for the call is really very nice, as it opens up a lot of functionality if you wanted to add in a Bearer token, for instance. But you'll have to be careful about the time... the same data will be used for retries, so you would have to give it sufficient time on the token to enable it to be used for any retry.

With this, I was able to put together the sequence of Tasks that would quickly dispatch the processing, and return the original webhook back to Stripe. Quite nice to have it all done by the Cloud Tasks... AWS would have required that I process the events off an SQS queue, and while I've done that, it's not as simple as firing off a Task and fogetting about it.

Nice tools. 🙂

I just got off the phone with a very good friend that helped me see something in a way that was always there, but I wasn't extending myself to see it from that point of view - and it really got me to thinking: How wrong was I really?... and the answer was: A lot. 🙂

We all live our little lives, and it's unusual to meet someone that can really see life from a few completely different perspectives. The most common way I know of is profound loss - someone recovering from the loss of a close loved one will have the ability to see life with that person, and without that person, and their perspectives will be entirely different. Grief changes most people. But that's not the only way people can have different perspectives.

I was talking to my friend, and mentioning that I was going through a tough time with some folks, and she suggested that I had it all wrong. And proceeded to tell me how wrong I was.

She pointed out that life really is what we make of it, and that I could choose to see things as how they effected me, or I could see it from a different perspective, and see that there were other ways of handling the exact same thing, and in a different way, not make it an us-vs-them situation.

I'm thankful for my friend, because it was what I needed to hear - even if I didn't want to. I need to change how I approach things... life doesn't have to be kind... there's no rule about that. But we can choose to insert kindness in what we do, and not let the kill-or-be-killed be the way we live our lives.

Sometimes I am really wrong. By a lot.

And I'm glad my friend was there to help me see it.

I'm looking at the stories this morning about the actions in Washington by the pro-Trump mob that breeched the Capital for the first time since The War of 1812... and I feel certain the outgoing administration doesn't care - we've seen four years of policies, speeches, actions, and inactions, that show all that are paying attention that a day like yesterday was virtually inevitable.

Sure, it could have been 4 years from now... but it was now, and we have seen it. And this morning, it's now clear that those that were objecting to the Electoral College vote count have changed their minds, and have decided that they would object no more. Enough was enough - finally... for them. And we have some sort of closure for this election.

We also have a 50-50 Senate, and I have to smile at the idea that the Founders expected this might happen, so they planned for it, and work will get done, compromises will be made, people will be unhappy, and some will be happy. Life will go on.

I wonder what will come in the days and weeks ahead... will we really have seen enough? Will we really change how we treat the less-well-off among us? I hope so. I really do.

A close friend, and his family, all have come down with COVID-19 as a result of some Christmas family time. I am so sorry for them. I know the intentions were good - and the precautions the family took were meant to keep everyone safe, but this is a nasty flu, and it's indiscriminate to whom it will hit. Closed spaces for several hours - like a Thanksgiving or Christmas celebration is just what it likes.

I think my household got it back in February 2020, after some trips to Seattle and San Francisco I had to take for work. I was in a packed airplane, and those cities at that time were hot spots. I got sick, for a few days, and then got better. My daughter was sick for a few weeks, and got better. While we haven't been tested, we survived without major complications.

I hope my friend's family does as well. This is a scary thing to have happen...

Another trip around the sun... I hope for just a nice, quiet day, and then Pizza tonight for dinner. It's my special day, so why not treat myself to pizza? 🙂

It's been a heck of a year, with so many things happening like never before... the pandemic, the election, the working at home, the rationing of paper towels... it's been a year of exceptional events. And not just for me... for everyone. But we keep going.

So I hope that everyone has a little breather today... a little rest... so that we can get back at it on Monday.

A few days ago, I sent an email to the iTerm2 developer, and asked him the following question:

...and maybe this is a silly request, but I would really enjoy the option to put the Emoji Picker on the TouchBar of my MacBook Pro while in iTerm2… I know it’s not a Big Deal - so dragging it off/on in the customization makes sense, but there are a lot of times in my Git commit messages that I’d like to be able to toss in an emoji

and this morning I got a (surprise) response:

Thanks for pointing this out. There’s no good reason why it shouldn’t be allowed. Commit 1ae34d90f adds it. You can test in the next nightly build, due out in about an hour. In order to avoid breaking existing setups, it’s not in the default setup. You need to choose View > Customize Touch Bar to add it.

which is perfect for what I was hoping to have.

One of the best uses I've found for the TouchBar on the MacBook Pro is the Emoji Picker - as it's perfect for Instant Messaging, and Twitterrific, and at The Shop it's a big thing to have a nice, representative emoji as the first character of a pull request title. This is OK with LaunchBar, but it's not as convenient as the TouchBar Emoji Picker, and that's really what I was hoping to use it for. But until recently, iTerm2 just didn't allow it in the configuration of the TouchBar.

I am as pleased as I can be. Sounds silly, but it's nice to see that your thoughts aren't completely left-field to others. 🙂

UPDATE: the v3.4.4beta2 release has the Emoji Picker. I'm just smiling. 🙂

At The Shop, we are using a completely different platform than I've used in the past - Node/JS and Express as well as Platter, for a back-end database. It's been a steep learning curve for me, but I have to say today was a nice day where I really started to feel like I was getting the handle on the tools. What has really exciting to me with Express is the ease with which I can build middleware to insert into the calling stack.

For ring middleware, in Clojure, it's not horrible, but it's not trivial to understand the calling order, and the passing of the handler to all the middleware. In Express, it's far simpler - you simply have a function that takes the request, the response, and the next in the line of the calling stack, and that's it. You can augment the request, and that's basically what a lot of middleware is about - adding authentication tokens, looking up permissions, etc. It's all adding to the request to be used in the simpler endpoint routes.

When working with Passport for the authentication framework, it's great that it fits in with Express so well, but one of the issues that I ran into today was that the middleware added to the top-level Express app would be executed before the Passport authentication middleware that was in place on each individual endpoint. It makes sense, not all endpoints need authentication, so adding that with Passport would naturally be done after the top-level middleware. But that makes some of the middleware I'd written unfunctional.

The Passport authentication scheme can be set up to easily add the user object to the Express request, and then for all endpoints, it's "Just There". I had expected to add middleware that would take that user and use it to look up other attributes and data to add to the request as well. But if the middleware I'd written was placed at the top-level, then it wouldn't have the user object on the request, and so it'd never work.

The solution was so elegant, I'm convinced that this had to be anticipated by the Express developers. 🙂 Each of the routes wired into the Express app takes a path and a router:

  app.use('/login', loginRouter)
  app.use('/company', companyRouter)

and when you add in the Passport support for JWT authentication with a Bearer token, you get something like:

  app.use('/login', loginRouter)
  app.use('/company', passport.authenticate('jwt', { session: false }), companyRouter)

where the /login endpoint is not protected by the JWT, and the /company endpoint is. This seemed like a very late stage to put in the Passport middleware, but as it turns out, Express can handle an array, or a list of middleware in the use() function. So we can say:

  const authStack = [
    passport.authenticate('jwt', { session: false }),
    tenantMiddleware,
    accountMiddleware,
  ]
  app.use('/login', loginRouter)
  app.use('/company', authStack, companyRouter)

where the authStack is the additional middleware for the individual routes, and it's handled in the order it appears in the array.

And it works like a champ. Just amazing, that we can create different stacks of middleware and then as long as we layer them properly, we can set up an amazing diversity of middleware. For this, it's great that we can group the authentication-focused middleware into an array, and then easily drop that on the endpoints that need it.

Very slick. 🙂

Yesterday started Advent of Code 2020, and it looks to be a fun theme - Vacation - and while I'm sure it's been planned well in advance of the current pandemic, I'd like to think that the folks in charge are thinking of all of us, and making it a little more enjoyable this year... because we've been cooped up all year.

The puzzels start easy, and then build through the month, and that's just as it should be - an easy way to introduce folks to coding with fun problems as opposed to so many made-up teaching problems they run into. They always make me smile.

Is that just what we all need now? A smile? 🙂

Today is the first day at the New Shop, and I'm a bit nervous that it's all going to be Node and React - they are tools I haven't done a lot of work in, but thanks to some help from a good friend, I feel I have a good start, and the Pragmatic Programmer's Simplifying JavaScript really is a good book to get up-to-speed on the latest changes to the language.

There's going to be a lot of learning, and it's going to be a little stressful at times, as I try to come up to speed as quickly as possible... but it's working with some very fine people, and this is the path I'm on... I need to learn all that I can - regardless of the circumstances.

I'm reminded of the chant: The King is dead. Long live the King! Life is a lot like that, it seems... and off we go! 🙂

Today I spent a little time with a good friend helping me get going on a good, versioned Node environment - a lot like RVM for Ruby - but for Node. I wanted to do this because it looks like I might be doing some work for a Node-based company where the development is all based in Node, and I wanted to make sure I got it all set up right.

I just finished reading a nice book on ES5, ES6, Promises, async and await, and all the new features of JavaScript called Simplifying JavaScript from the Pragmatic Programmers. They are having a Thanksgiving Sale, and it seemed like a great time to pick up a book that I'd probably like on the subject. I did.

It's been a long time since I spent any real time on JavaScript, and if I'm going to be taking a bit out of this project, I wanted to make sure I had come up to speed on JavaScript, and Node as well. The book was laid out well, with all the ideas based on a decent understanding of JavaScript, but not the latest additions. It really read well to me, and I was able to finish it in two days.

So, here's what I needed to do on my 16" MacBook Pro to get things up and running... 🙂

Start off by installing nodenv from Homebrew. This is the equivalent of rvm, and will manage the Node versions nicely for me.

  $ brew install nodenv

I then needed to add in the environmental set-up in my ~/.zlogin file by adding:

  # now do the nodenv stuff
  eval "$(nodenv init -)"

right after I set up my PATH and RVM environment things. It's very similar to the RVM approach, with directory-level controls, as well as system-wide defaults.

At that point, I can source my ~/.zlogin and then I'm ready to go. Next, is to install a good, long-term stable (LTS) version of Node:

  $ nodenv install 14.15.1
  $ nodenv global 14.15.1

where the second command sets that version as the global default for new projects, etc. You can always check the versions with:

  $ nodenv versions
  * 14.15.1 (set by /Users/drbob/.nodenv/version)

Next was to install a few global tools with npm that I'd need:

  $ npm install -g express-generator
  $ npm install -g nodemon
  $ nodenv rehash

where the first is the basic RESTful pattern for a service, and the latter is a way to run a Node app while monitoring the filesystem for changes to the files, and reloading them automatically. This will no-doubt prove to be exceptionally handy. The rehash command is something he's found to be nexessary when installing new global tools, as they don't seem to properly get picked up in the PATH without it. Fair enough.

At this point, we can make a new project, just to play with the new features in the book. Start by making a directory to put all this, and then use the express-generator to make the skeleton we need:

  $ mkdir playground
  $ cd playground
  $ express api --no-view

and now, in the api/ directory we have what we need to get started. Simply have npm pull everything down:

  $ cd api
  $ npm install

and we are ready to go.

There is an index.html file in the public/ directory, and we can use that... and running the Node server is as simple as:

  $ node bin/www
  ... this is the log output... 

or if we want to use the file-watching version, we can say:

  $ nodemon bin/www
  ... this is the log output... 

The port is set in the bin/www script, but I'm guessing the default is port 3000, so if you go to localhost:3000 you'll see the GET calls, and the page. Very slick... very easy. 🙂

Once I get this into a git repo, or start working on a real project/git repo, I'll see if I can get it loaded up on my iPad Pro using play.js - as it appears to be able to run all this, and have a web page to hit it... so that would be very interesting to work with - given the power of the iPad Pro, and the perfect size.

UPDATE: Indeed... once I pused the code to GitHub, and then went into play.js on my iPad Pro, I could Create a new project, from a Git Clone, and putting in the repo location, and the SSH Keys, etc. it all came down. Then it was just resolving the dependencies with the UI, and then setting the "start" command to be the npm command in the package.json, and then it ran.

Open up the play.js web browser, and it's there. On port 3000, just like it's supposed to be. And editing the file, refreshing the page - it's there. No saving, it's just there. Amazing. This is something I could get used to.