Today I've been working on some code for The Shop, and one of the things I've come to learn is that for about every feature, or service, of AWS, Google Cloud has a mirror image. It's not a perfect mirror, but it's pretty complete. Cloud Storage vs S3... Tasks vs. SQS... it's all there, and in fact, today, I really saw the beauty of Google Cloud Tasks over AWS SNS/SQS in getting asynchronous processing going smoothly on this project.
The problem is simple - a service like Stripe has webhooks, or callbacks, and we need to accept them, and return as quickly as possible, but we have significant processing we'd like to do on that event. There's just no time or Stripe will think we're down, and that's no good. So we need to make a note of the event, and start a series of other events that will to the more costly work.
This is now a simple design problem. How to partition the follow-on tasks to amke use of an efficient loadbalancer, and at the same time, make sure that everything is done in as atomic way as possible. For this project, it wasn't too hard, and it turned out to actually be quite fun.
The idea with Cloud Tasks is that you essientially give it a payload and a URL, and it will call that URL with that payload, until it gets a successful response (status of 200). It will back-off a bit each time, so if there is a contention issue, it'll automatically handle that, and it won't flood your service, so it's really doing all the hard work... the user just needs to implement the endpoints that are called.
What turned out to be interesting was that the docs for Cloud Tasks didn't say how to set the content-type of the POST. It assumes that the content-type is applciation/octet-stream, which is a fine default, but given the Node library, it's certainly possible to imagine that they could see that the body being passed in was an Object, and then make the content-type applciation/json. But they don't.
Instead, they leave an undocumented feature on the creation of the task:
// build up the argument for Cloud Task creation
const task = {
httpRequest: {
httpMethod: method || 'POST',
url,
},
}
// ...add in the body if we have been given it - based on the type
if (body) {
if (Buffer.isBuffer(body)) {
task.httpRequest.body = body.toString('base64')
} else if (typeof body === 'string') {
task.httpRequest.body = Buffer.from(body).toString('base64')
} else if (typeof body === 'object') {
task.httpRequest.body = Buffer.from(JSON.stringify(body)).toString('base64')
task.httpRequest.headers = { 'content-type': 'application/json' }
} else {
// we don't know how to handle whatever it is they passed us
log.error(errorMessages.badTaskBodyType)
return { success: false, error: errorMessages.badTaskBodyType, body }
}
}
// ...add in the delay, in sec, if we have been given it
if (delaySec) {
task.scheduleTime = {
seconds: Number(delaySec) + Date.now() / 1000,
}
}
The ability to set the headers for the call is really very nice, as it opens up a lot of functionality if you wanted to add in a Bearer token, for instance. But you'll have to be careful about the time... the same data will be used for retries, so you would have to give it sufficient time on the token to enable it to be used for any retry.
With this, I was able to put together the sequence of Tasks that would quickly dispatch the processing, and return the original webhook back to Stripe. Quite nice to have it all done by the Cloud Tasks... AWS would have required that I process the events off an SQS queue, and while I've done that, it's not as simple as firing off a Task and fogetting about it.
Nice tools. 🙂
