Having a Good Laugh Over Silly Policies
Over the last several days we've had significant issues with what appears is non-ASCII data entering the MindAlign servers and bring them all down. Since data like that is very unlikely to be entered by a human, it's thought it had to be a bot. Fair enough.
I've been sent messages by the MindAlign support group that I needed to shut down all the MindAlign bots I had because I was spamming the authentication server with connection requests. I've checked, and we're talking about approximately 10 requests a minute. Hardly excessive, and certainly not the source of continued instability in the system.
Yet I get messages to shut them down.
But to do so is to update/patch several production systems - not the thing you want to do quickly or in haste, and since I've never run these production apps without chatting, it is going to be easy to miss one or two places where the chatting is done. So it's a significant production support risk - all for 10 hits a minute. Hardly seems worth it, so I haven't done anything about it.
Until today.
Today I got a call from one of the MindAlign support folks and he told me what his email had said. I then pointed out the risk to production systems and he actually started to laugh. Clearly, he knew this was a much a fool's folly as I. I even said so. We shared a little giggle and I told him that he had done his job and I'd look into it.
After all, 10 requests a minute is certainly reason enough to cause global chat instability. Hmmm... he didn't think so either, but confessed that he was just 'under orders' to pass the word. I told him his duty was done and he laughed at that as well.
In the end, I put in the changes as best I could, tested them as best I could, and will get the final few restarted this evening/tomorrow morning so as to shut off this spamming. But the problem will remain, won't it? Certainly, because these bots that are so important have been tested and re-tested so many times under so many conditions that they are so unlikely to be the cause of the problem it should be an acceptable risk to leave them on.
It's the bots that are new in the last few days, or those that might chat binary data. Not these that have been working fine for years. It's just silly.
But we're putting production services at risk for this. All for 10 requests a minute. And no one but myself and the guy that called and laughed about it seems to see how incredibly thoughtless this is. But that's how some things go - no one stands up to the Emperor who has no clothes.