Designing for Continuous Operation
Today I had some problems with the messaging software we use at The Shop - 29West. To be fair, the problem wasn't completely 29West's, but it was due to the architecture of 29West. Even in pub/sub mode, 29West (non-multicast) is still a point-to-point protocol. This is a feature in their design. I can see it - take out the middle-man and centralized bottle-neck, and you can increase performance. Makes sense... I've used point-to-point socket protocols a lot and there's a ton of good in them.
But when it comes to a service, that's not a really great model to use. Reason being: if it goes down for any reason, then you have clients that are unable to connect to this "pub/sub" system, and that doesn't make a lot of logical sense.
I mean, a subscriber should be able to subscribe unless the complete infrastructure is down - not just the publisher. It's a matter of opinion, I'll concede, but I think it's an opinion that should be shared by a vast majority of the systems developers in the world. The entire point of pub/sub is to separate the 'pub' from the 'sub'. Having it be point-to-point breaks all that to pieces.
Which brings me to the point that seems to be clearly lacking in The Shop - the thought for continuous operation seems completely absent. Everyone plans on nightly shutdowns, and as such, never puts even the slightest thought into making their system work all week long. "There's a nightly restart, so I will to." - that's just lazy thinking and lazy building.
I'm not saying there aren't reasons for some systems to have nightly outages. What I'm saying is that if there's a reason for your system to have one, then go ahead and have one. But if you don't need it, then don't take it. If you do this, and the other systems that do need it figure out a way around their limitations, then you have a more robust environment as nothing needs to have a nightly restart.
So stop accepting other's limitations as yours - but better systems than that.