The All-Time Slowest Messaging Middleware
At the Shop, I've been part of an integration team that's trying to get this large system into place and running. There's a ton of connections to our other systems, and it's a big job, but the part that I'm working with most is the price injection - because I'm the guy that wrote the price feed, after all. Makes sense.
Part of this product is a messaging system - which makes perfect sense to me - build a strong backbone and then hang things off it and let the middleware take care of the delivery and communication. Good plan. Horrible execution. I mean the worst I've seen in ages. And I've got data to back it up.
The entire messaging system is single-threaded. In this day and age of multi-core, multi-CPU machines, this alone makes this message system a problem. The entire thing is a bottle-neck, as opposed to enabling multiple messages to go between multiple senders and receivers, this guy acts like he's the 93 year old mail clerk in the post office. He's only going to do one thing at a time, and it's a government job, so he can't be replaced.
Case in point, I sent a message - a simple message to be sure, probably less than 1kB, and it took 23 minutes to arrive. Now this is going from one box in the server room to another box in the server room. There's no reason it shouldn't be measured in msec. - let alone 20+ minutes. While I understand that the speed of delivery is based, in part, on the messaging database and the other traffic it has to deliver, there's still no excuse to deliver to a customer a messaging solution that's 20 minutes to deliver a 1kB message.
The funny thing is, I could read all the relevant data from the database (about 3000 records) in about 15 seconds, so while it's clear that something is holding it up, it's not the underlying database that's the problem.
I've had problems with this product before, and as such, I'm not about to name names, but if you're doing anything with a custom-built vendor-suppled middleware, it pays to look under the hood and make sure you can plug in something that's got known good performance like Tibco or IBM MQ. Make sure you're not stuck with this company's horrible middleware that's causing you problems for years to come.
[5/1/08] UPDATE: after a restart of the subscription connections, things seems to be working better today. I'm not sure what the problem yesterday might have been, but it's working better today. I'm going to watch it for several days, but now it's delivering the messages in under 7 sec., and while that's not the fastest message system in the world, it's certainly acceptable for this project. I hope it keeps up.