What Would I Build?
I've been playing around with Storm for a while now, and while I don't think there are all that many folks in the world that are expert at it, I'm certainly an advanced novice, and that's good enough for the amount of time I've put into it. I've learned a lot about how they have tired to solve the high-performance computing platform in clojure and on the JVM, and I've come away with an affirmation of the feelings I had when I was interviewed for this job, and discussing functional languages: Garbage Collection is the death of all functional languages, and certainly Storm.
I like the simplicity of functional languages with a good library of functions. Face it, Java took off over C++ because C++ was the base language, and Java had the rich object set that everyone built on. It made a huge difference in how fast people could build things. So if you want a functional language to have a lot of traction fast, you need to make sure that you don't send people to re-invent the wheel to do the most basic tasks.
But the real killer is Garbage Collection. I'm not a fan, and the reason is simple - If I'm trying to do some performant coding, I want to control when that happens, and under what conditions. It's nice for novices to be able to forget about this and still write stable code, but when you want to move 1,000,000 msgs/sec, you can't do it without pools, lockless data structures, mutability, and solid resource control. None of which I get in the JVM - or anything based on it.
So what's a coder to do? Answer: Write another.
There used to be Xgrid from Apple, but they dropped that. They didn't see that it was in their best interests to write something that targets their machines as nodes in a compute cluster, and they aren't about to write something where you can use cheap linux boxes and cut them out altogether. Sadly, this is a company, and they want to make money.
But what if we made a library that used something like ZeroMQ for messaging, and then we used something like C++ for the linux side, and Obj-C++ for the Mac side and made all the tools work like they do for Storm - but instead of using clojure and the JVM, and a ton of tools on the server-side to handle all the coordination and messaging, let's use something that's far more coupled with the toolset we're working with.
First, no Thrift. It's bulky, expensive, and it's being used as a simple remote procedure call. There are a lot better alternatives out there when you're using a single language. Stick with a recent version of ZeroMQ and decent bindings - like their C++ ones. Start small and build it up. Make a decent console - Storm is nice here, but there's a lot more that could be done, and the data in the Storm UI is not really easily discernible. Make it clearer.
Maybe I'll get into this... it would certainly keep me off the streets.