Building the Next Broker
I've been talking about The Broker for a while, and it's been through at least five iterations/revisions/re-writes that I can remember, but we're in our sixth now. What I've seen, as I hit it the hardest of almost anyone, is that the java version built on Netty is just too complex. The design is reasonable, but even that's complex because of the direct dial interface we put on The Broker to remove the massive load some clients can generate. It just wasn't working. So we tried a new approach.
I mentioned distributed erlang, and asked why we didn't build the original erlang Broker using that? It seemed to handle all the registration/location service issues. It also handled the problem of a centralized service in general. When one client makes a request to one service, it's only those two nodes that will see the load - not anyone else. It's a really nice idea - extremely simple, and therefore, probably the best.
So my boss started on this next version. He and I worked on it today and we're down to just a few things that need to be finished: the service registry and the service load (for load balancing). I'll do those tomorrow and we'll be feature complete.
The beauty of this design is that we can add as many native services to the Broker as erlang modules in the distribution package. Each one is very simple, and is loaded on demand from the elang runtime. This places us in the wonderful position of being able to add in the login service, the configuration service, and any other service that we want in the native erlang. Because this new design places a Broker node on each box running a service, this means that for the most used services, like login and configuration, we won't even go off-box to get the response.
This is going to have a huge impact to the reliability of the Broker. It's going to be able to be stopped/started on each node, as needed, it'll connect into the grid and share it's state, and will be able to handle all the routine services on it's own. This is a major win for me. I think it's going to make it great.