What's it all about, Alfie?

I think I'm getting to the limits of this cluster with the topology I'm trying to make - and the source data I have to work with. I've moved up to a parallelization hint on the choke bolt to the point that I believe we're just sloshing from one part of the topology to another and back to the first. There just aren't enough resources to do all this, and at the number of workers I'm using, that's not at all surprising.

Let's look at the message rates again:

We are moving a lot of messages, and the email sends are running, then stopping, and then starting back up again. It's a good test of a significant load.

Now let's look at the capacity of the bolts:

Here we see the transmit jump up after the decoration, and then fall off while the decoration stays high. Then it's back and the decoration goes higher... then off, then on... When we compare this to the last set - from yesterday, we see a more "rate limited" view of the system. We appear to just be robbing Peter to pay Paul - not a great thing.

This just gives us an enormous depth of data about the interactions of these components in a large-scale system. I think it's wonderful and fascinating, and it makes me smile.

UPDATE: I went back to the parallelization hint of 500 for the decorate bolt, and for the corresponding transmitter bolts and the graphs look very nice:

and:

This entry was posted on Friday, September 12th, 2014 at 9:06 am and is filed under Clojure Coding, Cube Life, Open Source Software. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.