{"id":7097,"date":"2014-09-11T16:16:40","date_gmt":"2014-09-11T21:16:40","guid":{"rendered":"http:\/\/bobbeaty.com\/wp\/?p=7097"},"modified":"2014-09-12T08:04:09","modified_gmt":"2014-09-12T13:04:09","slug":"more-topology-balancing","status":"publish","type":"post","link":"https:\/\/bobbeaty.com\/wp\/archives\/7097","title":{"rendered":"More Topology Balancing"},"content":{"rendered":"

\"Unified<\/p>\n

I've spent most of the day trying to get the topology working under load<\/em> from the batch email send process. In truth, I may never<\/strong> get it really perfectly balanced because it's a batch process and not a real-time, on-demand, kind of thing. I'm trying to count shotgun pellets after the gun goes off. It's kinda tough.<\/p>\n

But still I try. And I'm learning a lot about the way this topology and Storm is responding to the load. For instance, if you don't want to buffer tuples in the system - and for the most part, I don't, then use the :local-or-shuffle<\/tt> and then make sure<\/em><\/strong> that your data flow is balanced<\/em> on all bolts before<\/em><\/strong> that step. This will save a lot of lag<\/em> in the throughput as it can hand off one tuple to the next bolt without going through any buffering.<\/p>\n

What I've been playing with lately is significantly<\/em> increasing the size of the decorator<\/tt> bolt parallelization hint and the encoder and transmitter bolts to see if this will make a difference, or if it's just going to shorten the time we're at capacity by moving more messages through the system - but still always being at capacity.<\/p>\n

So I've had good luck, actually, and this is the message rate for a bulk email send (purple) and the corresponding output decorator<\/tt> (golden) and output messages (cyan):<\/p>\n

\"Message<\/p>\n

There's a lot to like about this graph over the older ones - first, the decorate<\/tt> and xmit<\/tt> are virtually identical - i.e. no<\/em><\/strong> buffering. Excellent. Also, the drop-off<\/em> on the output is nearly as good as the ramp-up<\/em>, so that means that we're really doing a pretty decent job of moving the data. I'm not unhappy with this graph at all. But the capacity graph is a different story:<\/p>\n

\"Message<\/p>\n

Here we see that we peaked<\/em> after<\/em><\/strong> the email send block was done, and that's a bit odd, but on the plus side, the encode and emit bolts also rose nicely saying that the decoding is starting to share the load more, and that's a good thing.<\/p>\n

My concern is still<\/em> the capacity number. I suppose I'll run a few more tests with higher numbers still and see if that makes any difference, but I have a feeling it's not going to make any change to the height<\/em> of the capacity surge<\/em> - but it will likely<\/em> lessen the duration<\/em>.<\/p>\n","protected":false},"excerpt":{"rendered":"

I’ve spent most of the day trying to get the topology working under load from the batch email send process. In truth, I may never get it really perfectly balanced because it’s a batch process and not a real-time, on-demand, kind of thing. I’m trying to count shotgun pellets after the gun goes off. It’s […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11,6,4],"tags":[],"class_list":["post-7097","post","type-post","status-publish","format-standard","hentry","category-clojure-coding","category-cube-life","category-open-source-software"],"_links":{"self":[{"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/posts\/7097","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/comments?post=7097"}],"version-history":[{"count":1,"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/posts\/7097\/revisions"}],"predecessor-version":[{"id":7098,"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/posts\/7097\/revisions\/7098"}],"wp:attachment":[{"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/media?parent=7097"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/categories?post=7097"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/tags?post=7097"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}