{"id":7250,"date":"2014-10-31T14:19:16","date_gmt":"2014-10-31T19:19:16","guid":{"rendered":"http:\/\/bobbeaty.com\/wp\/?p=7250"},"modified":"2014-11-05T07:50:48","modified_gmt":"2014-11-05T13:50:48","slug":"adding-optional-json-encoding-to-bolt","status":"publish","type":"post","link":"https:\/\/bobbeaty.com\/wp\/archives\/7250","title":{"rendered":"Adding Optional JSON Encoding to Bolt"},"content":{"rendered":"

\"Storm<\/p>\n

I wanted to see what the effect would be to the topology to remove the communication overhead between the JSON encoder bolt and the Kafka sender bolt. Even with the :local-or-shuffle<\/tt> directive, I was worried that the objects created were causing garbage collection, or some kind of overhead, so I modified the defbolt<\/tt> I was using to send kafka messages, to take an optional parameter to indicate that it should use cheshire's JSON encoding on the message prior<\/em> to sending it out.<\/p>\n

That way, if it's there, then it's encoded, and if it's not, then the overhead of this is very minor:<\/p>\n

  (<\/span>defbolt mk-<\/span>sender [<\/span>"message"<\/span>]<\/span> {<\/span>:params<\/span> [<\/span>cfg topic &<\/span> [<\/span>fmt]<\/span>]<\/span> :prepare<\/span> true}<\/span>\n    [<\/span>conf context collector]<\/span>\n    (<\/span>let<\/span> [<\/span>dest (<\/span>ep\/<\/span>config cfg)<\/span>\n          top (<\/span>get-<\/span>topic dest topic)<\/span>\n          zk (<\/span>or<\/span> (<\/span>:zkStoreConnect<\/span> dest)<\/span> (<\/span>rand-<\/span>nth (<\/span>:zookeepers<\/span> dest)<\/span>)<\/span>)<\/span>\n          p (<\/span>kp\/<\/span>producer {<\/span>(<\/span>kafka-<\/span>tag zk)<\/span> zk}<\/span>)<\/span>]<\/span>\n      (<\/span>bolt\n        (<\/span>execute [<\/span>tuple]<\/span>\n          (<\/span>doseq<\/span> [<\/span>raw (<\/span>.<\/span>getValues tuple)<\/span>\n                  :when<\/span> raw\n                  :let<\/span> [<\/span>msg (<\/span>case fmt\n                              :json<\/span> (<\/span>nil-<\/span>if-<\/span>empty (<\/span>json\/<\/span>generate-<\/span>string raw)<\/span>)<\/span>\n                              raw)<\/span>]<\/span>\n                  :when<\/span> msg]<\/span>\n            (<\/span>kp\/<\/span>send-<\/span>messages p top [<\/span>(<\/span>kp\/<\/span>message (<\/span>.<\/span>getBytes msg)<\/span>)<\/span>]<\/span>)<\/span>\n            (<\/span>emit-<\/span>bolt!<\/span> collector [<\/span>msg]<\/span> :anchor<\/span> tuple)<\/span>)<\/span>\n          (<\/span>ack!<\/span> collector tuple)<\/span>)<\/span>)<\/span>)<\/span>)<\/span><\/pre>\n
It works like a charm, and while the overhead wasn't even measurable, I left it in just in case there comes a time I want to simplify the topology by having a kayak bolt that also encodes to JSON.<\/p>\n","protected":false},"excerpt":{"rendered":"
I wanted to see what the effect would be to the topology to remove the communication overhead between the JSON encoder bolt and the Kafka sender bolt. Even with the :local-or-shuffle directive, I was worried that the objects created were causing garbage collection, or some kind of overhead, so I modified the defbolt I was […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11,6,4],"tags":[],"class_list":["post-7250","post","type-post","status-publish","format-standard","hentry","category-clojure-coding","category-cube-life","category-open-source-software"],"_links":{"self":[{"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/posts\/7250","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/comments?post=7250"}],"version-history":[{"count":1,"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/posts\/7250\/revisions"}],"predecessor-version":[{"id":7251,"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/posts\/7250\/revisions\/7251"}],"wp:attachment":[{"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/media?parent=7250"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/categories?post=7250"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bobbeaty.com\/wp\/wp-json\/wp\/v2\/tags?post=7250"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}