Expanded Latency Detection for Graphing
We have been detecting and calculating the latency of the main data stream we're creating, but we haven't really made it generally available to graphing via the tools built inside The Shop, and today was the time to make that change.
The extension was really pretty simple, we had everything we needed in-place, we just needed to add some data to redis, and then have a process that would read this data out of redis, clear the data, and the counting could continue. Pretty simple.
We picked the redis structure:
dd|current-latency|lt10 => long (count) dd|current-latency|lt60 => long (count) dd|current-latency|lt180 => long (count) dd|current-latency|lt600 => long (count) dd|current-latency|gt600 => long (count)
where the keys are for less than 10 sec, less than 60 sec, less than 180 sec, less than 600 sec, and more than 600 sec. These are not meant to be particularly fine grained, but they are more than adequate to see if there is a trend, and at less than 10 sec, we're more than fine.
The code to read/reset the values is amazingly simple:
(defn pull-current "Function to extract and reset the 'current-latency' counts from redis and return them to the caller. This is the way to monitor the latency as it develops so that differences don't have to be taken. Simply call this function on a regular interval, and graph - it's pretty simple." [] (let [lk (keys @_ls_counts) rlk (map #(str *master* "|current-latency|" (name %)) lk)] (into {} (map vector lk (wcar :detector (mapv #(or (car/getset % 0) 0) rlk))))))
I do love the way that carmine makes working with redis in clojure a breeze.
We then put this into one of the monitord scripts, and we had everything we needed. Nice.