What's it all about, Alfie?

Sure, I'd like to learn to be a really good designer, but I know there might not be enough time in my life to get good enough at it to justify the investment. But there are also a lot of things I'd like to get up to speed on - closure's core.async, Swift, and a lot of things like that. I would love to be able to learn, and then apply these tools. It sounds like a lot of fun.

But there's a different class of still - not a skill like design, not another tool, but the understanding of the interdependent parts of a complex system. For example, I'm doing more topology tests this morning and I'm seeing behavior in the relationships between the bolts that I simply would never have guessed. It's stunning, and it makes me smile.

There's no class for this. There's no Tutorial for this. This is you, the machine, and time. This is testing how well you understand this deterministic machine you've built, and looking at the input, can you understand why the behavior is what it is?

It's almost detective work.

What a blast.

I've spent most of the day trying to get the topology working under load from the batch email send process. In truth, I may never get it really perfectly balanced because it's a batch process and not a real-time, on-demand, kind of thing. I'm trying to count shotgun pellets after the gun goes off. It's kinda tough.

But still I try. And I'm learning a lot about the way this topology and Storm is responding to the load. For instance, if you don't want to buffer tuples in the system - and for the most part, I don't, then use the :local-or-shuffle and then make sure that your data flow is balanced on all bolts before that step. This will save a lot of lag in the throughput as it can hand off one tuple to the next bolt without going through any buffering.

What I've been playing with lately is significantly increasing the size of the decorator bolt parallelization hint and the encoder and transmitter bolts to see if this will make a difference, or if it's just going to shorten the time we're at capacity by moving more messages through the system - but still always being at capacity.

So I've had good luck, actually, and this is the message rate for a bulk email send (purple) and the corresponding output decorator (golden) and output messages (cyan):

There's a lot to like about this graph over the older ones - first, the decorate and xmit are virtually identical - i.e. no buffering. Excellent. Also, the drop-off on the output is nearly as good as the ramp-up, so that means that we're really doing a pretty decent job of moving the data. I'm not unhappy with this graph at all. But the capacity graph is a different story:

Here we see that we peaked after the email send block was done, and that's a bit odd, but on the plus side, the encode and emit bolts also rose nicely saying that the decoding is starting to share the load more, and that's a good thing.

My concern is still the capacity number. I suppose I'll run a few more tests with higher numbers still and see if that makes any difference, but I have a feeling it's not going to make any change to the height of the capacity surge - but it will likely lessen the duration.

Today has been a easy-going day of performance testing and topology tuning with the new data source I'm trying to integrate - email sends. This is coming in from a current batch process that's delivering batches of emails to different locations all around the globe. The trick with this is that testing it is kinda tough because if they aren't running a batch, you have no data to test with. Combine that with the fact that a lot of these tests with Storm need to run for a while to get to the steady-state condition, and it makes for a lot of staring at graphs.

I like to make a table of all experiments and their results, and for today's experiments this is what I ended up with:

While it's still developing, it's clear to me that we started out with more resources on the email send bolts than we needed, and re-writing the lookups was an important step.

UPDATE: with the increase of the workers to 30, we finally have something that handles the load at least as well as production. That's good enough for today.

I've been faced with a significant problem - cache data for 110,000,000 users in a way that's both fast, and efficient, so that we can pull data out of it at a rate of 50k to 100k times a second. Redis, being single-threaded, is great at fast hits for reasonable data sets, but storing more than 100 million of anything, and accessing it by hundreds of threads is going to blow out any one redis server - so you have to shard.

But how to shard efficiently?

Turns out, Java's MD5 is amazingly efficient. I wrote the following hash function that takes any string and hashes it into one of 8 buckets - I'm planning on having 8 redis servers on one physical box:

  (defn hash8
    "Function to generate a 3-bit hash on the provided string by using the MD5
    as an intermediate vechile, and then taking the last byte and 'mod 8' it.
    This is using the Java MessageDigest class to create the hash, so it's
    only as good as that class/function - but from our tests, it's very efficient."
    [s]
    (if (string? s)
      (let [ba (.digest (doto (java.security.MessageDigest/getInstance "MD5")
                              (.reset)
                              (.update (.getBytes s))))]
        (mod (aget ba (dec (alength ba))) 8))))

and I compared it to what I thought was going to be the much faster way: Simply adding up all the ASCII byte values and taking the last 3 bits:

  (defn smash
    "Function to generate a 3-bit hash on the provided string by summing
    the bytes of the string, and then taking the 'mod 8' of it."
    [s]
    (if (string? s)
      (let [ba (.getBytes s)]
        (mod (areduce ba i ret (long 0)
               (+ ret (aget ba i)))
             8))))

And to my surprise, when I ran a sequence of strings through both, the MD5 version far outperformed the byte array version, and I'm now convinced that it's because of the getInstance() call - Java is holding onto a generator, and serving it up to the caller as needed. Plus, they have to have really optimized that code to beat a simple adder.

In the end, I put this on the front-end of the redis calls with:

  (defn user-shard
    "Function to return the right function that will be used with 'wcar' to get
    the provided key from redis. This is specific just to the user data."
    [k]
    (case (hash8 k)
      0 :user-ids-1
      1 :user-ids-2
      2 :user-ids-3
      3 :user-ids-4
      4 :user-ids-5
      5 :user-ids-6
      6 :user-ids-7
      7 :user-ids-8))

and then it's used, with Carmine, as:

  (wcar (user-shard k) (car/get k))

When I look at the CPU and memory usage on the box, I'm seeing wonderfully balanced CPU usage - meaning that the sharing is very nicely distributed, and the memory usage for redis is very reasonable for the data set.

Great win!

This morning I'm once again trying to figure out a problem I've been having with the workers in the Carmine message queue implementation. Basically, the thread that starts the workers is doing just fine, but the workers themselves, are just stopping. I had no idea what to do about it - so I wrote to the author asking him about this.

He responded with the :monitor option to the worker function. I didn't see it in reading the code, but yes, there's a function that gets called when the queue is cycled, so I added a simple function there to reset an atom, and then in the thread that starts these workers, I inc that atom, and if it exceeds 50 sec of not being reset, then I know that it's taken more than 50 sec for the queue to cycle, and I try to stop/start the worker.

The basic monitor and it's worker look something like this:

  save-mon (fn [{:keys [mid-circle-size ndry-runs poll-reply]}]
             (debug "persistence worker heartbeat (iteration)...")
             (reset! _save_loops 0))
  saver (mq/worker (epr/connection :queue) *dump*
          {:handler (fn [{:keys [message attempt]}]
                      (save-it! cfg message)
                      {:status :success})
           :monitor save-mon
           :nthreads 1})

and then in the main body of the function we have something that checks to see if the _save_loops atom has been reset recently enough:

  (let [sc (swap! _save_loops inc)]
    (when (< 50 sc)
      (warnf "Persistence worker hasn't cycled for %s sec -- Restarting!" sc)
      (reset! _save_loops 0)
      (infof "Stopping the persistence worker... [%s]"
             (if (mq/stop saver) "ok" "FAIL"))
      (infof "Starting the persistence worker... [%s]"
             (if (mq/start saver) "ok" "FAIL"))))

This all took a while to figure out, but after a time, I got it working, and it appeared to be working. But the stopping and starting just weren't doing the right things. Add to this, the background that in the other data center, I had multiple installations of this where the workers weren't in trouble at all.

I'm starting to think it's the redis server. That will likely be the next thing I do - restart the redis server and hope that clears any odd state that might be there.

I do wish this would settle down.

UPDATE: I emailed Peter, the author, and he asked me to check the logs - and in this case the timbre logs - his logging package. These go to standard out, and I had forgotten about the. Sure enough, there was useful data there, and all the stops and starts were logged as well. At this point, I believe it's something in redis, and it has been successfully cleared out. But if it happens again, I'll dump the redis database and start fresh - it's just the queue data.

This morning I wanted to be able to add the Storm topology bolt capacity values to the in-house monitoring and graphing tools that The Shop uses. The reason for this is that I'm constantly checking the Storm UI to see what the capacity values are for the bolts on my critical topology, and it'd be so much nicer to be able to see them in a simple graph on the display that I'm already looking at for disk space, CPU usage, and also the higher-level metrics like the messages per second emitted from each bolt.

The latter is something I figured out by digging into the Nimbus JavaDocs, and it was still useful in this bit of detective work. But the biggie was the code that the Storm UI uses to generate it's response. That was a little harder to find, but when I found it, I knew I had what I needed to get the job done.

The resulting code wasn't too bad:

What really surprised me was that even the Storm UI was written exactly as I would have done it - all in clojure with compojure for the RESTful API. It was pretty sparsely documented, but in the end, I can understand an Apache project with sparse documentation - it's kinda to be expected.

I fiddled around with the return values and the difference between a StormTopology and a TopologyInfo - and how it's used in both the emitted counts and the calculation of the capacity for the bolts. But in the end, by looking carefully at the code as an example, I was able to get what I needed out of the library. Very nice.

This morning I was working on a simple addition to a suite of applications that does some experimental analysis on data streams, and it was all written in clojure. Wonderful language. The problem is that it's based on the JVM, so if I want to do something small, I still have a relatively long load time due to the JVM.

JRuby had this as well, but then there was RMI Ruby that was very fast to load, and it was great for smaller tasks. JRuby is great for larger projects where you need the JIT and speed you can get with the JVM.

I've seen ideas where you spin up a JVM, and have it sitting there, and then load up a new class loader and load up the code. But that means you have a JVM just sitting there all the time. That's not the answer. And I can understand the case of GUI apps - those are building up lots of buffers, etc. that you don't have to have with a simple command-line app.

Boy... that would be as useful as a really great Garbage Collector. Sure wish it was faster to launch clojure apps.

UPDATE: I have read a few blog posts about this, and I have to say that I've changed my mind on the subject - Java's JVM loader is just fine. It's the clojure loader on the JVM that's bad. So very bad. Tests on my MacBook Pro confirm that Java itself is not bad at all - very fast, once it's compiled. But clojure is still slow. Plenty of reasons for the slowness, but no clear answers.

Today I have spent most of the day tracking down some data in a massive Teradata database, and trying to relate it back to the data I needed so I could make the event stream cover this odd data as well as it covers the normal stuff. While I can understand the need for this, it's hard work because the field I'm getting in the incoming message is permalink, and it's an integer, and now I have to find what field this is in literally hundreds of tables in a massive, multi-TB Teradata install.

I was pretty sure that I had a good idea where to start looking, but I wasn't exactly sure, and true enough, I wasn't right on the money, but I was close. I then had hours ahead of me to get the right answer, and then build the SQL query that would return the rows I needed in a reasonable time-frame to build up the cache of data in redis that I was going to need.

Suffice it to say that it's not a lot of fun when you can't get folks to standardize on indexes or identifying codes in a system as large as this. But hey... it's done.

I was looking at the loading on one of my topologies today, and noticing that it wasn't catching up to a backlog from a restart as fast as I'd hoped. Since I had the capacity, it made sense to expand the topology and give it more workers, and more JSON decoder bolts, and more data filtering bolts. So I doubled the workers to 20, added many more bolts, and restarted it.

And the result was that it was slower. The capacity numbers went way up, and the backlog continued. This made no sense, but then it did. There's a balance to all things in a topology-based system like Storm. You can't increase one thing and expect it not to impact the system in other places.

So I took the changes out, and the speed returned. You can't tweak topologies without getting the performance data, and you can't measure it without upsetting the running of the topology. It's not an easy system to use, but it can be made to be quite useful. You just have to be careful.

I like Google Chrome. I use it everyday for all kinds of uses. But every now and then, they make a really silly change, and I find that it's just amazingly annoying, but there's little I can really do about it. A few versions ago, they added a drop-down in the tab bar for the logged-in username. This used to be a little image in the location bar, but they moved that out, and into the tab bar.

I know it's me - and it seems to me that the simple change would be to look and see how many profiles are defined for this guy, and if it's less than two, then don't show a thing - like it used to be.

I must be in such a minority that it's never thought necessary to fix. But it's ugly.