What's it all about, Alfie?

This afternoon I was working on adding more sharding to the caching I was doing with the redis servers - this time a 4-way shard based on the same sharding solution I had done just a few days ago. The modification was simple:

  (defn deal-shard
    "Function to return the right function that will be
    used with 'wcar' to get the provided key from redis.
    This is specific just to the deal data"
    [k]
    (case (hash8 k)
      (0 1) :deal-ids-1
      (2 3) :deal-ids-2
      (4 5) :deal-ids-3
      (6 7) :deal-ids-4))

and this was going to make it faster and easier to handle the load of the email sends we were getting. This was because I was a little worried that we were overloading the single redis instance for the deal data.

When I was putting this in the code I realized that the user caching I had set up was a complete sham! I had changed the keys to be much smaller, but only on the look-ups, and not on the writes! This means that every single hit was a cache miss, and the back-end database was required for every augmentation.

What a mess. I needed to fix the storage keys, let the old ones expire, and then populate it all again. With this in place, we are seeing a lot better response from the decorate bolt. Much better performance from the cache when it's actually used.

Crazy bug, but I'm glad I caught it.

This entry was posted on Monday, September 15th, 2014 at 2:27 pm and is filed under Clojure Coding, Cube Life, Open Source Software. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.