What's it all about, Alfie?

I have to say that one of the coolest things about Textual 5 was already in Textual 4.1.8, but I didn't really avail myself of it a lot - and that's the CSS styles for the UI. I've been fine-tuning the CSS for the effects in the window - like the messages from the server, and the messages about why I might have disconnected, and then the line separating the old from the new messages... it's all looking so much better, but it's all just simple pixel moves.

I really love this app. Wonderful support on IRC - no less!

I just noticed that there was a paid update to Textual 4.1.8 - Textual 5 in the Mac App Store. I can't even remember what got me looking in the #textual chat room, but there it was, and that was really good news. I'm a big fan of Textual, and that they had an upgrade for OS X 10.10 is really nice. I do with Adium had one.

The UI changes aren't major - which is nice, and it reads the configuration from 4.1.8, which is also really nice. They got rid of the hidden check box to not restrict the window size, but his concession was to make the minimum "small enough" that it wasn't needed.

Leave it to me to point out that I wanted about 20% less.

So I let him know, and we'll see if there's a change in the offing. It's not horrible as-is, but I would like to have the freedom to make it a little smaller... but I can live for now.

UPDATE: the CSS configuration of Textual is just amazing. I'm able to customize this view to a level that's more than I ever imagined. I'm sure there are lots of others that thinks this is no big deal - but to me, screen real estate is critical, and the ability to customize a view like this is just fantastic. Very highly recommended.

I honestly would have put money on the fact that this would not have happened today. Big money.

I'm sitting on the bus riding to work, and I realize that I'm pretty happy without a pain-causing personal relationship in my life. That was a wow! moment. I've been separated for about 2 years, and the divorce is in the works, but I would have bet real money I'd feel horrible for the rest of my natural life. But today... on the bus... for a few minutes... I didn't.

That was huge for me. Huge.

Then I'm in work, updating a few postings with the results of the tests I'd done overnight, and I'm back into the swing of posting like I used to. It's been a long two years, but I'm back to writing about what I'm doing, and it's really helping. I'm feeling like I'm enjoying myself again.

This, too, was huge for me.

I don't expect this to last all day... but the fact that I have felt this way tells me that I need to keep doing what I'm doing - keep moving forward, and then maybe this will come again. And maybe when it comes again, it'll last longer. Maybe.

OK, this has to be one of the hardest topologies to balance I've had. Yeah, it's only been a few days, but Holy Cow! this is a nasty one. The problem was that it was seemingly impossible to find a smoking gun for the jumps in the Capacity of the bolts for the topology. Nothing in the logs. Nothing to be found on any of the boxes.

It has been a pain for several days, and I was really starting to get frustrated with this guy. And then I started to think more about the data I had already collected, and where that data - measured over and over again, was really leading me. I have come to the conclusion it's all about Garbage Collection.

The final tip has been the emitted counts from the bolts during these problems:

where the corresponding capacity graph looks like:

The tip was that there was a drop in the emitted tuples, and then a quick spike up, and then back to the pre-incident levels. This tells me that something caused the flow of tuples to nearly stop, and then the system caught back up again, and the integral over that interval was the same as the average flow.

What lead me to this discovery is that all the spikes in the Capacity graph were 10 min long. Always. That was too regular to be an event, and as I dug into the code for Storm, it was clear it was using a 10 min average for the Capacity calculation, and that explains the duration - it took 10 mins for the event to be wiped from the memory of the calculation, and for things to return to normal.

Given that, I wasn't looking for a long-term situation - I was looking for an event, and with that, I was able to start looking at other data sources for something that would be an impulse event that would have a 10 min duration effect on the capacity.

While I'm not 100% positive - yet - I am pretty sure that this is the culprit, so I've taken steps to spread out the load of the bolts in the topology to give the overall topology more memory, and less work per worker. This should have a two-fold effect on the Garbage Collection, and I'm hoping it'll stay under control.

Only time will tell...

UPDATE: HA! it's not the Garbage Collection - it's the redis box! It appears that the redis servers have hit the limit on the forking and writing to disk, and even the redis start-up log says that there should be the system setting:

  vm.overcommit_memory = 1

to /etc/sysctl.conf and then reboot or run the command:

  sysctl vm.overcommit_memory=1

for this to take effect immediately. I did both on all the redis boxes. I'm thinking this is the problem after all.

Best results I could have hoped for:

Everything is looking much better!

Glui - a great tool for replacing Skitch, is having a little problem with OS X 10.10, and it's not that hard to see:

So I emailed the developer, and within 15 mins I got a response that the fix was in the Mac App Store, and I should see the update in a few days. Sweet! I use this for a lot of my journal posts, so it's nice to get it all fixed up, and I am really liking the OS X 10.10 UI, so it's just nice to see them so responsive.

Great tool - Great support!

Turns out that Adium 1.5.10 and Yahoo! IM isn't happy with OS X 10.10 - it's saying that it can't connect to the Yahoo! IM server and erring out. Normally, this wouldn't be an issue for me, but I've got a few friends that I stay in touch with all the time, and because of their employer, two of them have been kinda off-limits to me for a while - but one finally found his way back to me through Yahoo! IM.

While I'm really loving the changes in OS X 10.10, sadly, with the update, the Yahoo! IM connection has failed to work. And it's not at all helpful in any way:

What I've read from the Adium blog is that there is a known fix - but there's a potential security concern, and I'm not at all sure why they are delaying releasing a patch for Yosemite... but they are.

So I'm out of touch with an old friend until I can get an update from Adium. This is one of the issues with Open Source - abandoned software. I'm sure the point is that I can pick it up and fix it, and I might be able to, but it's a pain, and it's something I've come to depend on. In that, I like paying for things.

I think I've finally cracked the nut of the stability issues with the experiment analysis topology (for as long as it's going to last) - it's the work it's doing. And this isn't a simple topology - so the work it's doing is not at all obvious. But it's all there. In short - I think it's all about Garbage Collection and what we are doing differently in the Thumbtack version than in the Third-Rock version.

For example, we had a change of the data schema in redis so that we had the following in the code to make sure that we didn't read any bad data:

  (defn get-trips
    "The experiment name and variant name visited for a specific browserID are held
    in redis in the following manner:
 
      finch|<browserID> => <expr-name>|<browser>|<t-src>|<variant>|<country> => 0
 
    and this method will return a sequence of the:
 
      <expr-name>|<browser>|<t-src>|<variant>|<country>
 
    tuples for a given browserID. This is just a convenience function to look at all
    the keys in the finch|<browserID> hash, and keep only the ones with five values
    in them. Pretty simple."
    [browserID]
    (if browserID
      (let [all (fhkeys (str *master* "|" browserID))]
        (filter #(= 4 (count (filter #{\|} %))) all))))

where we needed to filter out the bad tuples. This is no longer necessary, so we can save a lot of time - and GC by simply using:

  (defn get-trips
    "The experiment name and variant name visited for a specific browserID are held
    in redis in the following manner:
 
      finch|<browserID> => <expr-name>|<browser>|<t-src>|<variant>|<country> => 0
 
    and this method will return a sequence of the:
 
      <expr-name>|<browser>|<t-src>|<variant>|<country>
 
    tuples for a given browserID. This is just a convenience function to look at all
    the keys in the finch|<browserID> hash. Pretty simple."
    [browserID]
    (if browserID
      (fhkeys (str *master* "|" browserID))))

At the same time, it's clear that in the code I was including far too much data in the analysis. For example, I'm looking at a decay function that weights the most recent experiment experience more than the next most distant, etc. This linear weighting pretty simple and pretty fast to compute. It basically says to weight each event differently - with a simple linear decay.

I wish I had better tools for drawing this, but I don't.

This was just a little too computationally intensive, and it made sense to hard-code these values for different values of n. As I looked at the data, when n was 20, the move important event was about 10%, and the least was about 0.5%. That's small. So I decided to only consider the first 20 events - any more than that and we are really spreading out the effect too much.

Then there was the redis issues...

Redis is the bread-n-butter of these storm topologies. Without it, we would have a lot more work in the topologies to maintain state - and then get it back out. Not fun. But it seems that redis has been having some issues this morning, and I needed to shut down all the topologies, and then restart all the redis servers (all 48 of them).

I'm hoping that the restart settles things down - from the initial observations, it's looking a lot better, and that's a really good sign.

This morning I just got a little antsy and decided that on my personal laptop, it was time to upgrade it Mac OS X 10.10 Yosemite. I am so glad I did. The flat look - on all the web sites of the reviews - is just what I like. Almost no chrome - no need for it. Smooth visuals on my retina MacBook Pro, but I'm guessing that's the key to this upgrade - having a rMBP. Still... I've got one :), so I'm loving this.

I'm still on the Light treatment as I don't find the transparency at all intrusive or distractive - as I've been using it on MacVim for a while. MacVim was my only concern about working, but that's fine. What I didn't expect was that Adium 1.5.10 could no longer connect to Yahoo! IM. That's not good because I still reach out to Jeremy on Yahoo!.

I looked on the web, and it turns out that this is a known issue with OS X 10.10, and there is a fix for it, but it's not out yet - not even in beta. I'm a little surprised by this as I didn't really dig into what the patch was, but they have been so proactive in the past, I'm surprised that there's no beta out there for this.

After all, this has been out in "general release" for quite a while now, and while they may not want to target each update, it's something else to know there's a fix, and let weeks pass and no beta or update. I guess I'll have to wait.

I have been working one the re-configuration of the experimental analysis topology to try and get stability, and it's been a lot more difficult than I had expected. What I'm getting looks a lot like this:

and that's no good.

Sadly, the problem is not at all easy to solve. What I'm seeing when I dig into the Capacity numbers for the bolts are that there's only one of the 300+ bolts that has a capacity number that exceeds 1.000 - and then by a lot. But all the others are less than 0.200 - well under control. So why?

I've looked at logs, I've looked at the logs - nothing... I've looked at the tuples moving through the bolts, and interestingly, found that many of them just aren't moving any tuples. Why? no clue, but it's easy enough to scale it back to me a more efficient topology.

What I've come away with is the idea that might be the different way we're dealing with the data. So this weekend, if I have time, I'll dig in and see what I can do to make this more even between the two. It's not obvious, but then - that's 0.9.0.1 software.

OK... it turns out that I needed to change my topology in order to get stability, and it's only taken me a day (or so) to find this out. It's been a really draining day, but I've learned that there is a lot more about Storm that I don't know - than that which I do.

Switching from having one bolt that calls two functions to two bolts each calling one function - should certainly have a lot more communication overhead. Interestingly, it's how I regained stability. The lesson I've learned - Make bolts as small and purposeful as possible.

Amazing...