What's it all about, Alfie?

I saw a tweet today from the guys at github and they have created a fantastic bit of JavaScript that "slides" the contents of folders right and left as opposed to fetching a new page on each "drill down" into a folder. It's amazingly neat in that they also made the browser's "back button" work. Pretty amazing stuff.

This morning I finished patching ZeroMQ for the Recovery Interval in Milliseconds option that I was looking to have in order to control the memory usage. I was able to build the code into the shared library, and then by carefully placing it in the project's lib directory, I was able to get the loader to pick it up as opposed to the RPM-installed older version. With this, I was able to verify that the option worked and the savings in memory was, once again, quite substantial. Most excellent!

I then followed the submission guidelines and sent a nice patch to the mailing list. One thing I noted was the Signed-Off-By tag and it's usage in the project. I can see why they have it in git - being generated from the kernel group, but also for a project like this. I really like this kind of stuff - to have the SCM tool understand the needs of it's users and anticipate them.

After I got the one patch submitted, I got a message on IRC from Martin saying I needed to subscribe to the mailing list. Oddly enough, I mentioned to him, that I was subscribed. He then asked me what email? I responded with my general incoming drop box, and then he pointed out that the email was coming from my Comcast account. Ah! Got it.

Rather than use my Comcast account for anything more, I decided to subscribe on my GMail account as it's not going to change, and then send in emails to the list on that same account. It's far easier to use that and then not worry about it than to worry about Comcast going away because of some better ISP in Naperville. I spent a little time trying to get outgoing email at EasyDNS, my DNS service, but that turned out to be a bit dodgey, and realized it's probably better to just stick with GMail and let that be it.

OK... one patch down, one to go - the Java client needs to have the same capabilities to set the Recovery Interval in Milliseconds, and while it used JNI to get to the C library under the covers, it needed to have a few things added to make it really work. Not nearly as difficult as the C library, but still, I followed the same guidelines and submitted a patch for that - now on my GMail address.

I saw a few minutes later that my patch was in the mailing list digest, which confirms that I got everything right with the GMail switch-over... nice. We're running with the changes on our boxes, but if they happen to make changes in the near future we'll just have to deal with it as that's the cost of being a submitter.

Neat stuff. Fun.

[3:30pm] UPDATE: I have to say... just when you think you have it all under control, life pops up and shows you you aren't as put-together as you think. Case in point: today I thought I had a good set of patches to ZeroMQ and it's Java client, but then I tried to receive the data and it was all a mess. I then had to send something to the mailing list that said "I'm a dufus", and then back out all the changes and verify that my code worked.

It didn't, and so I spent a good 30 mins finding out why not - only to see that one of the options on ZMQ was really not all that well documented - the Ignore Loopback option. They didn't mean the loopback interface, they meant any other client on the same box. So I had to drop that option and then things started working.

Well, at that point, it made sense to try and see if I could get the fixes in ZMQ back in and working. It didn't take long, and sure enough, it's working. So I sent in another patch with a few additional changes, and that should be good to go.

Nothing like being shown to be a dufus in front of people you'd like to impress. Yeah... class act all the way.

This morning I saw that Cyberduck 3.8.1 is out, and while I had gotten 3.8, I hadn't written about it because I hadn't really even fired it up. I have to say that while I didn't mind the nagware screen asking for you to register the product, the little tag on the title bar is a new level of intrusive.

OK... I'm not a registered user, but then again, I'm a Transmit user and pay for all it's updates, etc. Cyberduck is an emergency back-up when Transmit might not get the job done. But to be fair, for those that don't want to pay for an FTP client, and I know a lot of people in that boat, this is a great tool. It's fast and capable, and the nagging is the equivalent of in-app adds on the iPhone. Something I hate, and will pay to see removed, but something that is just fine with a lot of folks.

This morning I noticed that Google Chrome dev 9.0.597.10 was out, and the release notes indicate that it's just stability fixes and a few UI tweaks. Fair enough... not every release can be amazing, and it's getting into it's final form. There hasn't been a big change in the UI in quite a while, and the improvements in the V8 engine or other infrastructure components are going to be constantly ongoing.

Still... it's nice to see progress.

This morning I was chatting with Martin S. on the ZeroMQ IRC channel and there was a suggestion of how to handle the "socket recovery interval in msec" option in the code. He pointed out that I'd need to change the ZeroMQ code, and why didn't I do that and then send the patch to the mailing list and he'd incorporate it.

Sweet! A request for a (simple) patch to the codebase by the primary maintainer. I like this stuff. It's not hard, but there are a few wrinkles, and the coding standards are at least in existance, which is a huge help to the project. I just need to get a few things figured out, write the code, compile it all up, and then make the diff for the mailing list.

I'm sure there's going to be a lot of little details I learn as I do this, but it's nice to get a chance to contribute to another nice open source project.

UPDATE: I've pretty much got it all done, but the hint I received from the guy that really knows OpenPGM in the group is a little sketchy. He gave me an equation which includes the size of the transport package:

Easy workaround is to calculate the buffer size in sequence numbers in 0MQ and pass that onto OpenPGM. Then you can export socket options for 0MQ to set the buffer size in seconds, milliseconds, etc.

int sqns = (secs * max_rte) / tpdu_size;
pgm_setsockopt (sock, IPPROTO_PGM, PGM_TXW_SQNS, &sqns, sizeof (sqns));

I think I found what should go in that spot, but I wasn't 100% sure. So I replied to the guy on the mailing list and now I'm waiting for confirmation/correction from him. It shouldn't take too much longer to finish this up, and then I'll have a way to set the ZMQ_RECOVERY_IVL_MSEC - which, with a non-zero value, will override the ZMQ_RECOVERY_IVL value and use the value in milliseconds. Should be pretty easy to finish.

This morning I saw that Skitch 1.0 (and then 1.0.1) was released as a commercial product. Fantastic news! The basic Skitch is free with adds on the web pages where images are shown, and a few restricted features in the desktop app itself. For $14.95/year you can upgrade to Skitch Plus and remove the online ads, and unlock the app's features.

I like to support the developers doing great work so I signed up. Seems reasonable and hopefully, it will ensure that the service will be around for a long time to come.

I noticed this morning that WordPress 3.0.2 was out with some security fixes. It seems to say that they have now figured out how to update WordPress from within WordPress. If so, that would be very interesting indeed. I still used the SimpleScripts upgrade path that HostMonster has - I didn't see a need to "test" that part today. Maybe the next update.

But hey... I'm all up to date and that's great news.

This morning has been very enlightening on ZeroMQ. Very exciting stuff. As I was leaving yesterday I had made a test app for the ZeroMQ guys to check and then posted the following test results as I varied the value of ZMQ_RATE:

The data was pretty compelling. The effect ZMQ_RATE had on the memory footprint of the same data source was staggering. Thankfully, I put it all together in a nice email to the mailing list and I got a great hit from Martin S.:

Isn't it just the TX buffer? The size of PGM's TX buffer can be be computed as ZMQ_RATE * ZMQ_RECOVERY_IVL. The messages are held in memory even after they are sent to allow retransmission (repair) for the period of ZMQ_RECOVERY_IVL seconds.

So I added the following to the ZMQ transmitter's code:

  static int64_t     __rate = 50000;
  static int64_t     __recovery = 1;
  static int64_t     __loopback = 0;
 
  // we need to set this guy up properly
  top->socket->setsockopt(ZMQ_RATE, &__rate, sizeof(__rate));
  top->socket->setsockopt(ZMQ_RECOVERY_IVL, &__recovery, sizeof(__recovery));
  top->socket->setsockopt(ZMQ_MCAST_LOOP, &__loopback, sizeof(__loopback));

And then started running the tests again.

The results were amazing:

This was exactly what I was looking for! The ZMQ_RECOVER_IVL can't go below 1 sec, but for me even that's too much. If you're not here and ready to get ticks, then waiting a second is likely to be several hundred if not several thousand messages. It'd be fine with me to make it 0.5 sec - but Martin says that's the underlying resolution of OpenPGM.

Not bad. I'll take it. What a great morning!

[12/7] UPDATE: the option:

  static int64_t     __loopback = 0;
 
  top->socket->setsockopt(ZMQ_MCAST_LOOP, &__loopback, sizeof(__loopback));

is a massive red herring. It's not about the loopback interface, as my reliable multicast URLs are all targeted to specific NICs, it's more about being able to receive on the same box as the sender. I was trying to figure out why things "broke", and it's when I took this out that things worked again. Dangerously worded docs on this one... leave it out.

I've been trying to track down what I believed to be a nasty memory leak in my code today. The short-cut to the answer is that it wasn't a leak, and it wasn't in my code. But I'm getting ahead of myself.

The problem was manifesting itself as steadily growing memory on some of my ticker plants. In truth, it was probably all of them, but it wasn't effecting all of them equally. I have spent a lot of time on this over the past weeks, and today I was going to get to the bottom of this for sure.

So I started digging into the problem by shutting things off. What I found was that if I was listening to anything on the UDP socket and doing anything with it I was getting about an 8-byte increase every two seconds. Very odd. I had turned off ZeroMQ at the time, so the messages were just getting dropped in the trash, but they were being processed completely up to that point.

I was trying everything, and then I had to run to a meeting. I left the test running because I needed to hurry. It wasn't going to consume the box in half an hour, anyway.

When I came back I noticed that the memory had stabilized!

Now it was getting interesting. Very interesting. I started tracking things down and it turns out that the ZMQ_RATE parameter was a major factor in the terminal memory value. I then wrote up a simple test - something that I knew the ZeroMQ guys would appreciate, and started running it.

Again - major dependency on the value of ZMQ_RATE. I'll have to do more work on this tomorrow.

After quite a silence, Google Chrome dev 9.0.597.0 is out and there are some really nice fixes in this release:

All

• Ongoing work on IndexDB and GPU
• Tweaks/Fixes to Google Chrome Instant
• Extensions/Apps work
• Autofill related fixes

Known Issues

• Page becomes unresponsive when trying to play video - Issue 65772
• Certain HTML5 sites fail to load due to a compositor issue - Issue 64722

I like the GPU updates and the video updates, but I can pass on the "instant"... icky addition in my book, but their app, their choice.