Archive for the ‘Open Source Software’ Category

Trying to Tune CouchDB

Friday, November 9th, 2012

CouchDB

When faced with the idea that CouchDB might be running low on sockets (file handles), I decided to do a little digging into the CentOS 5 configuration, and see what I could do. Turns out it wasn't all that hard, and I got immediate results. The first thing is to up the hard and soft limits on the file handles in /etc/security/limits.conf for the couchdb user:

  couchdb  soft  nofile  32768
  couchdb  hard  nofile  60000

then simply restart CouchDB. Look at the process limits by:

  $ cat /proc/1234/limits

where 1234 is the pid of the erlang 'beam' process, and see what it says. You are also going to need to tell Erlang about the number of file handles in the /usr/local/etc/default/couchdb file:

  export ERL_MAX_PORTS=32768
  export ERL_FLAGS="+A 8"

The second little bit there is to allow 8 threads in Erlang to process disk updates. This multi-threaded filesystem access isn't on be default for Erlang, so if you're on a big box with lots of cores, it makes a lot of sense to turn it on.

With these, you have CouchDB running about as well as you can get. The rest is up to you.

Fixing JRuby

Thursday, November 8th, 2012

JRuby

While trying to figure out a problem with the scaling, I was getting a bunch of BindExceptions and they seemed to point to this point in the jruby-1.7.0 code:

  1. try {
  2. // This is a bit convoluted because (1) SocketChannel.bind
  3. // is only in jdk 7 and (2) Socket.getChannel() seems to
  4. // return null in some cases
  5. channel = SocketChannel.open();
  6. Socket socket = channel.socket();
  7.  
  8. if (localHost != null) {
  9. socket.bind( new InetSocketAddress(
  10. InetAddress.getByName(localHost),
  11. localPort) );
  12. }
  13.  
  14. try {
  15. // Do this nonblocking so we can be interrupted
  16. channel.configureBlocking(false);
  17. channel.connect( new InetSocketAddress(
  18. InetAddress.getByName(remoteHost),
  19. remotePort) );
  20. context.getThread().select(channel, this, SelectionKey.OP_CONNECT);
  21. channel.finishConnect();
  22.  
  23. // only try to set blocking back if we succeeded to finish connecting
  24. channel.configureBlocking(true);

If I was getting bind exceptions, then it had to be coming from line 104 - after all, that's the call to bind(). The problem was, this really needed to be preceded with a call to set SO_REUSEADDR to true so that we didn't have these kinds of issues.

Something like this:

  1. try {
  2. // This is a bit convoluted because (1) SocketChannel.bind
  3. // is only in jdk 7 and (2) Socket.getChannel() seems to
  4. // return null in some cases
  5. channel = SocketChannel.open();
  6. Socket socket = channel.socket();
  7.  
  8. if (localHost != null) {
  9. socket.setReuseAddress(true);
  10. socket.bind( new InetSocketAddress(
  11. InetAddress.getByName(localHost),
  12. localPort) );
  13. }
  14.  
  15. try {
  16. // Do this nonblocking so we can be interrupted
  17. channel.configureBlocking(false);
  18. channel.connect( new InetSocketAddress(
  19. InetAddress.getByName(remoteHost),
  20. remotePort) );
  21. context.getThread().select(channel, this, SelectionKey.OP_CONNECT);
  22. channel.finishConnect();
  23.  
  24. // only try to set blocking back if we succeeded to finish connecting
  25. channel.configureBlocking(true);

Simple change, but it'd make a huge difference to the operation of the socket. So I submitted a request to the JRuby team.

After a little bit, I realized that I needed to try and build this all myself, and see if I couldn't just prove the problem for myself. The difficulty was - How to build JRuby?

Turns out, it's not all that hard. First, fork the code on GitHub so you can issue Pull Requests with the changed code. Then, follow the directions to build everything about JRuby:

  $ git clone git://github.com/drbobbeaty/jruby.git
  $ cd jruby
  $ ant
  $ ant jar-complete
  $ ant dist

It should all build just fine. Then check everything in, and push it up to GitHub. Next, you need to tell rvm to build a new version of ruby from that GitHub repo:

  $ rvm get head
  $ rvm reinstall jruby-head --url git://github.com/

And we're almost done… Update the Gemfile to have the paths to the jruby-jars file where you built it, and then update the .rvmrc. I was able to get things running and even packaging a jar file for our deployment.

Hacking on the language. That's a first for me. Quite a fun experience.

What I noticed was that it wasn't the bind on line 104 - it was the connect on line 113. Java returned the BindException as a general Can't get the endpoint, and I found this out by putting in the catch statements and logging the host and port of the exception. This was very illuminating, and I'm very glad I did it.

I'm going to even send a pull request to see if they'll take it. It's a very useful debugging tool.

Working with Bash is Really Quite Fun

Thursday, November 1st, 2012

Ubuntu Tux

I know the Code Monkeys hate it, but bash is a really great little tool. It's on every box, it's got all kinds of capabilities, and it's simple to use. Now it's a slightly different syntax than Ruby, so I'm sure that has something to do with it's distaste with the kids, but that's their issue, not mine.

This morning I was having a little fun creating summary scripts for the two other processes we run as part of the nightly batch runs. They are also ruby processes, but the other guys haven't taken the time -- OK, who am I kidding? I wrote the only summary script there is, and I just hadn't gotten around to writing them for the other two processes.

One other teammate talked about doing it, but I know he'd have written it in ruby, and then tossed out what I had, and I wasn't about to let that happen. So I took the existing code and simply added parsers for the three files, and allowed the script to tell - based on the file name or a "forcing argument", which of the three parsers to use.

Pretty nice. Very clean, and it's one simple file.

It's probably not 100% complete at this point, but it's a great start, and it's able to point out when things have gone really wrong.

Google Chrome dev 24.0.1312.1 is Out

Wednesday, October 31st, 2012

This morning I noticed that Google Chrome dev 24.0.1213.1 was out and the big change appears to be in the new WebKit (537.17) and a bunch of fixes related to the bookmark syncing and stability issues. I have to say that I really like the new look of the blog as it's much cleaner than the old, and it's also nice they are moving along with WebKit updates.

Good news.

Growl 2.0 and Notification Center Integration

Monday, October 29th, 2012

Growl 1.3

This morning I'm still getting used to the OS X 10.8.2 changes - specifically, Notification Center. This is clearly a take on Growl, and as a long-time user of Growl, I know there are apps that are only recently Growl 2 compliant, so I really doubt they are wired into Notification Center. Thankfully, there's a mode in Growl to ship all notifications to Notification Center. But there's a catch…

All notifications that are redirected from Growl come into Notification Center as Growl notifications. That's no good. You loose at least half the really useful information in this. You need to be able to tell what app sent it, and then what the message is. Thankfully, someone thought of this and created Bark.

It's a Growl 2 plug-in, and it requires OS X 10.8 Mountain Lion, and it redirects all Growl notifications to Notification Center - leaving the application intact. You simply download it, install it, and then set the 'Default Action' to be 'Bark', and the 'Default Style' to 'No Default Display', and you're done!

I'm hoping this is going to do what it says it does, but I haven't seen any notifications come through, but I've got my fingers crossed, and I'm hoping this works exactly as advertised.

Restarting CouchDB on OS X from Homebrew

Thursday, October 25th, 2012

CouchDB

Every now and then I find that CouchDB on my laptop really needs to be restarted. Because it was installed with Homebrew, it's for an entry in my ~/Library/LaunchAgents/ directory - so it's best to do it nicely. Meaning don't just kill it. The simplest way to do this is just:

  $ cd ~/Library/LaunchAgents
  $ ls
  org.apache.couchdb.plist
  $ launchctl unload -w org.apache.couchdb.plist
  $ launchctl load -w org.apache.couchdb.plist

The unload stops it, and the load starts it again. Very neat and clean.

More Issues with JRuby 1.7.0 and Dir.glob

Thursday, October 25th, 2012

JRuby

We have been trying to move to JRuby 1.7.0, and running into a fair share of issues along the way. But still, we press on. Today we tried again, and failed, being stuck by another Dir.glob issue that has been a persistent little problem for us.

There are a lot of reasons we want to be on JRuby 1.7.0 - better socket implementation, better handling of exceptions up the stack, and better ruby 1.9 compatibility. All good things. But not being able to deploy in a jar has been a real pain. We need to be able to do this, and without it, we really can't move.

So we have been digging into the issues. Thankfully, for the latest one we have a work-around. It seems that JRuby 1.7.0 can handle Dir.glob, but only if the "path" does not start with jar:. Very interesting.

The work-around, then, is very simple:

  def project_root
    @project_root ||= File.dirname(File.dirname(__FILE__)).gsub(/^jar:/, '')
  end

will get the root of the project in the filesystem or jar, and then if it starts with jar:, it'll strip it off. We use this for all file pathing, and we're good to go!

Google Chrome dev 24.0.1305.3 is Out

Thursday, October 25th, 2012

Google Chrome

This morning I noticed that Google Chrome dev 24.0.1305.3 was released with some really nice updates including the V8 javascript engine 3.14.5.0 and a significant update to how bookmarks are handled in the omnibus. From the comments, it's pretty clear that the omnibus is used by a lot of people for much more than a place to type a URL or a google query - which is nice, but it's certainly not behavior I'm looking for, so I never really "missed" it.

But as always, it's great to see the progress - even if it's something I don't really use.

Google Chrome dev 24.0.1290.1 is Out

Wednesday, October 10th, 2012

This morning I noticed that Google Chrome dev 24.0.1290.0 was out with a respectable set of release notes. It appears they've felt with some crashing issues, but not a lot else. I suppose that's OK, it's still an improvement, and that's a step. I'm not sure where they are really going but it's nice to seem them still moving forward.

Google Chrome dev 24.0.1284.2 is Out

Wednesday, October 3rd, 2012

Google Chrome

This morning I saw that the Google Chrome team has upped the major version number of Chrome to 24.0.1284.2 with a few nine updates, and several bug fixes - notably a few Flash issues, crashes, and a new version of WebKit and the V8 javascript engine. Nicely done. I'm curious why this qualifies as a major release number, but I think it has more to do with when the version gets stable enough to deploy out to the masses, then the dev version has to up it's number to be unique. Sure seems like it anyway.

Well… nicely done to the Chrome team!