What's it all about, Alfie?

I was talking to a very good friend the other day, and we were chatting about what I thought of the Ruby world, and even the Clojure world, and I had to say that what I've seen of the two languages, and jruby as an implementation, is very nice - but the communities that sprout up around these two languages are really far more destructive than I've ever seen in my 30+ years of development experience. The ruby community especially is almost toxic in it's adoption of the "Lazy Coder". Don't get me wrong, I'm all for the Magic of the Gem, but there's a point when a library - be it boost or a gem, has to expose what it's doing so that the user can see if it's worth using in their implementation.

What I see most often in the tubists I work with is a complete blindness to what's really happening in the gems until such time as they are tracking down a bug, and trace it to one of these gems. Then they either abandon it for another gem, and re-work their code, or they fork it, make a pull request, and carry on.

I think the latter is admirable, but I think far too little thought is put into the code, and therefore the gems, before a bug hunt is underway. This means that if things work - and in most cases, these are simplistic web sites/services so they don't focus on edge cases, and "production" means "usually up, and mostly working", so things typically work with these gems, and the truth of the cost of these libraries is totally unknown to the user.

If I were doing this for fun, that'd be OK. I'd live with it because it's only a "fun project", but when I'm getting paid for this work, I know that performance matters. Heck, everything matters - from the documentation to the runtimes, to the maintainability, to the ease of deployment. It all matters. So you can't pretend that dropping a gem into your app is magically going to do anything. It's going to extract something for it's service, and that something you need to know about.

But that's the rubyist way… to have "magic" gems. You drop this in, and by convention you have a named method/function and all the rest is done for you. I can really appreciate that - JavaBeans was all about that. But it's how it's taken and abused that makes me shake my head. Use it, but understand what it's really doing. Then you can know when it's no longer the appropriate tool for the job.

Anyway… this is never going to change. In fact, I'm guessing it's only going to get worse with time. I'm an old-timer now. A dinosaur that looks to most like I'm more interested in using stone knives and bear skins, but I tell you this - lack of real understanding is the true source of bugs.

This afternoon I was trying to deploy out jruby project to UAT, and I got the following error:

  sendmail: fatal: chdir /Library/Server/Mail/Data/spool:
      No such file or directory

and I was immediately saddened by the development. What's happening is yet another of the rubyists shortcuts and magic gems - they wanted to have emailing from within ruby, and rather than make sure there's a decent, workable, SMTP gem - which there has to be, or there should be because of how easy it is to write, they went with the first thing they saw, and it uses sendmail.

Now I don't have anything against sendmail, but it's the completely wrong tool for the job. They have had to put in a user's name - so all emails seem to come from one person, as opposed to the person doing the activity. It's just a piece of junk, and for good reason - it's the wrong tool for the job!

But I have to make my laptop work with this. It's not running by default, and the reason is that it's the wrong tool for the job, but that's something I'll take up with them another day. Thankfully, we have a solution:

  sudo mkdir -p /Library/Server/Mail/Data/spool
  sudo gzip /usr/share/man/man1/{postalias.1,postcat.1,postconf.1,postdrop.1, \
          postfix.1,postkick.1,postlock.1,postlog.1,postmap.1,postmulti.1, \
          postqueue.1,postsuper.1,sendmail.1}
  sudo gzip /usr/share/man/man5/{access.5,aliases.5,bounce.5,canonical.5, \
          cidr_table.5,generic.5,header_checks.5,ldap_table.5,master.5, \
          mysql_table.5,nisplus_table.5,pcre_table.5,pgsql_table.5,postconf.5, \
          postfix-wrapper.5,regexp_table.5,relocated.5,tcp_table.5,transport.5, \
          virtual.5}
  sudo gzip /usr/share/man/man8/{anvil.8,bounce.8,cleanup.8,discard.8,error.8, \
          flush.8,local.8,master.8,oqmgr.8,pickup.8,pipe.8,proxymap.8,qmgr.8, \
          qmqpd.8,scache.8,showq.8,smtp.8,smtpd.8,spawn.8,tlsmgr.8, \
          trivial-rewrite.8,verify.8,virtual.8}
  sudo /usr/sbin/postfix set-permissions
  sudo chmod 700 /Library/Server/Mail/Data/mat
  sudo /usr/sbin/postfix start

and with these changes, the system has sendmail running, and the gem works.

I can't think of a more completely wrong solution to the problem, but these guys aren't about the "right" answers - they're about the "magic" ones. They want to just drop a gem into a Gemfile, bundle it, and then have it do all the magic. They'll give it a bunch of configuration, and rather than question the use of such a gem, they'll just completely contort the project to the point that it fits the usage of the unappropriate gem.

It's bizarro programming.

I hate it. I really do.

UPDATE: Funny developments… the gem we're using is called Pony, and can use SMTP or sendmail, you just have to configure it differently. Also, it turns out that sendmail is the preferred way to send emails at The Shop. Kinda odd to me… to put sendmail on all hosts to be able to send simple emails, but OK… I'm a "team player", I'll back off. But what a waste of cycles - there are so much easier ways to do this same thing. Even if we keep the same gem.

Seems the Google Chrome guys have been busy - this morning I noticed they dropped dev 26.0.1386.0 and with it, some great release notes. Seems 'Tanya' is taking over, and she sees these as important. Lots of nice little changes, but what's really important to me is that someone new is in charge, and they care about communication with the people using their product.

Very nice to see!

I told myself this morning that I didn't want to get upset at the folks I work with any more. I told myself it's a bad thing to do - and it never accomplishes my goal of actually getting something done. Never.

Yet here I find myself, some four hours later, fighting very hard not to be nasty to one of the Code Monkeys when he gets in. The story goes like this:

We were looking into a problem late yesterday, and I believed it to be possible that the choice of Rack web server had something to do with it. We were running 'thin' in development, but 'unicorn' in production. So this morning, I wanted to move development to 'unicorn', and see if I could reproduce the problem in development.

I look at the git repo and see that another developer had made these changes already. Well… some of the changes. He didn't remove the 'thin' gem from the gemset, and that needed to be done. And he didn't document how to start the 'unicorn' server other than the fact that there's a Procfile in the root of the project.

There is a place on the GitHub README.md for how to start this, and he hadn't changed a thing. So I figured a little googling might help out.

The only thing I could find was that a gem called 'foreman' uses a Procfile and it looks about the same, but we don't have 'foreman' in the Gemfile, so I can't see how he's using it. Unless he installed it globally and didn't update the docs to reflect that.

And, of course, he's not in and it's 8:40 am.

It's this kind of thing that really drives me crazy. They make a change - commit to master and don't update the docs. They either don't think it's important, or that it's clearly obvious, or that they are just plain forgetful. But for whatever the lack of discipline, they are now blocking me from working. That's pretty annoying.

If it were 10:00 pm, and I wanted to do a little late-night coding, I could see that. It's off-hours, and he's got time to get it ready. But they all know that we have the rules about master, and they all know when I get in. This is just having no discipline about what you're doing.

It's very hard for me to maintain a calm exterior when I see this around me all the time - every day. I don't think a one of these Code Monkeys takes this project seriously, and maybe that's OK for them. But when I'm constantly having to bust my hump for deadlines for this entire group, and getting so little help as this, it's really hard to not get angry at them.

This isn't calming me down… I had hoped that it would, but it's not. Gotta just stop writing then...

This morning I checked and Google Chrome dev 26.0.1384.2 was out with the now common lame release notes. It's like they aren't even trying. But hey, it's fast, renders pages nicely, and it works. In the grand scheme, I can live without the release notes if they keep making good software.

This morning I noticed that the crontab job I had to cleaning up the stale, unused jars that happen when you have a project in jruby or clojrue where everything is in a single jar, and they are all tagged with the GitHub SHA to tell them apart. Actually, it's about the nicest thing about jruby and clojure because it means that you don't have to hassle with directories of files, and it's super easy to have a symlink to the current version and then de-reference it in the script with a little:

  jar="$HOME/dark-magic-current.jar"
  actual_jar=`readlink $jar`

and then in the rest of the code for running the app, use $actual_jar and you can deploy on top of a running system without disturbing it. Restart, the script will pick up the new symlink, and you're in business. Pretty nice, actually.

Anyway, the bug was in the vacuum script:

line 26 used to read:

  for i in "${oldJars}"; do

and I learned that because oldJars was a variable, what I was doing by quoting it was to make that one value for the loop - and not a series like I had hoped. Simply removing the quotes took care of the problem as bash is smart enough not to have thought the space-delimited names in the variable were actually separate commands.

Bash is smarter than I thought. Nice.

Simple fix, and now we are checking all the jars for expiration.

Today I struggled trying to get the same keep-alive going in out production server, but because it's running Unicorn and not Thin, and Unicorn is not an event-driven server, it was a lot harder. To be fair, a lot of this had to do with the problems I was having with the deployment of software to the boxes. The Shop has a lot of progressive tools, but the deployment of packages is not among them. So there were several hours lost there.

But the real problem was figuring out how to integrate an EventMachine into the Unicorn server. Thankfully, I was able to find a simple way to do it:

  Thread.new { EventMachine.run }

So simple, but so completely non-obvious.

With that, it started working and I just continued to struggle with the fact that Unicorn doesn't expose a logger like Sinatra/Thin.

These are not the easy tools that folks make them out to be.

This morning I noticed that Google Chrome dev was updated to 25.0.1364.26 with it's now typically terse release notes. It's loading fast and rendering just fine, so I'm not going to complain, but you'd think that an engineer at Google would have a little more pride in their work than these lame release non-notes. Kinda sad.

Today I've been working very hard to get this ruby app (Sinatra) to have a keep-alive with requests that take a very long time to complete. Typically, you might think this was a case for streaming the data, and in some cases, that's true. But in many of the cases I'm dealing with, the server has to marshall a lot of data, piece it together, and then format the output. It's really not ready until it's all ready. So I needed a simple keep-alive.

Back in a previous job, I used Tomcat and just had a thread that sent a space every 10 seconds to the client until I stopped it. I then put this in the rendering code, and all of a sudden, everyone got this keep-alive code, and no one had to worry about a thing. I know it's not going to be that easy this time, but it's not that far off.

First, we need to make sure that we're using the 'Thin' server in Sinatra. WEBrick doesn't support an event model, and so won't be starting an instance of EventMachine. This is thankfully pretty easy to do, just require 'thin', and backup will detect it and we're good to go:

  require 'thin'

Next, we need to put the keep-alive in the individual calls. This one, for instance, takes a while to complete:

  get "/call_list_spreadsheet/:dbname/:extag.tsv" do
    content_type('text/tab-separated-values')
    attachment("call_list-#{params[:dbname]}-#{params[:extag]}.tsv")
    stream do |out|
      timer = EventMachine.add_periodic_timer(10) do
        out.write("\0")
      end
      out.write(CallListTSV.new(params[:dbname], params[:extag]).render)
      timer.cancel
      out.flush
    end
  end

the key component here is that little bit in the middle:

      timer = EventMachine.add_periodic_timer(10) do
        out.write("\0")
      end

Simply put, that's going to send a NULL to the client every 10 sec. until the timer is cancelled. This works like a charm to do what I need. Amazingly so. I was really quite pleased.

All this is in a nice little gist.

This morning I went to create a new Gist, and saw that GitHub has once again, upped the ante on the source control tools with a complete facelift on the Gist pages. This is really amazing. It now allows for tabs or spaces, and even allows for the size of the indentation. This is just superlatively cool!

I'm amazed that these guys are able to keep raising the bar. It's so unusual for me to see things that I love, and yet had no idea I felt I needed them. These changes are just more of the same from GitHub. What an amazing group of people.