What's it all about, Alfie?

Late yesterday I found a slick little javascript-based JSON formatter and syntax highlighter. What I was looking for was a simple and easy way to get the JSON data out of the demand service and display it to the user. The point was to have a clean way for the users to be able to see the low-level data for a specific demand in a nice way so that they can look into problems without having to go to a developer or support person.

The project manager said that we didn't need to have anything fancy - just display the JSON data for the object as it's reasonably self-documenting, and most of the folks could figure out what they needed from that. For the more serious use-cases, we're still going to need to have a more comprehensive dashboard, but for now, this would be something we could whip up pretty easily and solve 90% of the need.

The code was really amazingly simple:

I have the classifier and then the CSS for the individual classes. Then I just have to hit the server based on the demand ID and show the results. Very nice and clean.

Pretty stylish, too! I'm impressed.

I've been having issues with timeouts on a web app that displays certain metrics for the main project I'm on at The Shop. The set-up is that we have a Sinatra app - using the Unicorn server behind an nginx server that's a gateway for the service. I didn't set this up, so I can't really say Why we have the nginx server in front of the unicorn server, as the point of the unicorn server is to handle the load nicely, and it's all on one box, so we're not getting redirection, but for some reason we have this set-up, and we're getting these timeouts.

Initially, I tried to have an EventMachine timer that sent a NULL to the client every 10 sec. This appeared to work, but then we started having issues with the thread it was in dieing. Then we had to put in a check-with-restart on each call. Things clearly weren't getting any simpler, and then I decided to try to convert everything to jruby from MRI, as we really didn't have any good answers as to how this was being deployed and configured. There were just too many holes.

So I reached out to one of the guys that was supposed to be the architect of this deployment scheme. I didn't know who it was, but when I found out I was really quite pleased - it was a guy that I've been interviewing with several times over the course of the last few months at The Shop.

He explained how the deployment was done, where the config and init.d scripts were generated, how to modify their generation, and in short, how to get rid of the timeouts by simply changing a few of the timeouts in the unicorn and nginx servers. So I took his advice, stripped out the EventMachine stuff from the code and then gave it a whirl.

Worked like a champ.

What a difference getting the right help makes. Inside of about an hour I had all the information I needed and had a solution that was cleaner, easier, and more maintainable than I had started with. The problem seems simple: find help, get it, and solve the problem. But too many folks - including me, initially, didn't do that.

Shame on me.

I was talking to a very good friend the other day, and we were chatting about what I thought of the Ruby world, and even the Clojure world, and I had to say that what I've seen of the two languages, and jruby as an implementation, is very nice - but the communities that sprout up around these two languages are really far more destructive than I've ever seen in my 30+ years of development experience. The ruby community especially is almost toxic in it's adoption of the "Lazy Coder". Don't get me wrong, I'm all for the Magic of the Gem, but there's a point when a library - be it boost or a gem, has to expose what it's doing so that the user can see if it's worth using in their implementation.

What I see most often in the tubists I work with is a complete blindness to what's really happening in the gems until such time as they are tracking down a bug, and trace it to one of these gems. Then they either abandon it for another gem, and re-work their code, or they fork it, make a pull request, and carry on.

I think the latter is admirable, but I think far too little thought is put into the code, and therefore the gems, before a bug hunt is underway. This means that if things work - and in most cases, these are simplistic web sites/services so they don't focus on edge cases, and "production" means "usually up, and mostly working", so things typically work with these gems, and the truth of the cost of these libraries is totally unknown to the user.

If I were doing this for fun, that'd be OK. I'd live with it because it's only a "fun project", but when I'm getting paid for this work, I know that performance matters. Heck, everything matters - from the documentation to the runtimes, to the maintainability, to the ease of deployment. It all matters. So you can't pretend that dropping a gem into your app is magically going to do anything. It's going to extract something for it's service, and that something you need to know about.

But that's the rubyist way… to have "magic" gems. You drop this in, and by convention you have a named method/function and all the rest is done for you. I can really appreciate that - JavaBeans was all about that. But it's how it's taken and abused that makes me shake my head. Use it, but understand what it's really doing. Then you can know when it's no longer the appropriate tool for the job.

Anyway… this is never going to change. In fact, I'm guessing it's only going to get worse with time. I'm an old-timer now. A dinosaur that looks to most like I'm more interested in using stone knives and bear skins, but I tell you this - lack of real understanding is the true source of bugs.

This afternoon I was trying to deploy out jruby project to UAT, and I got the following error:

  sendmail: fatal: chdir /Library/Server/Mail/Data/spool:
      No such file or directory

and I was immediately saddened by the development. What's happening is yet another of the rubyists shortcuts and magic gems - they wanted to have emailing from within ruby, and rather than make sure there's a decent, workable, SMTP gem - which there has to be, or there should be because of how easy it is to write, they went with the first thing they saw, and it uses sendmail.

Now I don't have anything against sendmail, but it's the completely wrong tool for the job. They have had to put in a user's name - so all emails seem to come from one person, as opposed to the person doing the activity. It's just a piece of junk, and for good reason - it's the wrong tool for the job!

But I have to make my laptop work with this. It's not running by default, and the reason is that it's the wrong tool for the job, but that's something I'll take up with them another day. Thankfully, we have a solution:

  sudo mkdir -p /Library/Server/Mail/Data/spool
  sudo gzip /usr/share/man/man1/{postalias.1,postcat.1,postconf.1,postdrop.1, \
          postfix.1,postkick.1,postlock.1,postlog.1,postmap.1,postmulti.1, \
          postqueue.1,postsuper.1,sendmail.1}
  sudo gzip /usr/share/man/man5/{access.5,aliases.5,bounce.5,canonical.5, \
          cidr_table.5,generic.5,header_checks.5,ldap_table.5,master.5, \
          mysql_table.5,nisplus_table.5,pcre_table.5,pgsql_table.5,postconf.5, \
          postfix-wrapper.5,regexp_table.5,relocated.5,tcp_table.5,transport.5, \
          virtual.5}
  sudo gzip /usr/share/man/man8/{anvil.8,bounce.8,cleanup.8,discard.8,error.8, \
          flush.8,local.8,master.8,oqmgr.8,pickup.8,pipe.8,proxymap.8,qmgr.8, \
          qmqpd.8,scache.8,showq.8,smtp.8,smtpd.8,spawn.8,tlsmgr.8, \
          trivial-rewrite.8,verify.8,virtual.8}
  sudo /usr/sbin/postfix set-permissions
  sudo chmod 700 /Library/Server/Mail/Data/mat
  sudo /usr/sbin/postfix start

and with these changes, the system has sendmail running, and the gem works.

I can't think of a more completely wrong solution to the problem, but these guys aren't about the "right" answers - they're about the "magic" ones. They want to just drop a gem into a Gemfile, bundle it, and then have it do all the magic. They'll give it a bunch of configuration, and rather than question the use of such a gem, they'll just completely contort the project to the point that it fits the usage of the unappropriate gem.

It's bizarro programming.

I hate it. I really do.

UPDATE: Funny developments… the gem we're using is called Pony, and can use SMTP or sendmail, you just have to configure it differently. Also, it turns out that sendmail is the preferred way to send emails at The Shop. Kinda odd to me… to put sendmail on all hosts to be able to send simple emails, but OK… I'm a "team player", I'll back off. But what a waste of cycles - there are so much easier ways to do this same thing. Even if we keep the same gem.

Seems the Google Chrome guys have been busy - this morning I noticed they dropped dev 26.0.1386.0 and with it, some great release notes. Seems 'Tanya' is taking over, and she sees these as important. Lots of nice little changes, but what's really important to me is that someone new is in charge, and they care about communication with the people using their product.

Very nice to see!

I told myself this morning that I didn't want to get upset at the folks I work with any more. I told myself it's a bad thing to do - and it never accomplishes my goal of actually getting something done. Never.

Yet here I find myself, some four hours later, fighting very hard not to be nasty to one of the Code Monkeys when he gets in. The story goes like this:

We were looking into a problem late yesterday, and I believed it to be possible that the choice of Rack web server had something to do with it. We were running 'thin' in development, but 'unicorn' in production. So this morning, I wanted to move development to 'unicorn', and see if I could reproduce the problem in development.

I look at the git repo and see that another developer had made these changes already. Well… some of the changes. He didn't remove the 'thin' gem from the gemset, and that needed to be done. And he didn't document how to start the 'unicorn' server other than the fact that there's a Procfile in the root of the project.

There is a place on the GitHub README.md for how to start this, and he hadn't changed a thing. So I figured a little googling might help out.

The only thing I could find was that a gem called 'foreman' uses a Procfile and it looks about the same, but we don't have 'foreman' in the Gemfile, so I can't see how he's using it. Unless he installed it globally and didn't update the docs to reflect that.

And, of course, he's not in and it's 8:40 am.

It's this kind of thing that really drives me crazy. They make a change - commit to master and don't update the docs. They either don't think it's important, or that it's clearly obvious, or that they are just plain forgetful. But for whatever the lack of discipline, they are now blocking me from working. That's pretty annoying.

If it were 10:00 pm, and I wanted to do a little late-night coding, I could see that. It's off-hours, and he's got time to get it ready. But they all know that we have the rules about master, and they all know when I get in. This is just having no discipline about what you're doing.

It's very hard for me to maintain a calm exterior when I see this around me all the time - every day. I don't think a one of these Code Monkeys takes this project seriously, and maybe that's OK for them. But when I'm constantly having to bust my hump for deadlines for this entire group, and getting so little help as this, it's really hard to not get angry at them.

This isn't calming me down… I had hoped that it would, but it's not. Gotta just stop writing then...

This morning I checked and Google Chrome dev 26.0.1384.2 was out with the now common lame release notes. It's like they aren't even trying. But hey, it's fast, renders pages nicely, and it works. In the grand scheme, I can live without the release notes if they keep making good software.

This morning I noticed that the crontab job I had to cleaning up the stale, unused jars that happen when you have a project in jruby or clojrue where everything is in a single jar, and they are all tagged with the GitHub SHA to tell them apart. Actually, it's about the nicest thing about jruby and clojure because it means that you don't have to hassle with directories of files, and it's super easy to have a symlink to the current version and then de-reference it in the script with a little:

  jar="$HOME/dark-magic-current.jar"
  actual_jar=`readlink $jar`

and then in the rest of the code for running the app, use $actual_jar and you can deploy on top of a running system without disturbing it. Restart, the script will pick up the new symlink, and you're in business. Pretty nice, actually.

Anyway, the bug was in the vacuum script:

line 26 used to read:

  for i in "${oldJars}"; do

and I learned that because oldJars was a variable, what I was doing by quoting it was to make that one value for the loop - and not a series like I had hoped. Simply removing the quotes took care of the problem as bash is smart enough not to have thought the space-delimited names in the variable were actually separate commands.

Bash is smarter than I thought. Nice.

Simple fix, and now we are checking all the jars for expiration.

Today I struggled trying to get the same keep-alive going in out production server, but because it's running Unicorn and not Thin, and Unicorn is not an event-driven server, it was a lot harder. To be fair, a lot of this had to do with the problems I was having with the deployment of software to the boxes. The Shop has a lot of progressive tools, but the deployment of packages is not among them. So there were several hours lost there.

But the real problem was figuring out how to integrate an EventMachine into the Unicorn server. Thankfully, I was able to find a simple way to do it:

  Thread.new { EventMachine.run }

So simple, but so completely non-obvious.

With that, it started working and I just continued to struggle with the fact that Unicorn doesn't expose a logger like Sinatra/Thin.

These are not the easy tools that folks make them out to be.

This morning I noticed that Google Chrome dev was updated to 25.0.1364.26 with it's now typically terse release notes. It's loading fast and rendering just fine, so I'm not going to complain, but you'd think that an engineer at Google would have a little more pride in their work than these lame release non-notes. Kinda sad.