What's it all about, Alfie?

I'm in the midst of a new project here at The Shop, and I can understand that it's really new technology, and as such, very little is really known about it. Sure, if you listen to the conference talks, Storm is old news, but put it into production and all of a sudden a lot of people's hands lower in the crowd because it's just so bloody new. I'm trying to make it work.

But at the same time, I'm seeing emails about new distributed systems frameworks -- sounds a lot like what Storm is about, and management is asking for opinions. My initial opinion is pretty simple: Pick one and get good at it.

I'm here, but I'm worried that this place is the exact opposite of "Enterprise Tools" - they are the "Always Shiny". We have a tool for distributed, fault-tolerant, computing - so why are we looking at another? Should we assume that the selection we have is premature, and that based on what we have found, we need something better?

I'm not against competition, but then, you have to allow for the fact that you're going to have a hodgepodge of all kinds of systems in the end, as no one goes back and converts a working production system from one working tech to another, different, working tech. There's never time.

So why the search? Why not just get good at one of the leaders in the space, and then gain the critical experience to be able to really make it work?

I fear the answer is that too many people think the tool is the real power.

Nothing could be further from the truth. I've seen it done over and over again - what might be considered antique tech building some of the most amazing things because the people that used it knew it so well they were able to overcome the problems, and make amazing where a newcomer to the tech would see it as impossible.

I hope I'm wrong. I fear I'm not.

I just saw this on twitter:

A common fallacy is to assume authors of incomprehensible code will somehow be able to express themselves lucidly and clearly in comments.

— Kevlin Henney (@KevlinHenney) September 20, 2013

…and for the first time in weeks it made me want to post something.

I'm so horribly guilty of this that I don't even realize it. When I look at poorly documented code, I think the author was just lazy - because he's as smart as I am - right? Maybe not.

In fact, probably not.

To this day I don't see myself as any smarter than a lot of the professional developers I have worked with. Sure, there are some really junior folks, but I'm talking about the seasoned professionals - those guys that may have been working in the web space for a while, or working on back-end systems, or library builders… they are all just as smart as I am. The only difference, so I thought, between them and me is that I worked so much harder that it was just a matter of effort.

This little gem of a tweet says in 140 characters what I keep missing over and over again - that when you look at really bad code, it's often times more likely that the author didn't know any better, or was using too much StackOverflow, and really had no idea what they are doing. So adding comments to this mess is only going to increase the line count and not really add value to the work.

I need to remember this more often.

This week has been a little stressful for me - I've spent a few days off work getting the last of my things out of the house and into storage, and then signing some papers to sell the house. It's all a necessary part of life, I know, but it's stressful, and so I have to push through it.

What I didn't expect to have to deal with was a broken production app that supports some of the capabilities of the main project I'm on at The Shop. It's not a lot - it's not really looking all that great, but it's really useful to me in what I'm doing, and I depend on it every morning for writing up a status email that I send to the group about the overnight runs.

Anyway, for two days in a row one of the senior developers in the group - a relative newcomer, has broken production. The first day, I was pretty nice about it - just asking him if he checked production once he deployed the changes, and knowing full well he hadn't. The next day I was not as happy, and it started a significant email chain with him, the group manager, and myself about what we should be doing, and the qualities of leadership, in general.

The problem is that this guy was hired to be the Tech Lead of the group, but he's never really lead in a way that I felt worth following. He could certainly command, but that's not how groups at The Shop are run - it's meant to be a consensus of smart guys arriving at a good decision for the good of the team and business. There will certainly be differences of opinion, and our group has had many, but after a good talking session, we understand everyone's position, and consensus is reached. It might not leave everyone happy about things, but it works.

At least it used to.

Now it's not working, and I've tried to give it several months to work itself out. But after the second day in a row where no testing was done after deploying changes to production, I felt it was time to point out that this casual approach to production has to stop. That it's very simple to test when it's in production, and the lack of even the simplest of testing is really a sign of a much larger problem.

I could try to make light of the real problem, but it boils down to attitude. Always does, doesn't it? If you have to proper attitude about your work, then you care about how it's seen by others. You are careful about changes. You watch the details and make sure they are all covered.

Basically, you do a good job. Regardless of the job. Carpenter, Dentist, Doctor, Coder - all are the same. If you take care in what you are doing, the end result may not be perfect, but it's at least something you can defend as being the very best you can do.

In this case, he knew it was a mistake. And to do it two days in a row was - well… really inexcusable. So I pointed out that leadership is an isolated job - it's up to others if they choose to follow you. Command is an entirely different thing, and I think we have a problem with the words and definitions we're using for this position. He may have been hired as the lead, but it presumed that he was capable of doing that job. For me, at least, he can't.

I don't know what will happen. I doubt if The Shop will re-arrange staff to suit me, but it's possible that I can have my project separated to make it easy to not have to face the daily friction of dealing - or in my case not dealing with him. I hope that's the case, but I don't know that they will do that. If not, it's been clear that there are other groups in The Shop that would be glad to have me help them, so it's not all that bad, but it's uncomfortable now, and I've been able to keep it very professional and positive.

What gets me is that the original members of this group would have laughed a bit at the first day, and then roasted him alive on the second. That we have gotten to this point is very sad to me. I miss the old team.

I like what I do - I really do. I like the company I work for - there are a lot of nice folks here, and I generally like the decisions that management makes. But as with every life a little rain must fall, there are times that your time in a group is done, and it's best to move on. The ideas that shaped the group and got it to this point were necessary and good, but now it's time to let someone else take over and take it from here.

Of course, that's not how it feels.

It feels like the new folks to the business think they have a monopoly on the project even though they just joined the company. It feels like they have no respect for the ideas the project was built on, so that their changes to the codebase make no sense, and in fact are counter to the goals that the project was built on.

It feels like they are being jerks.

And who knows… maybe they are. Maybe they aren't. It's not only impossible to tell, it's also completely unimportant. You find yourself in the minority and it's time to move on. No anger, no grief… maybe a bit of sadness for what's been lost, but loss is part of life. You can't allow the project to be what the new blood wants it to be - sees it to be in their minds, if you're there holding them back.

It's also not really fair to just sit in the group and allow the changes to occur around you. That's just gold bricking. Yeah, you know the code, yeah, you like the project, but it's all going in a different direction and it's time to just cut the cord. Allow the project to be what it will be under their stewardship.

It's time for me to move out of this group. As much as I'd like to keep working on what I'm doing, it's not good for the group or me.

It probably shouldn't be surprising to me at all that the event that got me back to writing is arrogance of Code Monkeys. These are the young Rock Stars of the community that think that Ruby is the only language you need to know, and everything else is so much less that it's hardly worth their time. Bash, C++, Python - all "toy" languages to the Code Monkey, as the One True Language is ruby.

Of course, they are pumped up by people telling them they are amazing. They solve a few problems by throwing hardware at it, and they think they are the Oracle at Delphi of everything software, and as long as they can do just a little bit more than the next guy, their confidence and assurance grows. It's kinda sad.

I can remember experiencing several humbling experiences in grad school, and for those I am eternally grateful. I have no desire to ever be like these guys, but I can realize that I probably was back before grad school. But that was a long time before I got out into "The World". By then I knew… I was not all that. I was just a hard worker.

So this came to mind this morning I looked at a failed job in a cron email. I sent the guys responsible for the job an email saying they might want to up the memory for the JRuby JVM, as that was the cause of the failure. The Code Monkey responsible for the job said that he upped it for the run, but didn't change it in the code in the git repo because it wasn't needed.

Now in my mind this isn't possible. If the code is a one-time (a.k.a. throw-away) script, or code, then it's not in the repo as it has no value past it's one use. But if it has value past it's one use, then it's in the repo. If it has value, then it should be fixed for the memory problem. So he's either wrong in putting it in the repo, or wrong for not updating it. I know this, but I try to be nice and ask him "What's the harm in updating it?"

His response I could have guessed: "It's done, there's no reason to update it." Spoken like a true Code Monkey. They hate comments. Why? All ruby code is self-documenting. What they really mean is that everyone should get used to re-reading and re-understanding the code every time they approach it. Documentation just gets in the way of that constant process of self-re-learning. He had no belief that the script would ever be used again, but that's because he's got the time horizon of two weeks. And because he can't see it, it must not be needed. Done.

So I'm going to let it go. He's convinced he's shown me a thing or two. In reality, he's shown his inability to be a tech lead for me. That's his position in the group - Tech Lead. Not a chance in the world. I respect my boss - he's a sharp guy, and I respect his skills. But if he really thinks this guy has serious skills, then he's fooled. But hey… anyone can be fooled - look at me - fooled for 27 years.

Anyway, it's just amazing to me that these Code Monkeys are as plentiful as they are. Sure, I knew plenty of C++ coders that weren't any good. I knew plenty of Java coders that weren't any good, either. But it's the consistency of these Code Monkeys that's really throwing me for a loop.

But you know… maybe that's a good thing. After all, he got me to write something. Maybe I'm making a little progress?

Well, it looks like the major version just jumped with Google Chrome dev to 27.0.1423.0, and it looks more like a semantic change than a real significant update based on the release notes. Still… it's nice to see that they are still on their schedule of moving things along as they promote from dev to beta to release.

This morning I noticed that Google Chrome dev 26.0.1410.12 is out with what appears to be a good set of updates from the release notes but the minor version update indicates that they see this as minor bug fixes. Interesting take on how they see these changes. In any case, the new maintainer seems to be generating good release notes, and I'm all for that. Maybe the quick succession of bug releases is signaling a shift to 27.*… we'll have to wait and see.

Today has been a lot of skut work - clean-up stuff that has been sitting in the queue for months but no one wants to do. But if the project is going to really work, someone actually has to do it. So since I finished up a lot of tasks today, it seemed like a natural thing to just get to it and clear the decks.

None of this is hard stuff, it's just not very fun, and it takes time.

First off, I followed up with a request for backups to be made of all the database machines we use in the group. This includes CouchDB as well as PostgreSQL. It's nice in that the install of each of these packages places the data files in the largest partition on our boxes: /var/groupon/ so it's simple to just back up that partition. I submitted the request a few days ago, but hadn't heard anything back, so I followed-up asking if I was going to get a completion notice when the backups were working.

Response was: "Yup, likely tomorrow". Good enough.

Next, we needed to get Nagios monitoring of the free disk space on the boxes as well - so that should a process go crazy and start to fill up the disk, we can fix it before it becomes a database killer. This has happened to us on several occasions, and it's something to be avoided as the main processes can't run if the database is offline.

Finally, needed to do what I could to compact the CouchDB databases on the production and UAT hosts because we're at 93% disk space used, and there's very little headroom left. If the compaction of the views doesn't work, then I'm going to just drop the database and start fresh. We have a replicate of the production data, and with the backups (above) we'd be able to go back to it anyway. But this is something I'd rather not do, but it's certainly a sure-fired way to get the space.

It's not glamorous work, but it needs to be done, and no one else is picking it up, so I might as well just do it all and have it done.

This morning I've been fussing with MacVim and Sublime Text 2 and trying to come to terms with the fact that Sublime Text 2 has crashed on me a few times, and I have no desire to go through that again. I have downloaded the beta of Sublime Text 3, and it appears to be decent - though because it's using Python 3 virtually none of the packages I use in Sublime Text 2 are going to work properly, but at least I get the default behavior, and that's about 95% of what I use day-to-day anyway.

So I decided that it's time to give it a shot, and I switched from Sublime Text 2 to Sublime Text 3 on my main MacBook Pro. This isn't my work machine, but my machine, and as such it's not getting the same number of keystrokes as the work machine, but still… as I work on it I'm hammering on ST3, and hopefully, they will get it out of beta soon, and the Package maintainers will upgrade their code to be compliant with Python 3 and I'll be able to get back all the functionality I have/had in the older version.

Among the new features in ST3 is the extensive use of C++11, and that includes the move semantics which should make for a much more efficient app. I'm hoping to see that the crashes I've had go away, and if so, I'll be more than happy to foot the bill for the upgrade price.

This isn't to say I've given up on MacVim - I just read a message that the maintainer is looking to add the window/file/tab restore feature to it, but that it's "hard", and so I'm not expecting it anytime soon. Should that come to pass, I think I might have a much harder time choosing an editor. Yeah, it'd be tough not to use MacVim if it had that feature. It's such a powerful editor.

Today I decided that I wanted to see what was out there to update MacVim to be something more like Sublime Text 2 - or at least how I use it. Since starting at The Shop, I've come to really like the project-management features in Sublime Text 2 - specifically the ability to find a file with partial filename completion, and the same thing for methods/functions, as working in Ruby has shown me that the standard way of working on Ruby projects is to have a mess of files, all no bigger than a thimble.

But I had a feeling that there had to be something like this for Vim - and therefore MacVim, which I really like - but just need this kind of help with. So I asked my most Vim-savvy friend and he mentioned CtrlP.

Install this guy and a simple Ctrl-P brings up a searchable list - just like Sublime Text 2! This is directory-specific and includes most-recently-used files and buffers as well. It's a real treat. It's fast, clean, and does exactly what I need it to do. I've never heard of it, but it's really quite amazing.

The next real jewel today is VimClojure. This gives me the clojure syntax-highlighting and indenting that I really need to write in clojure in Vim. I'm doing about 90% of my coding in clojure these days, and the lack of a good default mode and syntax highlight for clojure is really something that kept me from really even starting to write clojure in Vim. With this, I'm golden.

I'm super happy with these two, but I really wish MacVim had the save/restore that other Mac apps have for windows and files. The ability to save and restore the state would be amazing and might make it so that I'm back to MacVim all the time. Sublime Text 2 is nice, and I've played briefly with Sublime Text 3 beta, but there's something about Vim and all the history I have with it that makes it exceptionally powerful for me.

I'll keep looking, and these plugins make Vim much nicer, but I'm not yet giving up on Sublime Text 2 - yet.