An Online Test for Tending a Garden
I was talking to a friend today that is looking for a new position. He's interviewing at a place that uses online tests as a measure of the skill of a developer. Problem is, he and I know that the ability for an online test to predict your ability to program is virtually zero. He likened it to having an online test for tending a garden.
Oh, he'll take the test, but the fact is that he's probably not going to perform on the test to the same level that he can perform in person. It's an entirely different kind of knowledge. That's not to say that online testing can't be done well, but it can't give you a good measure of a large spectrum of individuals. Even the best tests can't do that if they don't have some real human being interpreting the results. Multiple-choice tests are only able to really discriminate a very narrow band of skills.
For example, if you have a simple test - like a driver's license exam. That's good because it's measuring a minimum level of competency. You aren't called upon to make judgement calls in these tests (like you are in real driving) - that's left for the driving part of the exam. But it does register well your level of basic understanding of the laws governing the use of an automobile.
But let's see if that works for programmers.
Can you generate any multiple-choice exam question that can even remotely test a person's ability to 'use' a language? You can test if they can spot a syntax error... or what the result would be if this code was executed... or even what would be the proper form of a certain statement. But you can't really tell if they will be a good developer.
Mastery of the basics of the language is possible - do you know what the basic data types are, that kind of stuff. But what happens if a person hasn't used a certain class library - or a certain feature? Is it essential to mastery of a language to use every feature? I can't think of a language outside of assembly that I've used every feature of. Nor do I think that many really can. Maybe a compiler write could, but then that's all they are writing - tools that turn code into machine instructions.
But you can't really tell if one developer is going to be faster at implementing new features or bug fixes than another. Or if the designs one guy will create are going to have longer staying power than those of another person. Yet these are the essential questions of a developer's worth. How quickly can they add features and fix problems? How does their design stand up to the test of time? Knowing that they know the difference between an STL set and vector is not nearly as important.
Yet that's the only thing these online tests can really point out.
I would like to see more employers asking for code samples. The problem there is that almost every company I know of late would not want even one coding example shown to a potential competitor for fear of losing the 'edge' they felt they maintained. Artists of all kinds do this every day - it's their portfolio. I'd like to see developers build a portfolio as well. Show it to someone - if they are in the business of writing code it's not going to take them 15 mins to see if the code you've written is good, fair, or poor.
Do you have comments? Is there a basic style to the code? How are errors and exceptions dealt with? What level(s) do you take with each of these? It's all there in black and white if you look at the code. Even moderate to bad coders will be able to recognize good code.
I'm sure my friend will do fine. I've known him for more than twenty years and he's a good coder. If the test is even close to measuring what it claims to measure, then he's going to do fine. It's just interesting that as two ex-teachers with a lot of love for a good test, that we got to talking about this upcoming test of his today. Good tests are hard to write and even harder to grade. Anything online and multiple-choice is dodgey at best.