What's it all about, Alfie?

Today has been one of those days when it took a lot of effort to stay motivated just to keep going. When the highlights of your day are getting a loaner laptop and verifying that you can run a Java WebStart app as yourself - so as to prove to Bloomberg that we're using their API correctly, then you know it's a tough day.

I'm trying to get everything ready for the next steps in the validation of our usage of the Bloomberg Server API, and there's a lot of waiting. Waiting for Bloomberg to tell us what the requirements are for the next tests... waiting for the people involved here to get the software installed and the accounts created... waiting, waiting, waiting.

About the only technical challenge today was in trying to track down where some trades/ammendments went wrong - and only that was to answer a few questions as the folks came and asked. I wasn't directly asked to help in the search, so my involvement was really quite limited. But that's all there was today.

It's important to stay motivated so that when you are needed, you're there, ready to respond. But when dealing with external vendors and other folks, it's wise to set your expectations properly, for they won't see the need to respond as quickly as you do, and you're just going to have to wait.

I was talking to an old friend this weekend - it had been ages since we talked and we spent a lot of time on the phone getting each other up to date on what's been happening lately with the families, etc. It's fun to play catch-up with an old friend as they know a lot of the old jokes, but a lot of the neat, new stories are new to them, and it's always fun to tell stories about the kids.

Anyway... after a while we got to talking about work, and he has been going through a really tough patch at work where the place is not doing well, and the morale and attitudes of nearly everyone there is really taking a beating. People are slacking off, not doing their best - and the work shows. But I know my friend, and that's not like him - nor is quitting as he's big into the loyalty, commitment, etc. But he had a recent story to tell that even made me cringe.

He's been doing all that's asked of him for years, and recently, because this other group of developers is really not pulling their weight (I'd say they were lazy and inept, but he's nicer than that) he's been asked to do a little of their work because they are too busy with other things. Well... he does it. In my experience, it's worth lodging a complaint - respectfully, to let your manager know that this is not 'ordinary' - no more so than them asking you to be a janitor or security guard, and even though "The Team" needs you to do this, will the "Team" be there when you need them? Not bloody likely, which is my point about lodging the complaint - respectfully. But I digress.

He's been forced to do this work of the others and even on occasion told to get it done by deadlines when the principals are out of the office. It's like "Hurry up and get this done for Steve by Friday" and then Steve takes Thursday and Friday off. Why the rush, if they aren't going to be in the office? It seems reasonable that if the work needs to be done that fast, that he'd be in the office. But maybe it doesn't need to be done that fast, it's just the manager's way of trying to suck up to the guy.

This is why I really don't like these kind of 'rush' jobs - many times they really aren't a 'rush', they are just a rush from the point of view of the manager looking for some brownie points with their management. Which, again, is the reason I always try to make a respectful point of properly setting expectations.

In the end, my friend is unhappy, but he's resigned himself to the fact that this is the job. Much like any relationship, it takes work, and it's a given-and-take with a natural ebb-and-flow that means that sometimes it's not a lot of fun, and sometimes it is. I can certainly understand his point - having been married more than two decades, but there can come a point when a job - or an relationship, for that matter, is destructive for one or both parties. There are tons of times when people aren't fitting in at a job and are 'let go', or 'right-sized', but the other side of this same relationship happens as well. Jobs and managers can get overly oppressive, thankless, demanding, and in general, one-way. When that happens, it's time to step back and realize that there are times to sit out the ebb and flow, and yet there are times when it's really best to step away and decide if it's really a good idea to stay in the relationship.

I'm not going to tell him to get another job, I respect him too much. I know he's a good guy and when it's the right time to make a move - or re-set expectations, he's going to do the right thing. But it's hard to see a friend having this much trouble, when he's such a nice guy. Sigh.

Over the past two days I've been forced back into a project that I have been out of for a very long time. There's a need to get something done quickly, fixing a long lost (and wished forgotten) developer's code, and so I get pressed into this. Whatever... it's not what I wanted to do, and I stated that I was probably not the right guy for the task, but was forced into it anyway.

When I originally started this project - Oh so many years ago, my partner and I decided to make it cross-platform: Windows and Linux, IE and Firefox, because you never knew, and maybe someday we'd move to Firefox. So we made it that way. Things progressed, he left, another guy came, I gave the project to him, and I moved on.

Now I'm back, and I'm very surprised at what I'm seeing. Gone is the cross-platform capabilities - it's IE only now, Baby! It's not that hard to make it cross-platform, it just takes a little bit of effort. Why they didn't put this in, I can only guess - like most people, they all are looking for the easy way out. Since they don't have any users running Firefox now, they don't need to worry about it, so don't even spend a second thinking about it.

Problem is, it only takes a few minutes to get things so they work on both platforms. It's not that hard. I had to change a little Javascript to get the applet tags written out properly. I added a little compressed string code to the applet writer to make it faster to download and pretty much bing! it's done. Not that much to it. But you'd have thought it was weeks of work that just weren't worth it.

It's always worth it. Forget that someday you might actually need it, there's value in making something a little more generic than you need. Just in case you want to see if it's the platform or your stuff. Isolate the problem or variable. Easy. But I guess they are so wrapped up in IE they don't even care to look outside that box. I understand the logic, I'm just disappointed.

It's like the line out of Illusions by Richard Bach - Argue for your limitations and sure enough, they're yours. It's as if they wanted to make the system less capable. They aren't bad people, just not really making any effort to see beyond their immediate horizons. It's a waste.

I'll finish this page and then bow out and hopefully not have to deal with it for another several years. It's just not fun to come back to a system you had pleasant memories of to see it's all decayed and in need of repair. Sad.

I've been working pretty feverishly the last several days on getting my market data server updated with a new API from one of my data vendors. It's been a significant update because of the fact that in this new API they have implemented (and forced us to enforce) login validation and data permissioning. I don't begrudge them the validation and permissioning because it's their bread-and-butter, and it's up to us to decide if it's too much to deal with, but it's the way in which they are doing it that's a real pain.

Based on experience, exchanges charge fees to see their data. Seems reasonable. But they don't really care if you see 1 instrument's price 1000 times in a day, or 1000 instrument's price 1 times a day - or even if you only see 1 instrument's price 1 time a day - it's still the same price. So the permissioning for standard exchange fee-liability is by exchange. Very reasonable.

Given this, we should be able to have a list of valid exchanges for an individual user and then check each instrument for it's exchange and be done. It would mean that we'd have on the order of a few dozen exchanges per user - a very manageable set. But this vendor isn't like that - they want to be able to know exactly what instruments you're asking for, so you have to permission the data at the instrument level. This, too, isn't horrible, but it does mean that instead of having a few dozen things to track for a user, you now have upwards of a few hundred or more. It's still possible, just a lot more work to do with each request.

Then there's the way they do the permissioining... a user has a list of entitlements, and each instrument has a list of entitlements and we need to find a common entitlement between them for the user to see the instrument's data. Why not have a single ID for a user - or an instrument? Why have lists? I'm sure you can have the degenerative case of one item in a list, but from the looks of things, that's not the way it's going. Just really odd. Not impossible to work with, but odd.

So it feels good to be out from under that weight. It's all working, and we'll be able to move to the verification phase with the vendor (certainly understandable) and then on to the roll-out to production.

This morning I found an old co-worker on chat - Pete E. Pete used to work right next to me but then moved to another division and then to another Bank. Now he's in the middle of a six-month stint in Singapore for Morgan-Stanley. Amazing. I haven't talked to Pete in at least 3 years. We spent a little time catching up and I really liked finding out what he's doing and where he hopes it's taking him. Computers and the 'net really are making communication so effortless that the world has changed, and I was in the middle of it and didn't see it really close-up until today. Wow.

I was talking to a friend today that is looking for a new position. He's interviewing at a place that uses online tests as a measure of the skill of a developer. Problem is, he and I know that the ability for an online test to predict your ability to program is virtually zero. He likened it to having an online test for tending a garden.

Oh, he'll take the test, but the fact is that he's probably not going to perform on the test to the same level that he can perform in person. It's an entirely different kind of knowledge. That's not to say that online testing can't be done well, but it can't give you a good measure of a large spectrum of individuals. Even the best tests can't do that if they don't have some real human being interpreting the results. Multiple-choice tests are only able to really discriminate a very narrow band of skills.

For example, if you have a simple test - like a driver's license exam. That's good because it's measuring a minimum level of competency. You aren't called upon to make judgement calls in these tests (like you are in real driving) - that's left for the driving part of the exam. But it does register well your level of basic understanding of the laws governing the use of an automobile.

But let's see if that works for programmers.

Can you generate any multiple-choice exam question that can even remotely test a person's ability to 'use' a language? You can test if they can spot a syntax error... or what the result would be if this code was executed... or even what would be the proper form of a certain statement. But you can't really tell if they will be a good developer.

Mastery of the basics of the language is possible - do you know what the basic data types are, that kind of stuff. But what happens if a person hasn't used a certain class library - or a certain feature? Is it essential to mastery of a language to use every feature? I can't think of a language outside of assembly that I've used every feature of. Nor do I think that many really can. Maybe a compiler write could, but then that's all they are writing - tools that turn code into machine instructions.

But you can't really tell if one developer is going to be faster at implementing new features or bug fixes than another. Or if the designs one guy will create are going to have longer staying power than those of another person. Yet these are the essential questions of a developer's worth. How quickly can they add features and fix problems? How does their design stand up to the test of time? Knowing that they know the difference between an STL set and vector is not nearly as important.

Yet that's the only thing these online tests can really point out.

I would like to see more employers asking for code samples. The problem there is that almost every company I know of late would not want even one coding example shown to a potential competitor for fear of losing the 'edge' they felt they maintained. Artists of all kinds do this every day - it's their portfolio. I'd like to see developers build a portfolio as well. Show it to someone - if they are in the business of writing code it's not going to take them 15 mins to see if the code you've written is good, fair, or poor.

Do you have comments? Is there a basic style to the code? How are errors and exceptions dealt with? What level(s) do you take with each of these? It's all there in black and white if you look at the code. Even moderate to bad coders will be able to recognize good code.

I'm sure my friend will do fine. I've known him for more than twenty years and he's a good coder. If the test is even close to measuring what it claims to measure, then he's going to do fine. It's just interesting that as two ex-teachers with a lot of love for a good test, that we got to talking about this upcoming test of his today. Good tests are hard to write and even harder to grade. Anything online and multiple-choice is dodgey at best.

Today was one of those days that seemed to drag on for a very long time. The progress was there, but it was very slow at times. There was several hours of refactoring on some requests to the server to make them fit the style of the other requests, and then the addition of another request type was similar to the last, but different enough to make it hard work.

It's just been one of those days that you feel drained at the end. But you can look back and say Yeah... I'm glad it's over, but I did make good progress. The details are everything, and I'm glad to say that before I leave today all the details of these new additions are taken care of. The clients have the code, and it's working. The logging in the server is up to snuff. The code looks good as well as running well. All the details that make this a good project are there.

But I've left it all on the keyboard, as it were. Gotta go home now and get some rest.

Just now I realized that I hadn't plugged in my external FireWire drive, which is my TimeMachine drive, to my laptop yet this morning, so I plugged it in and in a few seconds it started cranking on the backup and everything was happy. And this lead me to realize that the upgrade for 10.5 was really quite marvelous. I have used TimeMachine probably less than a dozen times, but each time it's saved my bacon in a way that I didn't used to have. I've been able to create new machines from old dead ones... I've gotten back single files, and I've reached a level of relaxation with regards to the data on my laptop that I simply never used to have.

I've tried several programs, and ended up rolling my own eventually, but I didn't really keep up with it properly, and so I always had some exposure to loss. Now - it's only an hour. That's pretty impressive. It's really taken my data security to a new level. Thanks, Apple. Well done.

Today has been a very heads down coding day where I've spent almost all day getting the new features of the Vendor's new API working in the server and clients of my market data server. I was originally going to have the client-side code handle the authentication as I've done that in the past with pretty good success. But I know that this vendor is not really big into speed on his responses, so I was hoping to be able to build this all on the server-side and then cache the positive results for a day so that subsequent requests for the authentication would not hit the vendor's back-end systems all day long.

I was happy to see that they had what I needed - even if it took a little poking and prodding to get it to work, and know what data I needed to pass to it. In the end, I was able to incorporate it as a regular server-side feature which is going to make the cache a very welcome addition. Not the least of which is that we no longer are limited to their client-side code and response times.

With all this put into the market data server, I'll start tomorrow on the authorization - or the exchange-based filtering of the data. Again, I'm hoping to find it in the server, and I have a few leads to chase down before I'm sure, one way or the other, that I can or can't do it. It sure would be nice, though.

We have been forced into updating a Data Vendor's API because they never really envisioned that we'd use their desktop product in quite the way that we had - which is to say, built a server around it and made the content accessible to a large number of applications for the traders. So, when they saw what we did, they decided to make that unsupported, and brought out a product to be used in the same manner - for considerably more money. Surprise!

So, we had to upgrade, and in this upgrade, they biggest changes are in the authentication and permissioning of the data. It's understandable, they need to protect the data from unauthorized eyes, etc. The problem I'm running into over the last few days is the way they are having me (and all the other API users - all eight of us) do this authentication. First, everyone has to have a login to the service. Seems that's a good place to start and end. But no. They have an underlying 'ID' - that's guaranteed unique. Seems that's a good place to end then. But no. The minimal authentication is done in their Java client with an ID and the IP address of the machine they logged in on.

Now it seems to me that they already know this, and all they really need is the login. The login leads to the ID, and the login tells them the IP address of the machine they are logged in on. The way they have done it is doubly redundant. That's taking a bit of doing, but it gets even better in the C API.

In the C API - if I'm reading the documentation correctly, you need not only the ID, but four other pieces of information that are known by the service but I have to provide and have no way of knowing programmatically. I have to have the user go to a page on the service's application, read off four multi-digit numbers and then feed them to the C call.

That's an embarrassment. I can't believe they think this is a good scheme. I'm hoping that these four are redundant, or unnecessary, and the docs are simply out of date. But I have a feeling that they aren't and this is just the "way things are" with this vendor. I'm really shocked at this because I had a pretty decent way to have my market data server incorporate this into it's core so that data coming out of the server could be filtered based on the requested user. That would be wonderful as it would put all the permissioning behavior into one place which makes it far easier to control.

We'll see what I get back from one of their "API Experts". I'm really really hoping I'm wrong here.

UPDATE: I got a little help from one of the experts, but for the most part I have to say that getting bad information is worse than getting no information - which was the case today. I asked for the parameters I needed to pass in and got back a list that wasn't right. Once I figured out it wasn't right and what might the right set be, I was well into the afternoon and yet I did get it right. Then I had to realize that their idea of returning a negative was returning an error and not a negative at all. So I had to code for that as well. It's amazing... the people who created this additional functionality didn't even study the old style of doing things and it makes for a very clunky interface to the system. It's working, but as I had expected going in, it's not easy and it's a pain to make it look pretty.

Tomorrow I need to wrap this authentication into a request class and then integrate it into the market data server and finally add caching on the results so that multiple requests in a day don't need to each go to the service to be resolved. Once a day is going to be enough.