What's it all about, Alfie?

I've been working with this software package for months now, and while I can't use the name for fear of lawsuit, I think I'll borrow a page from J.K.Rowling and call it The Software Package Which Must Not Be Named or maybe just You Know What for short. OK, it's silly, but here's another silly lesson to be learned from this million-dollar fiasco: Sometimes, if things are slow enough, they will be technically working, but the users will think them broken.

Case in point with You Know What was a certain instrument's trade ticket. There were supplemental data fields on the form, and they appeared not to populate after being programmatically set on the committal of the trade. I was testing this very thing this morning and after seeing it happen in front of my own eyes, I reached for my pencil to make notes of the fields I'd populated, and the ones that hadn't been correctly populated with the calculate values.

After a few seconds of writing the notes, I looked up to see the empty fields and amazingly, they had been filled. I'm convinced that this was what was happening all along. The way in which this system is wired up is to save to one database, replicate all changes to another and then reach from the replicant. If the changes went in programmatically, then the GUI might not have known to update them, and so they had to go through the replication chain and that was going to take several seconds. It might just be that the users/testers were impatient, and when they didn't see the data in these fields immediately after the saving, they assumed the fields weren't going to be populated and closed the window.

So... this morning I'm going to sit with the tester and make them wait to see if the fields don't appear. I figure that it's about a 30 sec wait, and if they show up for him like they did for me, then we know it's not a bug... it's a feature. Unfortunately, this is a nasty feature, but there's no helping that short of a significant reconfiguration of the system. Don't know if and when they might happen, but if it does, then this delay should lessen. But we'll have to see.

For now, You Know What seems to be OK. You just have to be patient. I wish I felt that described more of the users, but I don't.

I was talking to a few guys around the office today - specifically about how they had not passed a test in their series of certification exams that several more people need to take around this place. I've written about the evils of these vendor exams - written by people that aren't professional educators, graded in the most cursory manner - computerized for the most part, and the resulting problems this makes for people in the workplace that need these certifications for their jobs.

But that's not what I'm writing about now. No, this is something I hadn't realized I'd picked up along the way to where I am today. I'm talking about understanding the difference between "testing" and "fact checking".

What I mean by the first is that anyone, at any time can make a test for someone. We do it all the time as parents ("Did you knock this over?"), we do this as developers ("Is this the problem with the system?") and as companies - in the form of these certifications. They aren't fair, don't really concern themselves with "right" and "wrong" - they are questions, challenges, and tasks that we must overcome to someone else's satisfaction in order to achieve what they have promised we'd receive.

"Fact checking" is all about the "right" and "wrong" - Can a 3kg spherical lead ball dropped from 1m crack a 1cm thick piece of glass? That kind of stuff. It's all something that that is by it's very nature objective. And for the most part, dispassionate.

"Testing" is all about someone's passion - I guarantee it. The person that wrote the material the test is based on... the instructor... the test's author - someone in this chain cares a great deal about this test. And as such, it's not about you, the test taker to tell them what's "right" and "wrong" - they already believe they have a firmer grasp on that than you - which is why you're taking the test and they are administering it.

What surprised me today was that I knew all this, and my co-workers didn't. Then I got to thinking about it and I realized where it all came from - grad school. I have to hand it to grad school, it really taught me this lesson. You sit in a class of less than a dozen grad students, and the professor knows you all by name. He's spending a lot of time trying to make this material understandable to you all, and so there's a lot he's got invested in the exams (tests). You need to make sure that when you take the tests, you give him what he's looking for. Period.

Maybe the university is an easy place to learn this, I don't know. But I do know that I have learned it, and I don't think of myself as cynical on this point. I think of myself as recognizing the concerns, issues, and feelings of the people involved. There's nothing to gain by making the test so simple it's useless as a gauge of the material retained. It's also equally useless to make it so hard that it's nearly impossible to pass - no one will pass it and therefore it'll have no value.

So there's a compromise there - an optimization of sorts. Again, grad school helps me there more than I initially realized. It's taken me 20 years, but I'm really starting to look back fondly to that place. Who knows... maybe we'll take a road trip and see the campus.

I've been in a lot of different business situations in my life - at the bottom, in the middle, at the top, big place, small place, conservative place, educational institution - and still I have to say that I'm amazed when I run into a boss that seems to need to have his people agree with him on a course of action.

Case in point: several years ago I was building an interface to a new ticker source from a group in the larger bank. There was an issue about what data to retain, and I thought that the existing model was fine, but a manager (not mine) in the Shop decided that we needed to retain more - specifically, an additional price.

He explained it and while I understood that I would be forced to do it, I wasn't about to agree that it was a good idea - it wasn't. But that didn't matter. If I had wanted to stay in a position where I was the one making the decisions, I'd have stayed there. I moved for a reason, and this was one of the things I knew I'd have to live with, and I was fine with it. The problem was, he wanted me to think it was a good idea. He just kept at it, and at it to get me to see this as a good idea.

Finally, my manager said "Give it up... Bob's going to do it, but he's not going to agree it's a good idea." He understood. Good.

Today was right along these same lines - even the same manager was involved. They asked me to coordinate all the development of a particular feature. I got all that tested and coordinated the two other guys and they said their stuff was done and tested. So I told one user to kick the tires and see what they thought. Interestingly, there was a difference in the two ways this package was executing the code, so there was a difference to the user. Nothing big, in my mind - we'll get it fixed as soon as the vendor's rep gets back to us about how to make them work the same.

But this manager told everyone we hadn't completed testing on the additions and to hold off testing. Seemed a bit excessive to me, but he's the manager, so I simply wrote back - 'Hey, we tested it, didn't find this, so we'll fix it.' And then the "please agree with me" chain started.

Over the next hour, about half a dozen emails went back and forth about the true state of the testing. I tried to explain that we did test it. But if he wants to be the final tester before anything is shown to the users, then just say so. Yet he continued to want to get me to agree that we hadn't tested it properly.

OK guy, get over it. You want to be liked, then be liked. You want to be the Boss, then be the Boss. But don't expect to be able to be the Boss and have everyone like you. You asked someone (me) to make the call, and then didn't like the call. OK, make the call yourself, but don't expect me to say I made a bad call... I just made one you wouldn't make.

Yikes... just be a bloody manager and say what you want.

I've been working with a system today that has nearly driven me to drink with it's hidden features. I know each complex system will have them, but when you are paying millions of dollars for a serious system, you should have as few of these hidden gotchas as possible. I submit that the higher the quality of the product, the fewer the number of hidden gotchas it has.

Take a good messaging system like IBM MQ or Sun ONE MQ. Both have serious price tags, both do a serious job, and while I may not like the configuration of one, or the other, I have to say that there are so few hidden gotchas in these two products as to really make them almost perfect. Ideal? No... but they do exactly what they say with no ambiguity and very little learning curve.

In comparison to the product I'm working with. They have a feature where you can define supplemental fields to the existing fields (attributes) to the stock objects in their system. You can add strings, doubles, dates, etc. Seems reasonably flexible, until you have to actually have to use them.

Say you have an object that has a built-in value (attribute) called 'date', and you want to add a supplemental field for 'expiration'. You can define the 'expiration' field just fine - as a date, even. You can then use the fields like:

  print obj.date
  print obj.expiration    // <- WRONG!!

You cannot "just use" the field you've defined even though the system has all the data it needs to work this out. No, you have to read it through a special method:

  print obj.read_supp('expiration')

and writing is even worse. You have to scan through all the supplemental fields on the type of object and see if there's the one you want, and then scan the object itself for this 'id' as you can't look it up on the name.

In short, this is an enormous pain, and there's even more. Even if you define a supplemental field as a double, you have to store it as a string! All supplemental values are going to be strings. The only advantage the 'type' has in the definition is the input checking in the GUI. It's a joke. I'm stunned.

In a good product there aren't any gotchas like these. Things work in a similar and symmetric manner. Sure, it takes more work on the part of the developers, but that's what it takes to have quality software - work. I'm continually amazed that this product sells any copies at all. (which is why I'm not referring to it by name, clearly)

If I"m lucky, this will die soon enough.

Because the chat infrastructure team seems to be unable to get their issues resolved about the MindAlign bots, and because I can't really honestly say that the bots I need are business critical, I've started looking for IRC servers that I can start up on my own, point all my servers at, and then simply have my own IRC system. The load is trivial, and with this, I'd be able to get things done, but there's always the risk of being tagged as a rogue chatter.

Given that it's been two weeks, and there's no accommodations being made for important but not critical bots, I'm thinking that even if I get nailed (and that's not a guarantee by any means) that my excuse is exactly that - two weeks, no communication, and no alternatives. Sounds good to me, anyway.

So I've been looking at the IRC servers, and it seems like IRCD-Hybrid is a decent tool, and the history of it leads me to believe that it's really different only in the very 'far out' IRC issues - things I'm not liable to hit ever. I just need a basic communication hub, and for that, this should do just fine.

I'll get it, build it and see if I can get it going on a box of mine. If I can, then good enough and I'll try running it there for a while. I'll convert a few processes over and then see how that flies for a few days. If all looks OK, I'll convert over simply to await the day that the MindAlign guys get things worked out.

UPDATE: so I got the code and configured it by reading every single line. And boy, oh boy... you have to read every single line as there's a line in the config file that will crash the server intentionally if left in the config file. So, read and configure.

The big change was that the default IRCD-hybrid 7.2.3 does not allow for NICK names to start with an underscore (_). Since all the bots for MindAlign have to start with that, I needed to get into the code and fix it. The relavent code snippit was originally:

  int
  valid_username(const char *username)
  {
    int dots      = 0;
    const char *p = username;
 
    assert(p != NULL);
 
    if ('~' == *p)
      ++p;
 
    /* reject usernames that don't start with an alphanum
     * i.e. reject jokers who have '-@somehost' or '.@somehost'
     * or "-hi-@somehost", "h-----@somehost" would still be accepted.
     */
    if (!IsAlNum(*p))
      return 0;
 
    while (*++p)
    { 

and needed to be changed to:

  int
  valid_username(const char *username)
  {
    int dots      = 0;
    const char *p = username;
 
    assert(p != NULL);
 
    if ('~' == *p)
      ++p;
 
    /* reject usernames that don't start with an alphanum
     * i.e. reject jokers who have '-@somehost' or '.@somehost'
     * or "-hi-@somehost", "h-----@somehost" would still be accepted.
     */
    if (!(IsAlNum(*p) || ('_' == *p)))
      return 0;
 
    while (*++p)
    { 

After this change, a simple recompile, and the bots were working great. This would be a nice change to the codebase to put this in the config file, but I can see that this is probably pretty stable, and so they might not want to mess with it.

In any case, I'm set now with my own IRC server for my apps. If they come after me, I have a solid defense - it's for the best for the business.

It's been a rough couple of days, as recent posts will attest. Plain and simple, it's been bad. But this morning I turned on my iPod and started listening to some of my music, and things started to seem not so horrible. It's an amazing thing with music - though I'm sure some research has been done to map the pleasure centers of the brain activated by music, I just know I feel a lot better when I listen to nice music.

I used to be a straight album (CD) guy - start an album listen to the whole thing. It's a carry over from the days when that's how music was delivered and that's how you played it. But a few years ago a good friend really got me into the playlist capability of iTunes, and it not only allowed me to stitch together my multi-album sets into complete 'volumes', it took a while but I started to really mix up the music and make real playlists.

Took a while, but what a difference. I used to run iTunes only when I need to rip a CD (I always bought CDs) - now it's running all the time pulling in podcasts and holding music, apps, and movies. Amazing.

While I never would have guessed it, this has really changed my ability to work in hostile environments. A little music can really fend off the gloom, and given the level of gloom I'm in, I need all the help I can get.

Never have so many suffered so much based on a single product selected by so few

My respects to Mr. Churchill, but had he been a developer today, he might say the same thing about a product I'm working with. It's become meeting madness, where it's common to have one, two or even three meetings a day on this single product that was supposed to simplify things here. Instead, it's making things far more complex than ever before.

Let's not even get into the fact that users are saying we're not supporting it to the same level of service that they have come to expect with other applications we've delivered. Ignore that people in the trenches have been saying that there needs to be significant staffing increases in order to get this product rolled-out and supported properly. Forget this stuff.

Just focus on the humanity.

It's horrific. People are beaten down. They are lifeless. They have given up hope. There's no motivation coming from the top... no sense that this is the good fight... it's just like trench warfare. Never ending. Depressing.

Someday this might work. I don't know anymore. Right now, I just want it to end.

That wonderful line from Office Space really hit home today. I have spent a lot of time with specs and documentation and very little time really coding or doing something useful. It's a lot of paper-pushing and very little meaty stuff to make the day pass quickly.

Consequently, I'm ready to get out of here and just enjoy the evening. I know it's in the middle of a (relative) heat wave - it's 90+ and very high humidity, but it's August in the MidWest - what can you honestly expect? This is what it's like here, and I've no interest in leaving. Misery loves company.

I'm hoping that tomorrow is going to be a day where I actually do more than I read or write. It's just that I'm feeling very managerial and if I wanted to do that, I'd have stayed in my previous job(s)... I'm here because I want to keep doing, but that doing is getting harder to find here. Sad to say.

Maybe things will pick up and there will be something that I have to dig into that takes half a day tomorrow - that would be sweet. Sounds odd, I know, but I really do hate the paper-pushing.

This morning I had a Java HotSpot error in one of the components of a production system of mine, and I have to say, it's not been very helpful. I've looked at the HotSpot file and the problem seems to be in java.util.zip.ZipFile.getEntry() and Google tells me this is because of a possibly corrupt jar file, or someone has written to the jar file and made it so that the reading get very confused.

Unfortunately, all the jars have been stable for quite a while, but maybe the filesystem was the issue. We might have had some NFS issues which might have lead to this. Unfortunately, I can't do a lot about where the jars are stored, they need to be centrally located and there's only the one filer. Bummer.

I've sent a few emails the the guys who wrote this component to see if they have any ideas about what might be done to increase stability. I'm on JDK 1.5.0_10, but would 1.5.0_15 be better? 1.6.0? I'm searching for anything because there's no way the files were corrupted. I restarted the app and it all came back. Odd.

I just wish there were more information or a clear explanation of other possible causes so I might find the real culprit of this crash.

UPDATE: when it looks like the chips are down, sometimes you get a break. While digging into this with the developer that wrote the component having the problem, he noticed that the crash occurred when a script was being run through the system. I noticed that the last two crashes occurred at 8:00 am for the last two Friday mornings - just when cron launches these jobs. BINGO! While he was going to track down if the problem was in the bsh jar I was using, I decided to take a completely different track - forego the bsh script and write it in perl. Then, I'm not calling the system to do the work for me, I can do it outside the system and it's much safer and cleaner. Not to mention faster. I recoded this and it works like a champ. It's going to make things so much easier. Whew!

Over the course of the last probably 8 years the number of pieces of paper I've printed out are few and far between. I have to print out two pages a week for my time and billing, and when I go someplace that I need directions to, I map it and print that out. It's a very little bit of paper.

I buy books electronically now. I get PDFs of manuals when I can. All in all, I'd like to be as paper-free as possible. And I've been doing a good job... until recently.

Part of my current assignment is a ton of paper-pushing. Read this spec, write this spec, what's the status of this spec? It's what I thought I was walking away from 12 years ago when I left my own company. I left management behind, and wanted to just do.

But every time I think I'm out, they find a way to pull me back in (excuses to Michael C.)

If I felt it was going to really be something then I'd be a lot more excited about it. But it's just paper because business feels like it's accomplishing something when it goes through fifty reams of paper a day through a printer. After all, what's getting printed has to be useful, right?

I've told them I'm not the guy for this job, but I've been told I am. Funny... I thought I'd have known that by now. Silly me. So I push paper. What a state.