What's it all about, Alfie?

I love my job, I really do. This place has the nicest people I've worked with in a long time. It's great. But it's not perfect. Case in point: I was working on trying to figure out a production problem yesterday where a few INSERT statements into a MS SQL Server database was throwing up exceptions about primary key violation and this, in turn, was causing SELECT statements to fail.

I can't blame this place for MS SQL Server's problem of locking out the SELECT statements when it's upset about primary key violations on INSERT statements. Face it - the database should just fail the transaction and then go on about it's business. There's no need to have locks that are going to gum up the works for SELECT statements. That's Microsoft's fault.

But the problem that really led to this was the fact that I had part of a primary key was 'PHL' and another row had 'phl'. Now, to every other database I've ever used these are distinct values. The datatype is a simple varchar(16), and it's always been unique to me. But not this database.

No, it was a design choice to make all keys (primary and foreign) as well as indexes to be case insensitive. Amazing. This would be OK if they forced all the varchar data to be stored upper-case, but they don't, and it isn't. Which means that every stored procedure or client code needs to deal with converting the data coming out to the proper case. Again, possible, but why?

The closest thing I got to an answer was that historically, this matches the way the data was handled in the flagship app from years gone by. OK, I can possibly see that, but not really. If you're going to invest in a serious relational database, then why on earth tie it's hands with something like case-insensative keys?

I've put code into the applications that deal with the database to make sure that things are all uppercased, but it's still a potential problem for me until I get this code through Q/A and out into production.

I've been working within the limitations of the AnnotatedTimeLine graph from the Google Visualization API set and have just been telling my users that the unfortunate "flash" to white of the graph while it's updating data is an unfortunate reality of the AnnotatedTimeLine, and that Google knows this is in issue and they are working on it.

Then I read an interesting post to the mailing list about how Google Finance didn't have this issue, and it got me thinking... How could they get around the flash? And in the show this morning, it hit me: Double-Buffering.

Simply have two graphs - one aligned in the z-axis behind the other. In fact, make them have the same coordinates, but just vary the zIndex value to "flip" them one in front of the other. Then, always draw to the rear (invisible) graph and when it's done, flip it to the front by setting sIndex to 49 on it, and setting zIndex to 0 on the one that was in front.

A simple use of an array of two graphs accomplishes this quite easily. I add in two variables - the index values for "foreground" and "background", and then you always draw to chart[bg] and so on. It's very clean and slick. The change of the z-axis values is done in the blink of an eye, it's very hard to see. Compared to the 3 to 6 seconds that the redraw used to take, it's nothing short of amazing.

In a very simplistic form:

  var chart = [];
  var fg = 0;
  var bg = 1;

and then in my initialization code, called from the Goolge setOnLoadCallback() method, I have:

  chart[0] = new google.visualization.AnnotatedTimeLine(chartDiv[0]);
  chart[1] = new google.visualization.AnnotatedTimeLine(chartDiv[1]);
  // tie us into the events for these guys
  google.visualization.events.addListener(chart[0], 'ready', graphReady);
  google.visualization.events.addListener(chart[1], 'ready', graphReady);

and then when I need to draw to the correct graph, I simply call:

  chart[bg].draw(graphData, chartParams);

When the graph is done drawing, I'll get a call to:

  function graphReady() {
    //swap the graphs quickly
    chartDiv[fg].style.zIndex = '0';
    chartDiv[bg].style.zIndex = '49';
    // now swap what we mean by back and front
    if (fg == 0) {
      fg = 1;
      bg = 0;
    } else {
      fg = 0;
      bg = 1;
    }
  }

There's a lot of the details I've left out because they don't really matter. The point is that you have to place them on top of each other, and then do this double-buffering. It's pretty nice.

Today I got a call from the original developer of this project I've somewhat taken over. He still has components of this project that are relatively new, and I haven't even been introduced to. Needless to say, there's no documentation whatsoever - other than what I've put into the codebase as I've been working on it.

So today when he called and said that one of his reports was missing a lot of data, I knew it was not going to be fun. He was really busy with this other project that his desk was depending on, and it was going to be up to me to figure out what had happened and how to get it back to "right".

Documentation: Zero

As I've said, this guy didn't document a thing. There's a few high-level 10,000 ft. documents about how things are supposed to work. Problem is, this project was built with Spring and while it's certainly got benefits, I have to say making a reasonable, understandable, documented system isn't one of them. Confused? Sure. Exotic XML config files? Roger.

I don't mean to harp on this, but I'm going to because it's just that important. If you can describe the system at a high level as simple and flexible, then the implementation ought to be simple and flexible. It seems that Java has created this entire cadre of developers that over-design even the simplest of things just because they can. It's one of the most frustrating things I've run into in years.

Every language has it's detractors, and my favorites are no exception. I'm not one to throw stones, living in a glass house, but this really does seem to be something of a pandemic. Systems built with tools that almost restrict the developer from making useful, well-documented, code and easily understood code.

Unit Tests: Useless

This is the project that has the most incomplete, yet copious, set of unit tests that I've ever seen. The unit tests passed completely, but at the same time, they didn't point out why I might have absolutely no data in the report. Does that sound like a good set of unit tests? Yet, I spent the better part of two days updating these same unit tests when I added a feature that took me about an hour. What's right with that?

If you're going to have unit tests, and more than that, have integration "unit" tests, then don't you think it should cover the case where a great deal of the result is empty? If it's a coding mistake, it should be caught in the tests. If it's a data issue that would blow out that much data (like a null pointer), then again, it should be flagged as at least a warning.

No tests failed or even logged problems.

Getting to the Bottom of It All

I looked at the SVN change logs for all the files. Nothing there would make all the data disappear. I looked at the data in the database - maybe a lead there, but no, it was a false alarm. I looked at the data moving around the system at a level that wasn't easy to locate as there wasn't any documentation (as I've said), but in the end I was able to see the data coming out of the test system and the production system. Interestingly, the data at this level wasn't blank. It appeared that the Flex client was interpreting the data as something it didn't want to display.

Now we were getting somewhere.

When I looked at the data more closely I noticed that the numbers were remarkably similar. Interesting. Then I noticed that one part of the data was the wrong symbol. Very interesting, now. So I asked the original developer on this point, and he said that the symbol I had was wrong. OK, let's go there.

I had put the configuration of the system into a database so that it's far easier to maintain and basically is driven off the data that's maintained by the individual desks. When I put in these normalizing contracts I put in what I believed to be the right contracts. Turns out, for this report, for this group of instruments, it was wrong.

A one field change in the database and a restart of the Tomcat instance and Bingo! data was there in the report.

Lessons Learned

At the end of the experience I was sad to realize that I hadn't really learned a lot from this other than what I already knew: poor documentation, poor tests, poor checking on the client, the list goes on and on. While I know it's like whipping a dead horse, as long as I'm forced to work with this system it'll continue to be a source of anguish to me. It's got a lot of nice things in it, but the design and documentation aren't among them.

The code is consistent for the most part, and the goals and attempts are admirable, but in the end it's another victim of that pandemic of over-design and under-documentation. Make it simple... make it clean... work on the documentation... make it something that when another developer looks at it, it makes them code better. Set the Gold Standard for code. Be better than you have to be.

I learned those lessons years ago, and I only home some of them rub off.

Today has been a really stressful day. I was getting hit with production problems at the same time I was learning that the code changes I'd made in this project I inherited weren't sufficient to stand-alone, and in order to really work I had to make significant changes to the code.

Very stressful. Didn't eat until late. Gulped it down.

Had to fix the code first, it had the greater risk of me not getting it done. The problem was that I believed that the changes I had made completely removed the necessity for the config files (Spring - ick) to have this list of instruments. Instead, I wanted to read them from the database where they were held. Makes tons of sense to not have to maintain the same data in two forms. Problem was, when I removed the files I found that they had actually been referenced in other places, and I needed to get those lists out of these other places.

That didn't turn out to be as easy as I'd have hoped. With the components responsible for their own loading, it was difficult to see where I might be able to use Spring to reference the containers with their database-loaded information. But there was. Then I hit another problem - a circular reference in the Spring.

This was getting to the point I was ready to re-write the whole thing.

But I got an idea - get a 'core' set of containers and have them queried for their instruments as opposed to doing everyone. Now we're talking. With that, I was able to fix things up in a few hours. Still... the unit tests were the vast majority of the time, and a significant hassle as the containers are not easy to make in isolation.

Once I got all that done I was able to check it in and send off an email to the other developer on the project and tell him to run through all the tests to make sure that I haven't broken anything. Just making sure.

Then I could focus on the production problem... that was another toughie, but I was glad to be able to find the difference. It wasn't so much a problem as it was a difference. The production box was using specific 'Ask' prices and the users were thinking it should be using 'Bid' prices - and as you can imagine that would change things considerably.

When I matched those conditions the numbers started lining up very nicely. So I asked them if it was going to be 'Ask' or 'Bid', and they said 'Bid', so I changed production to use 'Bid' and we should be good tomorrow.

Finally, I had another production problem with the data from London. It turned out that one of the London machines had crashed on me - it happens about twice a day in London. All the other sites are fine, so I'm convinced it's the VMs in London. I passed on the information to the London guys and they said they'd be looking into the VMs tomorrow. Good enough.

Super heavy, stressful day, but dog gone it, I got it all done. All of it. That's pretty nice. Now I can go home and feel I earned my dollar today.

I spent a lot of today still working on getting the unit tests fixed up for the changes I made. I've said enough on this already, but the amazing thing to me was when I got through them, I was able to add in the significant new functionality in about 30 mins. This goes back to the best way I work - create human test cases, and understand the code to know what needs to be tested in each release cycle. It can't always be done, that's true, but when I can work that way, I get a lot more done.

I'm hoping to get to do a lot more of the latter and a lot less of the massive unit testing.

I've been spending all day trying to get the unit tests for this inherited application working, and I'm closer to being done, but I'm not done. It's like the rock Prometheus has to roll up the hill, only to see it slide back down. If I want to add in new features to the codebase, then I have to face these tests, and it takes me far longer to update the tests than it does to test the code, or write it in the first place.

I know there are paradigms that say "Write the Tests First", but I can't actually imagine doing that unless you're writing small, easily testable, modules. This integration testing is not for unit tests, and I keep harping on it, as I'm continually forced to deal with them.

In order to make things work today, I had to create "Mock" services that just fed in the test data. It's again making no real progress in the testing code, we don't know that the database interface is working, and while we can run those tests, it's not doing the kind of integration testing that is going to make me feel any better.

So it's slugging it out all day with these things. I really do get very tired of it.

I've been working on some interesting additions to one of the projects I'm working on and for the first time I've had to work on this Java app that runs on Windows. There's no Ant build scripts for it, though I suppose I could install Ant on my box, create a build.xml and try to make a go of it, but the command line is pretty nasty and the original author of the component used Eclipse to do the development and testing. So I decided that the easiest thing was to use Eclipse and then I'd be sure that I was doing the same steps he was.

Seemed reasonable.

Because I'm no Eclipse wizard, I had him help me set up my workspace/environment so that it'd build and such, and then I could work on the changes. What I found was not terribly surprising to me, but experiencing it was still shocking. Eclipse is an advanced development environment and it saves a lot of people a lot of time, but I found that it was constantly slowing me down.

I'm far far faster developing in Vim and using Ant - on linux. But this is one little component and so I'll stick with this way of doing things as it makes sense to mirror how the other developer is writing the code, but Holy Cow! I had no idea that it was still this intrusive.

Well... I don't have to use it every day - just those days I'm adding to this little component.

I was working this afternoon and happened to catch out of the corner of my eye the founding partner I work for talking to someone (whom I didn't recognize, but had a suit so he's not "native"). He was very animated... hands up in the air, big smile on his face, looking my way.

I took out my earbuds and looked over at him with a look that said "Is there anything I need to do?" His response was a fantastic compliment: "I was just bragging about you".

Wow.

Double Wow.

I've always hoped that the things I did were noticed, and here I've been at the job a little less than 4 months and already it's something they are that happy with. That's really neat. I mean - Holy Cow! You don't have to do that stuff, and certainly not tell me about it.

Sweet. Put a big smile on my face.

In the last few weeks I've written several times about my dislike of extensive unit testing when it comes to large, complex systems. Unit tests are perfect for building blocks - simple model objects that have well-defined inputs and outputs that can be extensively tested. It's my days in the IC fab industry that makes me wary of testing. The guys at HP had amazingly hard test vectors to test every component as quickly as possible for all possible defects. It wasn't easy.

But I've been working on a system where the unit tests are large, complex, and difficult to maintain -- and don't test enough of the system. But it's assumed that these tests are complete coverage of the system. Well... today it bit me in the behind, and I have solid reasons why I now hate this philosophy.

This morning, I found that the update from last evening wasn't working exactly as it should. In fact, it wasn't working properly for a lot of cases. I was trying to see if is was the data - hard to tell, that's for sure. Maybe it was the install... nope. I was getting pretty desperate. Then someone mentioned checking the Tomcat localhost log. Bingo!

There were a bunch of exceptions that told me exactly what the problem was. Why is this not in the catalina.out log? I'm guessing it's the way this webapp is set up, but all my exceptions should go to catalina.out. Here nor there at this point, I had found the problem.

The problem was that one component was sending a record of six elements, and the other system was expecting seven. Clearly, this was the problem, and when I fixed the sender to add the missing data, it all worked fine. But it was 3 hours of a nightmare because of the impact to production.

All the unit tests would never have picked this up because it was between two components in the same system. More over, there was no testing of this specific feature in my tests, the other developer's tests, even Q/A didn't test this. While I'm not mad at anyone (other than myself), I think this highlights the folly of thinking that unit testing of large, complex systems, is anything more than a waste of time. Small, easily identifiable blocks - sure. But webapps? Please.

So now I've learned something valuable, and when the next person tries to spout the Gospel According to Unit Testing, I'll have a good story to share with them.

Today I spent the majority of the day updating jUnit tests for this new feature that I need to add to this system I have inherited. The addition of the feature wasn't too bad. There were wrinkles as we're not really handling the booking of positions like you might in an accounting system with all the levels, etc. that you might otherwise do, but that's part of this business - it's a limited subset of products and how they are organized. No biggie, still it took a little bit to work it in.

The vast majority of the day was the updating of the jUnit tests. I've thought about this a lot, and there's a lot to be said for Unit Tests, and there are real benefits to Integration Tests, but I think that confusing one for the other can not only make for bad testing, but a false sense of security.

First, it's important that Unit Tests include - at a minimum, the following:

• Edge Conditions - if you're not checking for the minimum and maximum allowable values then you're not really testing the unit. If you use only one datapoint, the "black box" might just work for that point and nothing else.
• Improper Inputs - if you code up checks on the inputs, then you need to test those so that you're certain that should the user pass you in an illegal value, you'll catch it.

this is by no means a comprehensive list - it's just the minimum you need to test in order to have a reasonably high level of confidence that what you've written is working. Face it... it's all about code-coverage. If your tests only hit 10% of the code in a module, the remaining 90% can have millions of bugs. You need to try and get as good code coverage in your tests in order to increase your confidence in the tests.

Then there's the issue of using unite testing frameworks to do integration testing. This is a really an extension of the problems associated with incomplete code coverage testing. Face it, if you're not testing all of your units completely, then trying to test three or more with the same coverage is going to make things even more dicey. Sure, you can do it, but the tests become less and less about really creating assurance as they are about habit.

Someone used to writing unit tests may not be able to draw the line. They're so used to writing them they just can't stop. They drive them far past their point of usefulness. Oh sure... there is some utility in the limited tests, but it's nothing like an assurance that things are working properly. It's more like "Yeah, I haven't messed this horribly."

This is really nothing more than simple testing with live data.

I have seen cases where the unit testing is complete and amazing. But in those cases, it's a lot of work and maintaining the tests is a significant bit of work. I've also seen cases where the testing doesn't ensure nearly anything, but it's still a ton of work because of the integration set-up required.

Test appropriately. This includes unit tests and human-driven tests.