What's it all about, Alfie?

I love my job, I really do. This place has the nicest people I've worked with in a long time. It's great. But it's not perfect. Case in point: I was working on trying to figure out a production problem yesterday where a few INSERT statements into a MS SQL Server database was throwing up exceptions about primary key violation and this, in turn, was causing SELECT statements to fail.

I can't blame this place for MS SQL Server's problem of locking out the SELECT statements when it's upset about primary key violations on INSERT statements. Face it - the database should just fail the transaction and then go on about it's business. There's no need to have locks that are going to gum up the works for SELECT statements. That's Microsoft's fault.

But the problem that really led to this was the fact that I had part of a primary key was 'PHL' and another row had 'phl'. Now, to every other database I've ever used these are distinct values. The datatype is a simple varchar(16), and it's always been unique to me. But not this database.

No, it was a design choice to make all keys (primary and foreign) as well as indexes to be case insensitive. Amazing. This would be OK if they forced all the varchar data to be stored upper-case, but they don't, and it isn't. Which means that every stored procedure or client code needs to deal with converting the data coming out to the proper case. Again, possible, but why?

The closest thing I got to an answer was that historically, this matches the way the data was handled in the flagship app from years gone by. OK, I can possibly see that, but not really. If you're going to invest in a serious relational database, then why on earth tie it's hands with something like case-insensative keys?

I've put code into the applications that deal with the database to make sure that things are all uppercased, but it's still a potential problem for me until I get this code through Q/A and out into production.

With the new release of FireFox 3.5, the plugin authors have been working to get their code to be compatible with the new release, and this morning I noticed that Firebug had an update. I use this at work to look into the failing JavaScript code I (sometimes) write and be able to see what's really happening. Excellent tool.

I've been working within the limitations of the AnnotatedTimeLine graph from the Google Visualization API set and have just been telling my users that the unfortunate "flash" to white of the graph while it's updating data is an unfortunate reality of the AnnotatedTimeLine, and that Google knows this is in issue and they are working on it.

Then I read an interesting post to the mailing list about how Google Finance didn't have this issue, and it got me thinking... How could they get around the flash? And in the show this morning, it hit me: Double-Buffering.

Simply have two graphs - one aligned in the z-axis behind the other. In fact, make them have the same coordinates, but just vary the zIndex value to "flip" them one in front of the other. Then, always draw to the rear (invisible) graph and when it's done, flip it to the front by setting sIndex to 49 on it, and setting zIndex to 0 on the one that was in front.

A simple use of an array of two graphs accomplishes this quite easily. I add in two variables - the index values for "foreground" and "background", and then you always draw to chart[bg] and so on. It's very clean and slick. The change of the z-axis values is done in the blink of an eye, it's very hard to see. Compared to the 3 to 6 seconds that the redraw used to take, it's nothing short of amazing.

In a very simplistic form:

  var chart = [];
  var fg = 0;
  var bg = 1;

and then in my initialization code, called from the Goolge setOnLoadCallback() method, I have:

  chart[0] = new google.visualization.AnnotatedTimeLine(chartDiv[0]);
  chart[1] = new google.visualization.AnnotatedTimeLine(chartDiv[1]);
  // tie us into the events for these guys
  google.visualization.events.addListener(chart[0], 'ready', graphReady);
  google.visualization.events.addListener(chart[1], 'ready', graphReady);

and then when I need to draw to the correct graph, I simply call:

  chart[bg].draw(graphData, chartParams);

When the graph is done drawing, I'll get a call to:

  function graphReady() {
    //swap the graphs quickly
    chartDiv[fg].style.zIndex = '0';
    chartDiv[bg].style.zIndex = '49';
    // now swap what we mean by back and front
    if (fg == 0) {
      fg = 1;
      bg = 0;
    } else {
      fg = 0;
      bg = 1;
    }
  }

There's a lot of the details I've left out because they don't really matter. The point is that you have to place them on top of each other, and then do this double-buffering. It's pretty nice.

Today I got a call from the original developer of this project I've somewhat taken over. He still has components of this project that are relatively new, and I haven't even been introduced to. Needless to say, there's no documentation whatsoever - other than what I've put into the codebase as I've been working on it.

So today when he called and said that one of his reports was missing a lot of data, I knew it was not going to be fun. He was really busy with this other project that his desk was depending on, and it was going to be up to me to figure out what had happened and how to get it back to "right".

Documentation: Zero

As I've said, this guy didn't document a thing. There's a few high-level 10,000 ft. documents about how things are supposed to work. Problem is, this project was built with Spring and while it's certainly got benefits, I have to say making a reasonable, understandable, documented system isn't one of them. Confused? Sure. Exotic XML config files? Roger.

I don't mean to harp on this, but I'm going to because it's just that important. If you can describe the system at a high level as simple and flexible, then the implementation ought to be simple and flexible. It seems that Java has created this entire cadre of developers that over-design even the simplest of things just because they can. It's one of the most frustrating things I've run into in years.

Every language has it's detractors, and my favorites are no exception. I'm not one to throw stones, living in a glass house, but this really does seem to be something of a pandemic. Systems built with tools that almost restrict the developer from making useful, well-documented, code and easily understood code.

Unit Tests: Useless

This is the project that has the most incomplete, yet copious, set of unit tests that I've ever seen. The unit tests passed completely, but at the same time, they didn't point out why I might have absolutely no data in the report. Does that sound like a good set of unit tests? Yet, I spent the better part of two days updating these same unit tests when I added a feature that took me about an hour. What's right with that?

If you're going to have unit tests, and more than that, have integration "unit" tests, then don't you think it should cover the case where a great deal of the result is empty? If it's a coding mistake, it should be caught in the tests. If it's a data issue that would blow out that much data (like a null pointer), then again, it should be flagged as at least a warning.

No tests failed or even logged problems.

Getting to the Bottom of It All

I looked at the SVN change logs for all the files. Nothing there would make all the data disappear. I looked at the data in the database - maybe a lead there, but no, it was a false alarm. I looked at the data moving around the system at a level that wasn't easy to locate as there wasn't any documentation (as I've said), but in the end I was able to see the data coming out of the test system and the production system. Interestingly, the data at this level wasn't blank. It appeared that the Flex client was interpreting the data as something it didn't want to display.

Now we were getting somewhere.

When I looked at the data more closely I noticed that the numbers were remarkably similar. Interesting. Then I noticed that one part of the data was the wrong symbol. Very interesting, now. So I asked the original developer on this point, and he said that the symbol I had was wrong. OK, let's go there.

I had put the configuration of the system into a database so that it's far easier to maintain and basically is driven off the data that's maintained by the individual desks. When I put in these normalizing contracts I put in what I believed to be the right contracts. Turns out, for this report, for this group of instruments, it was wrong.

A one field change in the database and a restart of the Tomcat instance and Bingo! data was there in the report.

Lessons Learned

At the end of the experience I was sad to realize that I hadn't really learned a lot from this other than what I already knew: poor documentation, poor tests, poor checking on the client, the list goes on and on. While I know it's like whipping a dead horse, as long as I'm forced to work with this system it'll continue to be a source of anguish to me. It's got a lot of nice things in it, but the design and documentation aren't among them.

The code is consistent for the most part, and the goals and attempts are admirable, but in the end it's another victim of that pandemic of over-design and under-documentation. Make it simple... make it clean... work on the documentation... make it something that when another developer looks at it, it makes them code better. Set the Gold Standard for code. Be better than you have to be.

I learned those lessons years ago, and I only home some of them rub off.

This morning I saw that the Mozilla crew finally released Firefox 3.5! This is great news as I have been waiting on this for a long time for work. I'm hoping to see the performance of Firefox 3.5 on Windows to be up to snuff and capable of handling my web pages without crashing. The initial results seem to indicate that it's possible that this version of Firefox is far better at memory usage than the previous versions - and even better than Google Chrome! That would be great.

Even if that's not the case, it's nice to see Firefox making a big move in the JavaScript area. Very much appreciated.

Today has been a really stressful day. I was getting hit with production problems at the same time I was learning that the code changes I'd made in this project I inherited weren't sufficient to stand-alone, and in order to really work I had to make significant changes to the code.

Very stressful. Didn't eat until late. Gulped it down.

Had to fix the code first, it had the greater risk of me not getting it done. The problem was that I believed that the changes I had made completely removed the necessity for the config files (Spring - ick) to have this list of instruments. Instead, I wanted to read them from the database where they were held. Makes tons of sense to not have to maintain the same data in two forms. Problem was, when I removed the files I found that they had actually been referenced in other places, and I needed to get those lists out of these other places.

That didn't turn out to be as easy as I'd have hoped. With the components responsible for their own loading, it was difficult to see where I might be able to use Spring to reference the containers with their database-loaded information. But there was. Then I hit another problem - a circular reference in the Spring.

This was getting to the point I was ready to re-write the whole thing.

But I got an idea - get a 'core' set of containers and have them queried for their instruments as opposed to doing everyone. Now we're talking. With that, I was able to fix things up in a few hours. Still... the unit tests were the vast majority of the time, and a significant hassle as the containers are not easy to make in isolation.

Once I got all that done I was able to check it in and send off an email to the other developer on the project and tell him to run through all the tests to make sure that I haven't broken anything. Just making sure.

Then I could focus on the production problem... that was another toughie, but I was glad to be able to find the difference. It wasn't so much a problem as it was a difference. The production box was using specific 'Ask' prices and the users were thinking it should be using 'Bid' prices - and as you can imagine that would change things considerably.

When I matched those conditions the numbers started lining up very nicely. So I asked them if it was going to be 'Ask' or 'Bid', and they said 'Bid', so I changed production to use 'Bid' and we should be good tomorrow.

Finally, I had another production problem with the data from London. It turned out that one of the London machines had crashed on me - it happens about twice a day in London. All the other sites are fine, so I'm convinced it's the VMs in London. I passed on the information to the London guys and they said they'd be looking into the VMs tomorrow. Good enough.

Super heavy, stressful day, but dog gone it, I got it all done. All of it. That's pretty nice. Now I can go home and feel I earned my dollar today.

I spent a lot of today still working on getting the unit tests fixed up for the changes I made. I've said enough on this already, but the amazing thing to me was when I got through them, I was able to add in the significant new functionality in about 30 mins. This goes back to the best way I work - create human test cases, and understand the code to know what needs to be tested in each release cycle. It can't always be done, that's true, but when I can work that way, I get a lot more done.

I'm hoping to get to do a lot more of the latter and a lot less of the massive unit testing.

I've been spending all day trying to get the unit tests for this inherited application working, and I'm closer to being done, but I'm not done. It's like the rock Prometheus has to roll up the hill, only to see it slide back down. If I want to add in new features to the codebase, then I have to face these tests, and it takes me far longer to update the tests than it does to test the code, or write it in the first place.

I know there are paradigms that say "Write the Tests First", but I can't actually imagine doing that unless you're writing small, easily testable, modules. This integration testing is not for unit tests, and I keep harping on it, as I'm continually forced to deal with them.

In order to make things work today, I had to create "Mock" services that just fed in the test data. It's again making no real progress in the testing code, we don't know that the database interface is working, and while we can run those tests, it's not doing the kind of integration testing that is going to make me feel any better.

So it's slugging it out all day with these things. I really do get very tired of it.

I've been looking at Eclipse on Windows for a bit, because I have to for this one project I'm working on at work, and thinking about what it can do made me think that I've probably been a little stuffy about the IDE. I'm probably not giving it the credit it deserves. Refactoring, automatic compilation... it's probably a nice tool, and I should give it a try.

So I'm downloading the new 'Galileo' version (3.5) from the Eclipse Download site, and going to replace the older version I have. I'm not sure if it's a big difference, but it's probably something I should give a go - if for no other reason than at least trying to understand what the developers are aiming for.

Better to understand and have a working knowledge of something than to dismiss it out of hand.

I've been working on some interesting additions to one of the projects I'm working on and for the first time I've had to work on this Java app that runs on Windows. There's no Ant build scripts for it, though I suppose I could install Ant on my box, create a build.xml and try to make a go of it, but the command line is pretty nasty and the original author of the component used Eclipse to do the development and testing. So I decided that the easiest thing was to use Eclipse and then I'd be sure that I was doing the same steps he was.

Seemed reasonable.

Because I'm no Eclipse wizard, I had him help me set up my workspace/environment so that it'd build and such, and then I could work on the changes. What I found was not terribly surprising to me, but experiencing it was still shocking. Eclipse is an advanced development environment and it saves a lot of people a lot of time, but I found that it was constantly slowing me down.

I'm far far faster developing in Vim and using Ant - on linux. But this is one little component and so I'll stick with this way of doing things as it makes sense to mirror how the other developer is writing the code, but Holy Cow! I had no idea that it was still this intrusive.

Well... I don't have to use it every day - just those days I'm adding to this little component.