What's it all about, Alfie?

This morning I had a Java HotSpot error in one of the components of a production system of mine, and I have to say, it's not been very helpful. I've looked at the HotSpot file and the problem seems to be in java.util.zip.ZipFile.getEntry() and Google tells me this is because of a possibly corrupt jar file, or someone has written to the jar file and made it so that the reading get very confused.

Unfortunately, all the jars have been stable for quite a while, but maybe the filesystem was the issue. We might have had some NFS issues which might have lead to this. Unfortunately, I can't do a lot about where the jars are stored, they need to be centrally located and there's only the one filer. Bummer.

I've sent a few emails the the guys who wrote this component to see if they have any ideas about what might be done to increase stability. I'm on JDK 1.5.0_10, but would 1.5.0_15 be better? 1.6.0? I'm searching for anything because there's no way the files were corrupted. I restarted the app and it all came back. Odd.

I just wish there were more information or a clear explanation of other possible causes so I might find the real culprit of this crash.

UPDATE: when it looks like the chips are down, sometimes you get a break. While digging into this with the developer that wrote the component having the problem, he noticed that the crash occurred when a script was being run through the system. I noticed that the last two crashes occurred at 8:00 am for the last two Friday mornings - just when cron launches these jobs. BINGO! While he was going to track down if the problem was in the bsh jar I was using, I decided to take a completely different track - forego the bsh script and write it in perl. Then, I'm not calling the system to do the work for me, I can do it outside the system and it's much safer and cleaner. Not to mention faster. I recoded this and it works like a champ. It's going to make things so much easier. Whew!

I've spent a lot of time today working with a good guy in the ticker infrastructure team to get bugs in his group's code worked out. It's not been easy on him, that's for sure. Today I showed him no less than 4 bugs, each requiring new libraries be built - or in the case of one, the infrastructure itself had to be patched and upgraded.

I gotta hand it to him, it's a massive change, but I'm amazed that his developers didn't find any of these bugs in their testing. I'd have thought they'd have seen these as I did upon first inspection. I guess this is what he's going through to get his guys to get the bugs fixed.

In the end, I know it'll work, but I just feel really bad for him when I keep pointing out bugs less than 5 mins after a new release that he's thinking fixed the very last bug. Nope. There's a few more.

At least he's taking off for the weekend and getting some rest. I'll email him the next set of bugs in the morning and he can get to them on Monday. He deserves a break. It's close, but it still needs work.

I've used a lot of different source control systems in my days. Some have seemed to be really nice, only to be replaced by others that are even nicer. Some were replaced by worse systems, but for the most part, it's about the integrity of the development process. Period. Some make it easier to work in teams, some make it easier to work in a distributed manner, but all should have the same key feature: it allows you to version your files so you don't loose anything.

Which is why I was stunned to be in the middle of a discussion about the "one" source control system we should be using. Why? We've used ClearCase, and it worked OK, but then they didn't like supporting it and told us to move to a different system. We've used CVS for ages and it's never failed us. You can use Subversion now, and it's fine. I'm not sure, but I'm willing to bet that just about any reliable SCM is going to work. Just pick one, make sure it's safe, backed-up, and then get on with the job.

To say a group needs to use one at the exclusion of another is silly. You don't use one development environment at the exclusion of another - use Eclipse if you want... use Emacs, use Vi... just get the job done. Why would a group of professional developers even consider making rules like this?

OK, maybe some places feel it's important to have a single tool for each category so that people can get used to it and not have to change. Fine. Then by that logic, if someone is using a tool, don't make them change. If it's working, then it's solved. If it's not, then fix it. But making rules like this when nothing's broken is just plain silly.

When I've used projects that are in Subversion, I used subversion. When they were in ClearCase, I used that. What's the big deal? There is no one tool that's so vastly superior to the others that it can't be made to work, just pick one for a project and stick with it. Seems pretty clear to me.

But I'm in the minority here, it seems. They want to say all new development needs to be in one system. Until they don't like that and move to another, as they did when they moved to this one they are favoring now. It's crazy, but that's corporate life, I guess.

This morning I was able to restart the last of my servers that had a re-configuration with the MindAlign system to disable it. These were the 10 authentications per minute that were spamming the authentication server... well... the last of them are finally disabled and things seem to be going swimmingly. Good for me. I wrote up the documentation on how to re-enable them once MindAlign decides to allow bots once again - whenever that is. I just wanted to make sure that if I got hit by a truck, someone could read the docs and get things going again.

I know from experience that this is not going to make any difference in the MindAlign stability, but they have been rather insistent that we shut down these spamming bots and so I have done it as quickly as is prudent. We'll see if they are as quick to restore complete MindAlign service.

I'm not holding my breath.

Today has been the first day of really hammering on the new ticker infrastructure for any issues. There were several, and over the course of the day the Team Lead and I have figured them out. Currently, there are only a handful of differences, and those might be related to the timing of the configuration changes as they were done throughout the day. We'll have to wait and see what data tomorrow has. If I'm right, all these remaining differences will vanish and we'll be left with a clean set.

It's nice to make progress on something like this. It turns out a better product for our group and well as getting the global support group to have a better product for everyone.

I've been working a lot with MindAlign lately, and specifically with the case where the login credentials are not valid and the server returns 'ERROR:504' as opposed to the package that the protocol indicates. Since I hadn't seen anything like this in their spec, I have just been assuming that it's a protocol error and throwing it up the exception chain as a general error in the code I'd written for MindAlign in BKit and CKit.

Well... that's no good. So I took the few minutes to clean that up and create a generally more informative exception message saying that the authentication server didn't like the login. This isn't going to revolutionize anything, but it might save someone 5 mins debugging, and that's good enough.

I just got finished fixing a problem that I should have seen coming. Yesterday, due to some chat server issues, I needed to update an app of mine to get a little fix for a socket problem so I did a standard deployment. Problem there... the development codebase has been moved to a different version of a data provider's API, and so the deployment to production totally hosed the installation. Yikes!

OK, time to be clever... checkout everything on the last good build of the server, rebuild it and ship it out. Whew! it worked. That is... until this morning.

This morning a user came by saying they could not get data from the server, and I checked and he was partly right. Some data could be obtained, and some couldn't. After a few tries, it seemed that the tables-of-tables data was the thing that was causing the server to crash. Not good. But what could it be?

The code hadn't been touched in ages (before the move to the new API), so that wasn't it. Then I thought it had to be in the libraries - there had to be something that was causing the shared library to mess up - maybe a bad header file. So I took the time to get all the headers up to date and then rebuilt everything again.

This time it was fine - and it makes perfect sense. There must have been a change in the CKTable's header that was in the shared library version that wasn't in the header file I was using and they two were causing the crash when the code hit it. I should have seen this coming and been much more careful about the updating of the libraries in the build the night before. Yes, it's not a mistake I'll make again.

We use MindAlign in the Shop to have a secure chat environment, such as it is. Over the last few days, it's been problematic to say the least. Today, in fact, they have narrowed down the problem to a global corporate loss of DNS due to MindAlign.

I am not going to bad mouth MindAlign, it's a package that the corporation has picked, and we're going to use it. Period. But today when I came into work I noticed (again) that almost all my production applications (servers and services) had one core/CPU hammering away doing seemingly nothing - but very very fast.

I found that once again we were having MindAlign problems and I decided that this morning I'd dig into the problem and find out why they were all spinning like mad. The first step on that path was to get a few stack dumps, and it's a lot easier to do that in Java than it is in C++, so I did a few thread dumps on an afflicted Java app and after three of them, it because pretty clear where the problem was - in the BKIRCProtocol object. That's the thing that talks to the IRC server, which is the core of MindAlign.

It appeared that the IRC protocol was trying to throw an EOFException because the socket was not returning any data, and was, in fact returning an error code indicating that it wasn't all there anymore. The problem was, that at the higher level, where the EOFException was being caught, I was assuming that the socket was closed - which is why I'd get the EOF condition in the first place. By making that assumption, a socket with an indeterminate state talking to the MindAlign server was "there", but not "there enough".

I put in an explicit disconnect() on the socket connection and that should take care of things. Simple one-line fix (with an accompanying 10 line comment to say why it's being done. Then I was on to the C++ version.

Thankfully, the code is very similar, but the socket communications isn't. But I dug into the code about the same spot - a socket that's almost dead, but returning that it's still connected even though it's not returning any data. What I found was code that looked like this:

  // now read up to the "\n" NEWLINE that the IRC server sends
  if (!error) {
    retval = mCommPort.readUpToNEWLINE();
  }

where readUpToNEWLINE() should return only when a line is received from the IRC server terminated in a NEWLINE (\n) character. The problem was, it was returning an empty string and that was all I needed to know. If it returns nothing, then the only acceptable reason is that there was a timeout. So, check for that, and all other conditions cause us to disconnect from the server and on the next pass, a new connection will be created.

Something like this:

  // now read up to the "\n" NEWLINE that the IRC server sends
  if (!error) {
    retval = mCommPort.readUpToNEWLINE();
    /*
     * It's possible that the data is empty - but the only way for
     * that to be acceptable is for a timeout to have occured. So,
     * if the data is empty and a timeout *didn't* occur, then we
     * need to disconnect this guy and the next pass through, we'll
     * be able to connect again and set things up properly - we
     * hope.
     */
    if (retval.empty() && (errno != ERR_READ_TIMEOUT)) {
      disconnect();
    }
  }

At this point, I think I have the MindAlign issues under control. There's still the issues with the corporate servers, etc., but I can't do anything about them, and we'll just have to wait for them to get fixed up properly by the support teams. It's just nice to have these fixes into BKit and CKit so that I won't get bitten by this problem in the future.

This morning I was updating one of the Numbers spreadsheets I had and was trying to push the envelope a little more and see what tools Numbers really had for - well... crunching numbers. I had some data that I wanted to have massaged a little more automatically, as opposed to the brute-force method that I had been using for the past year - since I moved the data from Mesa to Numbers.

The first thing was to have a better calculation of the YTD comparison of monthly numbers. The issue was that I only wanted to look at those months I had data for this year in the previous year. Making it as apples to apples a comparison as possible. The trick there was that the comparison needed to be a sum of the data, so it was possible to make a new column with the IF() function, but it was done even better with Numbers' SUMIF() function.

By simply making the test range this year, and the comparison ">0" and the sum range the previous year's data, I get the sum for the previous year's data for only those months that I have data for this year. Excellent! One thing done automatically that I needed to do.

The next thing was to have the year-on-year monthly changes automatically computed without messing up the sums or averages by putting a '0' in the cell. This was pretty easily done with an IF() call where the conditional was the value for this month of this year being great than zero, and the true clause was the percent change. The false clause was simply the empty string, and that worked out pretty well.

When I was done with the process I had a workbook that looked the same, but acted entirely differently, and much more like a cool, complex, spreadsheet should operate. Now I put in the raw data and shazam! the complete workbook gets all the data it needs. Very cool.

Yes, I realize this isn't that big a deal, but the little I do with spreadsheets is very much brute-force and the steps to find out these functions and use them well was an interesting exercise.

OK, I've been coding for a long time. I've used lots of development environments and the only one that I've found that really holds my interest is ProjectBuilder/Xcode on Mac OS X. From the first time I used it on NeXTSTEP 3.3, I knew it was something far beyond an editor with build tools. It was just so clearly built by developers, for developers. I'm still of that same opinion today.

This morning I read about an interesting tool that analyzes the Obj-C/C code and tells you where you might have issues with leaks, etc. Nothing that hasn't been covered a dozen times before in a dozen different tools, but this one is a little different. First, it's a syntax checker. Again, lint did this for decades, but this is different in that it knows about the idioms of Obj-C and the class structure that Mac OS X uses. That's huge.

Next, it's slick in it's reporting in that it's all web-based and generates all this very simply. I'm not a big fan of web apps, but for reporting, it just can't be beat. What they have done here is the very best of web reporting for coding issues, and I have to say it's remarkable.

I decided to put in one of my projects to see what it might find. Interestingly, it found a possible error in one of my classes in that I wasn't creating a - (void)dealloc method for the instance. Now, technically, the class involved is the main controller class, and it would not get deallocated except on shutdown, so there's no real problem there, but it's nice to keep things clean and try to do the right thing all the time. So I added the very simple method and re-ran the checker. Clean, save for a dead store issue that I'm not about to change as it's really just the last part of a method that doesn't return the error code. It's more of a coding style than an error, but I appreciate that it's telling me about this.

While I'm not doing a lot of Obj-C coding these days, it's tools like this that make me want to change jobs so that I did do a lot of Obj-C coding. The Mac is a wonderful platform, and its development tools are the best I've ever seen. With the addition of tools like this code analyzer, it's even better. Long live the Mac!