What's it all about, Alfie?

Over the last several days we've had significant issues with what appears is non-ASCII data entering the MindAlign servers and bring them all down. Since data like that is very unlikely to be entered by a human, it's thought it had to be a bot. Fair enough.

I've been sent messages by the MindAlign support group that I needed to shut down all the MindAlign bots I had because I was spamming the authentication server with connection requests. I've checked, and we're talking about approximately 10 requests a minute. Hardly excessive, and certainly not the source of continued instability in the system.

Yet I get messages to shut them down.

But to do so is to update/patch several production systems - not the thing you want to do quickly or in haste, and since I've never run these production apps without chatting, it is going to be easy to miss one or two places where the chatting is done. So it's a significant production support risk - all for 10 hits a minute. Hardly seems worth it, so I haven't done anything about it.

Until today.

Today I got a call from one of the MindAlign support folks and he told me what his email had said. I then pointed out the risk to production systems and he actually started to laugh. Clearly, he knew this was a much a fool's folly as I. I even said so. We shared a little giggle and I told him that he had done his job and I'd look into it.

After all, 10 requests a minute is certainly reason enough to cause global chat instability. Hmmm... he didn't think so either, but confessed that he was just 'under orders' to pass the word. I told him his duty was done and he laughed at that as well.

In the end, I put in the changes as best I could, tested them as best I could, and will get the final few restarted this evening/tomorrow morning so as to shut off this spamming. But the problem will remain, won't it? Certainly, because these bots that are so important have been tested and re-tested so many times under so many conditions that they are so unlikely to be the cause of the problem it should be an acceptable risk to leave them on.

It's the bots that are new in the last few days, or those that might chat binary data. Not these that have been working fine for years. It's just silly.

But we're putting production services at risk for this. All for 10 requests a minute. And no one but myself and the guy that called and laughed about it seems to see how incredibly thoughtless this is. But that's how some things go - no one stands up to the Emperor who has no clothes.

Today has been the first day of really hammering on the new ticker infrastructure for any issues. There were several, and over the course of the day the Team Lead and I have figured them out. Currently, there are only a handful of differences, and those might be related to the timing of the configuration changes as they were done throughout the day. We'll have to wait and see what data tomorrow has. If I'm right, all these remaining differences will vanish and we'll be left with a clean set.

It's nice to make progress on something like this. It turns out a better product for our group and well as getting the global support group to have a better product for everyone.

I've been working a lot with MindAlign lately, and specifically with the case where the login credentials are not valid and the server returns 'ERROR:504' as opposed to the package that the protocol indicates. Since I hadn't seen anything like this in their spec, I have just been assuming that it's a protocol error and throwing it up the exception chain as a general error in the code I'd written for MindAlign in BKit and CKit.

Well... that's no good. So I took the few minutes to clean that up and create a generally more informative exception message saying that the authentication server didn't like the login. This isn't going to revolutionize anything, but it might save someone 5 mins debugging, and that's good enough.

Again today we're having problems with a key infrastructural communication system. Whose fault it is is really unimportant... we're not changing this component out anytime soon - it's a key component in a company that literally spans the globe. It's here.

That being said, it is interesting that today I get a mail from them saying it's going to be running in a restricted mode for the next week. This places a non-trivial squeeze on most of my applications. I use this in every app to communicate with it - control it... monitor it... check health... lots of stuff. So to be without it for another week is non-trivial and more than a little annoying.

Sa as I was saying, they are asking us if we can migrate to a less-secure version that we used to all be on about a year or so ago. It was a multi-year effort to get everyone migrated to the new secure product, but they want to know if we can move back in a few days. What? Oh... yeah, that's right... when you can't fix your own stuff, the best ploy is to ask the users to change and hope that they can change fast enough not to get sick of the outage and decide to move off your product for good. I mean, can these guys be sincere about this?

For me, it's less than a dozen applications, and in some cases the changes are simple configuration changes and some are more structural. But if we change to the old system it's going to mean a ton of logistics... new login credentials for each of the 120 connections I have... new names of channels... lots of chances to get something messed up in the change.

That's why it took more than a year to get the move up right. It had to be sure not to break anything. I don't know what the problem is with their system - it could be really tough to figure out. But to get hundreds of applications reconfigured only to reverse it when they got it all cleared is a little more than a simple exercise. It's silly.

They need to find and fix the problem. If they have to call in smarter people, then do it. If they have to get in new servers, do it. I can think of any of a number of ways to hack the system to work in a pinch, but they aren't asking me and I'm not offering unsolicited advice. But the mere idea that we can easily manage the risk of a global change like this in a few days is absurd. It'd take weeks, and they had better be able to fix this problem faster than that.

Well, there's certainly been a lot of noise on the net recently about Apple's iPhone/iPod Touch App Store. Good, Bad, Angry, Happy... all over the map. While I've only done a little bit of playing around on the App Store, I do have a few ideas on the App Store now. Overall, nat bad, but there's a lot Apple could do to make it better.

First, because I have the Touch, there's not a lot of apps that I'm going to buy because I leave the WiFi off most of the time to conserve battery power, also there aren't a ton of places where I'll have a Hot Spot and not pull out my laptop instead of my iPod. So it's got limited utility to me, but still, there are a few nice things, and as a sales tool for up-selling to the iPhone the Touch is a nice entry point.

So... the thing that seems to be the most significant issue is the fact that developers can't control the timing of releases to the App Store and then to the users. It's a nice idea to have the App Store handle all that, but the complaints I've seen and read certainly put Apple's speed of movement through the store as somewhat less than ideal. It's slow, and there's no way to give developers any idea of when their stuff will be out on the store and available for updating.

For someone who likes to make continual, small improvements to my code, this is a very bad thing. The delays in the publishing to the store require a different level of testing and certification by the developers. Face it, a lot of the Mac apps use Sparkle, or something similar to get automated updates out to their users pretty darn quickly. They control every aspect of the publishing. Now there's a publisher in the middle, and their speed isn't all that good. That makes a lot of the applications on the store look a bit dodgey. After all, what's it say to get an app that you and the developer knows is buggy, with a new release coming "soon" - whenever Apple gets around to posting it.

Not a good relationship builder.

I can see that there is the need for Apple to move more quickly, or allow for the direct downloading of applications from web sites - much like plugins and themes for Firefox. You download it and then it's there. Problem is, this allows for all those malicious web sites to download bad things and take over your phone... so I can see why they aren't too crazy about it.

There's also the unavoidable round of 'junk apps' - dozens of To Do apps and such. Hey, there's nothing you're ever going to be able to do about this stuff. It's new, it's there, and there are always going to be folks that are fast and trying to make a quick buck. Let it go... not even worth talking about. Darwin (natural selection) will weed them out and there's noting that solves this problem like time.

Then there's the legal battles over the simple act of talking about the SDK. This is something that has to be an oversight on the part of Apple and needs to be fixed to give developers a better feeling about Apple. Basically, some folks have read the SDK license agreement and said that GPL software can't be built for the iPhone because you have to be able to release details that are part of the NDA in the SDK license.

While I can understand this as a part of the initial SDK rollout, Apple can't expect to control people's conversations at dinner or over the net. They just aren't capable of doing that. But it's enough to scare people from talking about it in forums, etc. This is a very bad thing, and Apple needs to loosen up - like it has for the Mac OS X SDK. Let people talk. It's good for the platform. They have to know that they will get back a hundred-fold what they put out there - that's the way the developers are with Apple.

Yes, it's a new platform, and they don't want viruses, but they can't stop the virus writers, and so all they are doing is hurting those that can really help the platform.

So... it's got a lot of good things to like about the App Store, but a lot of things that need a little tweaking. I'm sure it'll get better, but it needs to make significant improvements in the publishing very soon or they will loose a lot of momentum they have generated with the launch.

It's been a bit of a day, as they might say. Plenty of issues from the Corporate Overlords (chat infrastructure group) about what I needed to be doing to shut off the requests my apps were making - going so far as to call it spamming... then there were several meetings that seemed to have marginal utility to me, but maybe they had more utility to others, I don't know.

Thankfully, there was a little coding in there, and I was kept busy with a new pricing infrastructure update that's on the way, but for the most part, I just found it hard not to think about a nice, beach vacation.

Yeah, one of those where you do a lot of sitting around... basically, a lot of nothing. Sure, I just got back from a nice vacation in Orlando, but that was a lot of work. Lots of walking, seeing, doing... not a lot of resting.

Next time - lots of nothing.

I just got finished fixing a problem that I should have seen coming. Yesterday, due to some chat server issues, I needed to update an app of mine to get a little fix for a socket problem so I did a standard deployment. Problem there... the development codebase has been moved to a different version of a data provider's API, and so the deployment to production totally hosed the installation. Yikes!

OK, time to be clever... checkout everything on the last good build of the server, rebuild it and ship it out. Whew! it worked. That is... until this morning.

This morning a user came by saying they could not get data from the server, and I checked and he was partly right. Some data could be obtained, and some couldn't. After a few tries, it seemed that the tables-of-tables data was the thing that was causing the server to crash. Not good. But what could it be?

The code hadn't been touched in ages (before the move to the new API), so that wasn't it. Then I thought it had to be in the libraries - there had to be something that was causing the shared library to mess up - maybe a bad header file. So I took the time to get all the headers up to date and then rebuilt everything again.

This time it was fine - and it makes perfect sense. There must have been a change in the CKTable's header that was in the shared library version that wasn't in the header file I was using and they two were causing the crash when the code hit it. I should have seen this coming and been much more careful about the updating of the libraries in the build the night before. Yes, it's not a mistake I'll make again.

We use MindAlign in the Shop to have a secure chat environment, such as it is. Over the last few days, it's been problematic to say the least. Today, in fact, they have narrowed down the problem to a global corporate loss of DNS due to MindAlign.

I am not going to bad mouth MindAlign, it's a package that the corporation has picked, and we're going to use it. Period. But today when I came into work I noticed (again) that almost all my production applications (servers and services) had one core/CPU hammering away doing seemingly nothing - but very very fast.

I found that once again we were having MindAlign problems and I decided that this morning I'd dig into the problem and find out why they were all spinning like mad. The first step on that path was to get a few stack dumps, and it's a lot easier to do that in Java than it is in C++, so I did a few thread dumps on an afflicted Java app and after three of them, it because pretty clear where the problem was - in the BKIRCProtocol object. That's the thing that talks to the IRC server, which is the core of MindAlign.

It appeared that the IRC protocol was trying to throw an EOFException because the socket was not returning any data, and was, in fact returning an error code indicating that it wasn't all there anymore. The problem was, that at the higher level, where the EOFException was being caught, I was assuming that the socket was closed - which is why I'd get the EOF condition in the first place. By making that assumption, a socket with an indeterminate state talking to the MindAlign server was "there", but not "there enough".

I put in an explicit disconnect() on the socket connection and that should take care of things. Simple one-line fix (with an accompanying 10 line comment to say why it's being done. Then I was on to the C++ version.

Thankfully, the code is very similar, but the socket communications isn't. But I dug into the code about the same spot - a socket that's almost dead, but returning that it's still connected even though it's not returning any data. What I found was code that looked like this:

  // now read up to the "\n" NEWLINE that the IRC server sends
  if (!error) {
    retval = mCommPort.readUpToNEWLINE();
  }

where readUpToNEWLINE() should return only when a line is received from the IRC server terminated in a NEWLINE (\n) character. The problem was, it was returning an empty string and that was all I needed to know. If it returns nothing, then the only acceptable reason is that there was a timeout. So, check for that, and all other conditions cause us to disconnect from the server and on the next pass, a new connection will be created.

Something like this:

  // now read up to the "\n" NEWLINE that the IRC server sends
  if (!error) {
    retval = mCommPort.readUpToNEWLINE();
    /*
     * It's possible that the data is empty - but the only way for
     * that to be acceptable is for a timeout to have occured. So,
     * if the data is empty and a timeout *didn't* occur, then we
     * need to disconnect this guy and the next pass through, we'll
     * be able to connect again and set things up properly - we
     * hope.
     */
    if (retval.empty() && (errno != ERR_READ_TIMEOUT)) {
      disconnect();
    }
  }

At this point, I think I have the MindAlign issues under control. There's still the issues with the corporate servers, etc., but I can't do anything about them, and we'll just have to wait for them to get fixed up properly by the support teams. It's just nice to have these fixes into BKit and CKit so that I won't get bitten by this problem in the future.

Twitter is nice when the applications you use post their updates on it, so that you get notified as soon as possible of the new release. Another wonderful such happening occurred today with MarsEdit 2.2.

This release has an improved text-clip manager for the images and files - very nice. It's got a few new defaults and supposedly a lot more speed. So much more speed, it seems, that the default is for the preview window to open when editing a post. This is a significant change as previously this was cautioned against due to the processing hit it would cause.

I've updated, of course, and am busy posting this on the new version. Hopefully, I'll be happy with the speed, but the one bug/feature that bothers me most is the minimum size of the composition window. I know he's not going to change it, but it would be nice if he decided to re-do the GUI and as a result, relaxed the window width.

That's all I want... just relax the minimum window width. I can do it in Interface Builder, but that's a step I'd really rather not take.

UPDATE: one of the most striking things is the launch speed. It's on par with TextEdit, and that's saying something. Very impressive indeed. I wonder what other speed improvements there are waiting for me to find.

Today I fired up OsiriX to add in the MRI images from Liza's hip and it told me that 3.2.1 had been released. Not as simple and neat as Sparkle, but not bad. I got the 32-bit version and then loaded up Liza's MRI images. Lots of images, as it turns out.

Since this is the first set of MRI images I've loaded into OsiriX, I was very interested to see how it would treat them, and what capabilities it'd have for looking at them. Well, I was certainly not disappointed. This is amazing.

The images can be viewed in a movie where the slices are sequenced based on the order in which they were placed on the media - which I only assume was the order in which they were taken. Very exciting. Looks just like those amazing images you see on TV when they "move" through the human body.

Then there's the 3D representation. Now I'm not doctor, but this stuff is very cool, and the body seems to take on a more machine-like quality when you look at it as this level. I can see what my sister and brother-in-law said about surgery - it's not a person so much as a machine you're working on. Makes it possible to remain a little detached and therefore better prepared to handle the problems that might arise.

Anyway, get it if you have any x-rays or MRI images. It's amazing.