What's it all about, Alfie?

This morning I got the latest version of GAIM - 0.11.0pre6 and applied my patches to perl.c and built it. It had the same problem that I had seen earlier. So I decided to fix this guy. I looked into it and it failed at the exact same spot as before - in the call to XLoadQueryFont(). But I believed that this could not have been the problem as it was deep in GTK 1.2.8 which works wonderfully. So I searched on...

When I ran it in gdb or ddd, I sometimes got it to work. In fact, I could sometimes get it to work from the command-line. So I tried to turn off the optimization on aim.c and that seemed to help, but not 100% of the time. Then I noticed another problem that I've been having but thought nothing of - GdkSpell error.

One of the errors I had been getting is the GtkSpell error at the beginning of GAIM, and since I decided to fix all the problems starting at the top I thought I could fix this one and then get back to the original XFontQueryLoad() problem. So I found that while I had aspell and pspell, I needed ispell for GAIM. So I went to the Ispell web site (found it off of Google) and downloaded and installed it. Nothing much, but it's meant to be very cross-platform so there's no configure script, etc. Just hand-editing a few files. But inside of 20 mins I had ispell running.

All of a sudden GAIM worked all the time. I mean all the time. So I put the optimization on aim.c and it still worked. This was really good news. What I had stumbled upon is a problem in how they were handling the death of ispell when the user didn't have it. There must be a bug in there somewhere, but it could bt in GtkSpell or GAIM... I just don't care now. Since I have ispell it's fixed. Excellent!

I also talked to Joel this morning and he's reporting that with the latest changes I've made to CIA over 99% of the cell nuclei are being found. Good enough for me! I think that's a big high-five to myself for the work. We have to move it to the web hoster, but that's to come as we still don't have FFTW up there. I sent an email about doing it, and we'll see when it is installed. Lots of work to do there, but most of it is on the backs of the other guys so I can rest for a while.

Today is one of those special days when a lot of hard work comes to a head and you finally break though that brick wall you've been banging your head against for days, weeks, or in some cases, months. Yes, indeed, today I finally got CIA to recognize the cellular nuclei with a reasonable percentage. It was one simple idea - isn't it always? But that simple idea makes the work so much nicer.

Basically, what we needed to do was to isolate the fat from the non-cancerous cells. To do that we needed something that was similar to an edge-detector, but different enough that an edge-detector didn't work. I had tried it. So I came up with the idea of RMS error. Root Mean Square error - something from my old EE days would come to the rescue. I take the pixels in a neighborhood of the point in qiestion and calculate the RMS error in each of the three channels - red, green, and blue, and then save that for this point's value. The smaller the number the more 'constant' the image in the neighborhood of that point. Then I simply look for regions of small RMS error, and paint them with a color that will return a FALSE to the filter.

WHAM! Like a dream it worked. The fat was removed from consideration and the non-cancerous cells within the fat remained. It was really great. After that, it was a simple matter for the filter to do it's work and isolate the cell nuclei. I'm impressed. The idea was sound, the execution sound, and the results are great. "It is a good day for Science" as Dexter would say. I have to agree.

Over the weekend I spent a bit of time trying to get sendmail working, and only managed to get it working when I changed the hostname on sparky to that horrible name whilch includes the IP address from MediaOne. What it did show me was that with the proper DNS, it would work fine.

One of the reasons for getting sendmail working was to be able to email folks from within bidwatcher via pine. When I tried to configure pine to use SMTP, I kept getting errors that I couldn't see completly. This time, I decided to give it a little more effort, and ended up getting a configuration that works - pretty much. The address that I'm sending from still has my login name as opposed to the MediaOne email address, but with a Reply-To: header, I'm able to make it so that people can reply to that email address. Not perfect, but it should work nicely.

While I was on it I looked into the idea that bidwatcher could pull up Netscape's mail feature, as that would be the best for me. If that could work, then I'd be able to keep all my sent mail in one place. Also, there's no problem with the sent address when you use Netscape. So I dug into the code for bidwatcher.

I was surprised to see that the code for launching a new web page and pulling up the email composer was so similar. I simply cleaned up the browser code, and used a lot of it to clean up the emailing code. It worked - well... almost. The problem is that now eBay is not providing the email addresses to people - instead, you have to compose your message on their web page, and they send it for you. I'm sure this is because of the off-market deals that eBay is trying to crack down on. I wish I could get my exchange idea running... Anyway, with the fix in, it looks like there's no need as eBay has changed the rules. Too bad...

Today also saw several phone calls regarding headhunters, and other folks looking to see if I was right for their firm. Lots of nibbles but nothing serious yet. To keep my mind occupied I have been doing some more CIA work - this time adding the ability to get the background histogram based on the filter used to locate features. The idea is that after filtering, the Her2 slide is basically shades of light blue. What we're trying to do is differentiate the non-cancerous cells in the fat from the fat - eventhough they are colored very similarly. My hope is that by removing the stained portion from the image we'll have two distinct peaks - one for fat, the other for cell nuclei. Then getting the cell nuclei would be a multi-step process: filter out the stained matter, histogram the rest, pick a filter based on the histogram, and then extract features from the result. It's possible, but I'm not holding my breath in case it doesn't work.

The runs of CIA are getting long as well... certain runs are well past 30 mins. on sparky, which means there's almost no chance that this will be a real-time system to the user. Even with 1.5GHz P4 machines, the run time would be minutes, and that's just not acceptable for a hold time. I've told the guys about this, and Joel assures me that a good UI can be fashioned. I agree, but it's a submit/request one and not real-time, which is what we were going for in the beginning. Oh well... the best laid plans of mice and men...

Last evening I realized that I really didn't like the fact that from the Journal pages you couldn't get back to sparky's home page. Clearly, this is a UI problem, so I decided to add a menu bar at the top of all the pages to make it easier to navigate. The first version of this was to have the non-current pages appear in the menu bar. I didn't like this because it meant that the menu kept changing - which is bad form, I think. Then it hit me... have the current page appear in-place in a different color. This works very nicely. I can tell where I am and can get to every other place easily. Very nice addition to the site.

This afternoon I finally received the cable I purchased through eBay that converts 50-pin SCSI to 68-pin SCSI so that I can cable from sparky's CD-ROM to the 36GB Quantum Atlas IV external drive that I also purchased through eBay. When I had all the components I needed, I powered off sparky and went about adding in the drive.

It's pretty straight-forward, but I had forgotten that after I run format to partition the drive, I needed to run newfs to make the UFS file system. Oh, I knew I needed to run something to make the filesystem, but I had forgotten the command. A few hits on the man pages, and I found it. Nice, that Unix... Anyway, it took a little bit but I got everything moved around like I wanted. Basically, this new 36GB drive is /usr/local where I have almost all of the stuff that's continuing to grow - the apache site(s), the PostgreSQL databases, etc. So now they have so much room to grow it isn't even funny. Also, I've freed up a lot of space on /usr so it's not crowded any longer.

If I need it, I can certainly move my development directory to /usr/local/development with a link back to my home because most of the development stuff is tools and libraries - not code that I'm actively developing. Nice plan, if I need it. Right now, I've got plenty of room on all partitions and that's good enough for me.

I've also done some more CIA runs today for Joel. They are taking a lot more time than I had originally hoped. The latest run took 2002 sec. or about 33 min. 22 sec. That's not even remotely close to responsive even if we assume a six-fold increase in processing speed - which I doubt we'll get from the web hoster Joel's signed up with. So... it means that we are going to have to have an asynchronous method of operation. Something where the pictures/images are submitted, and processed, and later the results are generated and available for delivery/pick-up.

I'm thinking that this may not be such a bad idea... what if we made it such that you needed our client to send in the image and specific information that you, the user, have to provide to make it work. Then, we get a little help on the local end and also control who gets into our system. Just an idea... chances are, it'll still be better if we can keep away from any client-side utilities.

UPDATE: I ended up making a /usr/local/src and moving all the canned source out of my Development directory to that location. Now I've freed up a lot of space on the 4.5GB drive that is /user and put all the sources in a very reasonable place. Kudos, dude.

Well... now I think I've seen it all. I found a bug in CIA today regarding the loading of a BMP file - basically, I had the red and blue bytes for each pixel reversed. How this showed up, however, is a tale to tell...

I have been trying to come up with an effective filter for the nuclear matter of the cells on the 7561/Her2 sample - you can see this on my CIA homepage. Basically, I can easily get the stained material, but the docs also want the number of cells, and their size to do a membrane comparison. So we've been trying to nail down what to look for. Well... today I hit on a big problem - the filter seemed to be acting exactly opposite to what I intended. I had computed HSL components of several RGB values, and the fat should have a high Hue, but was being nocked out by the high Hue filter. This was very troubling.

I checked the threshold calculations - they were right... then I checked the filters... they were right, and finally I got specific coordinates from GIMP and used them. I found the RGB values were wrong, and within five minutes, I realized that they were wrong in a very specific way - red and blue were transposed. Once I fixed it in the code the filter acted correctly. It was amazing... the data was bad in just such a way as to make the results look as if I had the binary filter in the wrong direction. How incredibly odd.

Now that I have the BMP reader working properly, I'm going back to look at some of the runs I've done to see if the Hue-based filters that didn't seem to work now work because the data is more in line. Preliminary results show that it's looking good and we could have made a big break-through today. What a trip!

This morning I checked the Ximian web site to see what the status of Red Carpet was. Red Carpet is their maintenance tool for Linux that keeps your distributions up to date, along with Ximian GNOME and Red Carpet itself. Their previous work, Helix GNOME Updater, is an excellent tool that I've used for a long time. The initial release of Red Carpet had the problem of enforcing dependencies in Red Hat, and because of a foolish oversight in the making of the PostgreSQL 7.02 RPMs, I had to install them as --nodeps, which lead to problems. The update that I obtained today fixes that and several other things. Excellent work by the Ximian folks.

This afternoon I found out that the RealPlayer7 on my Linux machines has expired. So I go to Real Networks and see that they have finally started supporting Linux as well as they support Windows. There's a Linux RPM on their site for RealPlayer8 - the latest version. That's incredible news! So I get it, install it on my laptop, and test it. Looks good. Plays the content. I'd say I'm pretty happy now.

I upgrade my other Linux box, and then look at the Solaris/SPARC version. Uh oh... now we're getting more like the old Real Networks. They haven't updated the player for SPARC - it's still RealPlayer7, and the news from their support groups is that they have upgraded the UltraSPARC version, but they haven't upgraded the SPARC version. This is making me nervous because Real has the annoying habit of making their code expire, and that means that I need to see a new version of RealPlayer7 or 8 for SPARC really soon.

I then checked on the Irix situation, and again, if you run 6.5, they have RealPlayer 8. The problem is that it only supports the MIPS4 architecture - so barney is going to be out of luck there as well. It appears that Real is supporting the latest hardware eventhough older hardware is still very much in use. If I try to run the Real Player 7 for Irix 6.3, I get messages saying that it's can't decode certain content, and in reality, it doesn't decode any content at all.

So it seems as though sparky and barney are not considered to be viable hardware platforms by Real Networks. Well... I certainly understand their position, but I think they are as wrong as they can be. They need to embrace the Unix community to hedge against Microsoft and MediaPlayer. If they support Unix well then they have the ability to claim better interoperability, which is what a lot of web sites are going for. Look at Flash plugins... they're trying to support Unix, and it's done pretty well by them.

Anyway, the bottom line is that my Linux machines have RealPlayer8 and sparky has a lingering RealPlayer7 while barney doesn't have a functioning player. Gosh! I wish it were better supported.

Last evening I noticed that there was a new release of GAIM - the AIM/Yahoo/MSN instant messaging client that I run all the time. I downloaded it and patched the src/perl.c file to properly work with perl 5.005_03 on Solaris. You see, perl makes an assumption about the environment array - namely, that it is mutable. On Solaris/SPARC that is not the case, and moreover, there's no documentation/reason to believe that it should be. I've talked to the Perl folks, and they've patched their code but it's still in perl 5.005_03. So, I've got a work-around that I patch into GAIM (and other code that uses this same perl plug-in feature). Then I built GAIM.

What I found was that there was an error when I ran GAIM 0.11.0pre5:

        Gdk-ERROR **: BadShmSeg (invalid shared segment parameter)
          serial 61 error_code 131 request_code 131 minor_code 2
        Gdk-ERROR **: BadShmSeg (invalid shared segment parameter)
          serial 76 error_code 131 request_code 131 minor_code 2

that I traced down to a call to XLoadQueryFont() in the stack trace:

        gdk_font_load()
        gtk_style_new()
        gtk_widget_get_default_style()
        gtk_widget_peek_style()
        gtk_widget_init()
        gtk_type_new()
        gtk_window_new()

When I stepped through the call to XLoadQueryFont() I could occaisionally get it to work - but when it did, I didn't get the proper icons on the buddy list for expanding and collapsing the groups in my buddy list. I reverted to gaim 0.11.0pre4, and everything is fine.

I've posted a message on the GAIM SourceForge page, and we'll see if anyone comes up with something. I have a feeling that it's got something to do with the more lax Linux/i86 architecture versus the SPARC/Solaris platform. I've run into this with perl, and I can easily imaging that the 0.11.0pre5 is doing something that's throwing off the subsequent call to XLoadQueryFont(). In any case, I'm back to 0.11.0pre4 and it's working very nicely. We'll see what happens when and if I get an answer from the GAIM folks.

Some days I hate DHCP and MediaOne. OK... it's not DHCP, it's just MediaOne's use of it. If I had a fixed IP address then I could possibly work up a few DNS records somewhere that would point to the machines. Why's this important? Well... without forward and reverse DNS entries sendmail won't really function. At least I've tried, and the version on Solaris 7 isn't being helpful. I went through this on my NeXTSTEP 3.3 machine and got it working but the version of sendmail is changed and the configuration changes used in the version I have on NeXTSTEP don't do what's necessary on Solaris. Not that this is that surprising, but it'd be nice if sendmail had a simple configuration file default to hand off to another SMTP server. I know I could figure it out, but it's not that important, but it is a pain.

I tried sparky, barney and mao - representing Solaris, Irix, and Linux, and on each I was unable to simply mail out of the box because of sendmail. Oh... I can do it with Netscape because it allows me to communicate directly with the SMTP server, and put in a login to that SMTP server, and while sendmail seems to allow the specification of a great number of things, these don't easily pop up to be configured.

I know that if I had proper DNS I'd be able to handle this because the default configuration of sendmail on Linux does this fine - I've got it working on another host. It's just MediaOne who isn't allowing us to register names. What a pain...

For now I'm stuck with making my own mail client and sticking with Netscape. Looking at those two opportunities, Netscape is best.

UPDATE: I found a site called easyDNS that does exactly what I need. For $55/yr I can get a domain name registered (through them) and have their DNS servers hold the DHCP addresses of my machines. There are little clients for Linux/Solaris/Windows that send any DHCP-related IP changes to their service so that their DNS is constantly being kept up to date. With this, I could easily get DNS for my machines and then sendmail would be working just fine. Certainly something to think about.

Over the weekend I spent more than a little bit of time working on getting an old Jaz drive working on barney, my SGI Indigo2 machine. The SGI FAQ says that it's as easy as pie, but what I found was something different - at least for me. In the beginning all was going well, and then I heard what I believe to be the "Click of Death". After that, nothing I could do would bring the Jaz disk to life.

When I looked inside the drive mechanism, it appears as though the drive heads had been mashed. It was the oddest sight. I have no idea what caused the problem, but I'm worried that putting a new Jaz drive on barney will only act to increase it's hunger for more drives, so I decided not to try again. Besides, I really didn't need to the Jaz drive on barney - it was just a good use of an available resource to backup important files that sat on that machine.

Then this morning I checked out a few more web sites for possible jobs. My search hasn't been as successful as I'd hoped, but thankfully there's no hurry. What I'm looking for in the meantime is an interesting project to keep my mind occupied and my skills sharp. I've been working on CIA pretty much non-stop for the last several days, and need a break. Don't know what it'll be, but I need to find something to balance it out.

I have been working on the Journal through the pgaccess interface, and while it's good enough to get the job done in a pinch, it's not nice enough for me to be able to make the kinds of updates that I want to. So... I looked into making a pgaccess form and seeing if that would solve the usability problems. No good. So I decided to see if PHP could do the trick. I was in luck.

The key for me is the security. I wanted to be able to do this on the web, but I also wanted to secure it so that I'd be the only one able to modify the journal. Thankfully, PHP gets the remote browser's IP address, and saves it. This means that my two pages - the viewer, and this, the editor, can check that address and compare it to the address of the machine on which I'll do all the updates. Then, it was a simple matter of creating this update page and enhancing the viewing page to allow for updates coming in. Pretty simple, really.

The result is that I get a very nice editor for my journal entries. I can go back and edit entries, and add new ones as I see fit. No need to worry about URLs or other problems. It's nice and clean. I think this is going to be even nicer than the older version. What a deal!

Over the course of the last two days I've been adding an edge detection filter to the cell analysis code in the hopes of being able to better identify the cell nuclei. The belief from the Guys was that an edge detector followed by the existing feature extraction would easily give us representative nuclei. What I've found is, unfortunately, not good news.

I went searching on the net for any edge detection code. What I had originally found was that most of the good code was not available for just anyone. They wanted to make a profit on it. That's understandable, but it leaves me at a disadvantage. I still needed the code. I thought of the Sobel code in The GIMP, but didn't want to deal with the GPL that comes along with it. So I went back to the web. This time I got lucky.

I found a codebase called SUSAN. I also found an implementation of the base Sobel algorithm and one called Canny. With these two I was sure I could get a working edge detector going. And I was right.

I've cleaned up the code on the SUSAN algorithm and incorporated it into the base cia code. I've added a few arguments that indicate what channels to use for the gray scale conversion, and whether or not to output the edge detected image for viewing. In all, I'm pleased with this part of the work. What isn't so pleasing is the results, or lack thereof.

It seems that it really doesn't make a difference if I use edge detection based on RGB, Hue, Red - they all give results that aren't as good as just intelligent selection of Hue or Red. This is a shame, because Richard seems to think that it's easily done with programs he has. Alas, that's not what I'm finding out.