What's it all about, Alfie?

I had to change the script that processes the page from The Weather Channel's site because they changed the format of the information. Not a big deal, but when you're scraping stuff off the pages, it's a constant battle to stay abreast with their changes. It would be much nicer if they simply offered a database link that I could connect to... 🙂 In any case, the change was pretty easy to make once I knew what the problem was.

Then I received a call back from one of the headhunters I'd been dealing with in my latest round of job searches. The company I had interviewed with a few days ago was still uncertain about me - and the only negative things they could think of were the fact that I didn't wear a suit, and what they considered to be my limited experience with C++.

Now I don't usually rant in this journal, but here's something I'm going to rant about... I've been doing C++ work for over 15 years - OK, not every day all 15 years, but I don't write Java every day, all day, and I'm still outstanding at it. Yes, I've taken a few years off from C++ - sticking primarily to C and Java and web tools, but I'm as good in C++ as I ever was. More importantly, the technical interviews I've had with this company have all seemed quite fine with my technical skills. So much so, that they've said to my face the C++ isn't an issue. But to the headhunter they say different.

So it's one of two things - they are looking for a reason to ding me, or they aren't honest to me or the headhunter. Either is bad. Both means I don't like them.

Then there's the dress code. Now I'm not a slob - I'm very neat in appearance, and take great pride in a clean cut appearance. I bathe regularly, brush my teeth twice a day, and make sure to wear clean, pressed, clothes.

But not a suit.

I don't mind if others want to wear one, and I'll admit that there may be deep-rooted emotional damage in my background that is the root cause for this aversion to suits, but I just won't wear them. I don't own one. I can't image buying one. I own three ties, and only wear them and a sports coat in times of death and marriage. I look very nice in a button-down collar Lands End shirt and slacks. I always have polished shoes, and never try to pass off "hiking boots" as business shoes.

But that wasn't good enough for this one person. No... he was surprised that I didn't wear a suit to meet with the President of the company. Now mind you, the President himself wasn't surprised, it's this other person... but that's beside the point. The person making this judgement upon me was unkempt, wore hiking boots, and for an IT Director had the most incredibly messy desk I've ever seen. While a certain amount of clutter is normal, once the clutter stands for more than a few weeks, it fails to have any value and just indicates sloppy organizational skills. This was the man passing judgement on my dress...

Thankfully, experiences like this make me all the more convinced that this is not a place I want to work. For if they make these judgements in light of their own incredibly poor showing, then it's a situation that's bound to be bad sooner or later. I'm a very neat and orderly person, and I couldn't care a whit about what someone wears - so long as it's neat and clean. But if this place, or even just this person, is making these kinds of judgements about me, and then not being honest to either me or the headhunter, then I'm glad they're only so-so about me. I'm certainly only so-so about them.

I'm willing to give everyone a second-chance. I think a lot of the problems we have these days is because people rush to judgement and don't give second-chances. So if this place were to call me and talk up-font about these issues, I'd probably be fine with it. After all, maybe he'd had a bad day and maybe he worshiped the ground the President walked on. I don't know. But I do know that one rush to judgement is fine - so long as it's talked out. Failing to talk it out, or making another is unacceptable for me.

And so the search continues...

Today I decided that I wanted to get into the more secure connections. Not that I think my systems are open to massive attacks, but there's a chance that something might be going wrong, and I want to at least have the tools to protect myself. So I got OpenSSL and OpenSSH and decided to figure them out.

The first thing is to build OpenSSL. It's easy and builds cleanly depositing itself into a reasonable place in /usr/local/ssl. Then build OpenSSH. If you've done it right, the defaults will make a working sshd - but you still have to get it to start up and shut down properly. So I spent a few minutes putting it into the chkconfig system on Linux and Irix, and then made a simple script for Solaris. Nice.

Now I have SSH running on all of the machines that touch the world, and that's good. I also have the clients on several others which is even better. Now I can get to places a little more securely. One of the nicest clients is PuTTY for Windows 95/98/NT/2000 - excellent piece of code.

Today I decided to clean up the Her2 processing of CIA. Basically, we needed to have a controlled two-pass system - the first to gather the stats on the cell nuclie and the second to gather the stats on the stained membranes. This needed to be thought out before coding as the file names, as well as database records, would tend to walk all over each other. With a little help from this journal's design I decided to be smart about it and have a single command-line argument (-m) indicate to CIA that it needed to do both passes.

First, I had to differentiate which pass I was on - that was pretty easy, actually. Then when I had that, I just modified the names of the output files to include a "_nuc" in the middle of the name for the first, 'nuclear', pass. This worked out great. Then I made the main code fork off a second process (could have used multiprocessors here but opted against that) and have it do the first pass, then when it finished, the second pass on the stained matter would work. In truth, there's no conflict between the passes, and it could have easily been done at the same time. It's an easy change, and if we get the processor horsepower I can do that. But for now, it's serial.

When I ran this it all worked beautifully. I was really happy that I could easily add in the default options when dealing with a membrane sample - like ditch the aspect ratio, etc. Very nice indeed. So now it has automatically generated filters on both passes, and everything is ready for production data. I'm pretty happy with the clean-up.

I'm still unable to build this on the production site, but that's up to Joel as much as anything - they don't seem to be as responsive to me as to him, so I'll just send on what I need them to do to him and let him deal with it. I was trying to get Ploticus to compile, but needed the XFree86-devel RPM loaded, and so I sent a message off to him. After Ploticus, then we need FFTW and then we can build CIA.

Today I got the latest GAIM release - 0.11.0pre7 in the hopes that the MSN section would be fixed and then MSN, Yahoo, and AIM would all be working again. Well... the MSN section is functioning fine, but the Yahoo code is now generating a crashing error about memory allocation that I believe is related to the GDK/GTK code. When I try to login to Yahoo, I get "Out of memory during ridiculously large request." I've tried to track this down, but due to timing issues with the login, I can't seem to get it pinned down.

This is further complicated by the horrible error message that is given to me. When I try to narrow it down in gdb or ddd I get nothing. So I've sent off messages to the GAIM folks, and I hope they can shed some light on this. I'm getting a little tired of all this breaking, but I understand that this is what some folks do when they release code. What I need to do is to get things working and then stay on that release until I hear from these guys that they have a significant, stable release, then I can move to it.

UPDATE: - I've taken the time to track down all the possible issues and found the problem - sloppy coding by the developers. There is a routine called yahoo_parse_config() where a (char *[]) array is getting parsed for configuration values. This code makes two mistakes - first, is doesn't check at the top of the loop for empty strings, and secondly, it assumes that the result of a strchr() is going to be not-NULL. They then use this assumed value in a calculation for the size of a malloc-ed array. Bad Style. When I added the checks for the empty strings and then for the result of the strchr() everything worked. It's amazing when you look at it. All these problems could have been solved with just a little defensive programming. What a shame.

Anyway... I submitted a fix for both problems to the maintainers and we'll see if it gets incorporated into their code.

This morning I got the latest version of GAIM - 0.11.0pre6 and applied my patches to perl.c and built it. It had the same problem that I had seen earlier. So I decided to fix this guy. I looked into it and it failed at the exact same spot as before - in the call to XLoadQueryFont(). But I believed that this could not have been the problem as it was deep in GTK 1.2.8 which works wonderfully. So I searched on...

When I ran it in gdb or ddd, I sometimes got it to work. In fact, I could sometimes get it to work from the command-line. So I tried to turn off the optimization on aim.c and that seemed to help, but not 100% of the time. Then I noticed another problem that I've been having but thought nothing of - GdkSpell error.

One of the errors I had been getting is the GtkSpell error at the beginning of GAIM, and since I decided to fix all the problems starting at the top I thought I could fix this one and then get back to the original XFontQueryLoad() problem. So I found that while I had aspell and pspell, I needed ispell for GAIM. So I went to the Ispell web site (found it off of Google) and downloaded and installed it. Nothing much, but it's meant to be very cross-platform so there's no configure script, etc. Just hand-editing a few files. But inside of 20 mins I had ispell running.

All of a sudden GAIM worked all the time. I mean all the time. So I put the optimization on aim.c and it still worked. This was really good news. What I had stumbled upon is a problem in how they were handling the death of ispell when the user didn't have it. There must be a bug in there somewhere, but it could bt in GtkSpell or GAIM... I just don't care now. Since I have ispell it's fixed. Excellent!

I also talked to Joel this morning and he's reporting that with the latest changes I've made to CIA over 99% of the cell nuclei are being found. Good enough for me! I think that's a big high-five to myself for the work. We have to move it to the web hoster, but that's to come as we still don't have FFTW up there. I sent an email about doing it, and we'll see when it is installed. Lots of work to do there, but most of it is on the backs of the other guys so I can rest for a while.

Today is one of those special days when a lot of hard work comes to a head and you finally break though that brick wall you've been banging your head against for days, weeks, or in some cases, months. Yes, indeed, today I finally got CIA to recognize the cellular nuclei with a reasonable percentage. It was one simple idea - isn't it always? But that simple idea makes the work so much nicer.

Basically, what we needed to do was to isolate the fat from the non-cancerous cells. To do that we needed something that was similar to an edge-detector, but different enough that an edge-detector didn't work. I had tried it. So I came up with the idea of RMS error. Root Mean Square error - something from my old EE days would come to the rescue. I take the pixels in a neighborhood of the point in qiestion and calculate the RMS error in each of the three channels - red, green, and blue, and then save that for this point's value. The smaller the number the more 'constant' the image in the neighborhood of that point. Then I simply look for regions of small RMS error, and paint them with a color that will return a FALSE to the filter.

WHAM! Like a dream it worked. The fat was removed from consideration and the non-cancerous cells within the fat remained. It was really great. After that, it was a simple matter for the filter to do it's work and isolate the cell nuclei. I'm impressed. The idea was sound, the execution sound, and the results are great. "It is a good day for Science" as Dexter would say. I have to agree.

Over the weekend I spent a bit of time trying to get sendmail working, and only managed to get it working when I changed the hostname on sparky to that horrible name whilch includes the IP address from MediaOne. What it did show me was that with the proper DNS, it would work fine.

One of the reasons for getting sendmail working was to be able to email folks from within bidwatcher via pine. When I tried to configure pine to use SMTP, I kept getting errors that I couldn't see completly. This time, I decided to give it a little more effort, and ended up getting a configuration that works - pretty much. The address that I'm sending from still has my login name as opposed to the MediaOne email address, but with a Reply-To: header, I'm able to make it so that people can reply to that email address. Not perfect, but it should work nicely.

While I was on it I looked into the idea that bidwatcher could pull up Netscape's mail feature, as that would be the best for me. If that could work, then I'd be able to keep all my sent mail in one place. Also, there's no problem with the sent address when you use Netscape. So I dug into the code for bidwatcher.

I was surprised to see that the code for launching a new web page and pulling up the email composer was so similar. I simply cleaned up the browser code, and used a lot of it to clean up the emailing code. It worked - well... almost. The problem is that now eBay is not providing the email addresses to people - instead, you have to compose your message on their web page, and they send it for you. I'm sure this is because of the off-market deals that eBay is trying to crack down on. I wish I could get my exchange idea running... Anyway, with the fix in, it looks like there's no need as eBay has changed the rules. Too bad...

Today also saw several phone calls regarding headhunters, and other folks looking to see if I was right for their firm. Lots of nibbles but nothing serious yet. To keep my mind occupied I have been doing some more CIA work - this time adding the ability to get the background histogram based on the filter used to locate features. The idea is that after filtering, the Her2 slide is basically shades of light blue. What we're trying to do is differentiate the non-cancerous cells in the fat from the fat - eventhough they are colored very similarly. My hope is that by removing the stained portion from the image we'll have two distinct peaks - one for fat, the other for cell nuclei. Then getting the cell nuclei would be a multi-step process: filter out the stained matter, histogram the rest, pick a filter based on the histogram, and then extract features from the result. It's possible, but I'm not holding my breath in case it doesn't work.

The runs of CIA are getting long as well... certain runs are well past 30 mins. on sparky, which means there's almost no chance that this will be a real-time system to the user. Even with 1.5GHz P4 machines, the run time would be minutes, and that's just not acceptable for a hold time. I've told the guys about this, and Joel assures me that a good UI can be fashioned. I agree, but it's a submit/request one and not real-time, which is what we were going for in the beginning. Oh well... the best laid plans of mice and men...

Last evening I realized that I really didn't like the fact that from the Journal pages you couldn't get back to sparky's home page. Clearly, this is a UI problem, so I decided to add a menu bar at the top of all the pages to make it easier to navigate. The first version of this was to have the non-current pages appear in the menu bar. I didn't like this because it meant that the menu kept changing - which is bad form, I think. Then it hit me... have the current page appear in-place in a different color. This works very nicely. I can tell where I am and can get to every other place easily. Very nice addition to the site.

This afternoon I finally received the cable I purchased through eBay that converts 50-pin SCSI to 68-pin SCSI so that I can cable from sparky's CD-ROM to the 36GB Quantum Atlas IV external drive that I also purchased through eBay. When I had all the components I needed, I powered off sparky and went about adding in the drive.

It's pretty straight-forward, but I had forgotten that after I run format to partition the drive, I needed to run newfs to make the UFS file system. Oh, I knew I needed to run something to make the filesystem, but I had forgotten the command. A few hits on the man pages, and I found it. Nice, that Unix... Anyway, it took a little bit but I got everything moved around like I wanted. Basically, this new 36GB drive is /usr/local where I have almost all of the stuff that's continuing to grow - the apache site(s), the PostgreSQL databases, etc. So now they have so much room to grow it isn't even funny. Also, I've freed up a lot of space on /usr so it's not crowded any longer.

If I need it, I can certainly move my development directory to /usr/local/development with a link back to my home because most of the development stuff is tools and libraries - not code that I'm actively developing. Nice plan, if I need it. Right now, I've got plenty of room on all partitions and that's good enough for me.

I've also done some more CIA runs today for Joel. They are taking a lot more time than I had originally hoped. The latest run took 2002 sec. or about 33 min. 22 sec. That's not even remotely close to responsive even if we assume a six-fold increase in processing speed - which I doubt we'll get from the web hoster Joel's signed up with. So... it means that we are going to have to have an asynchronous method of operation. Something where the pictures/images are submitted, and processed, and later the results are generated and available for delivery/pick-up.

I'm thinking that this may not be such a bad idea... what if we made it such that you needed our client to send in the image and specific information that you, the user, have to provide to make it work. Then, we get a little help on the local end and also control who gets into our system. Just an idea... chances are, it'll still be better if we can keep away from any client-side utilities.

UPDATE: I ended up making a /usr/local/src and moving all the canned source out of my Development directory to that location. Now I've freed up a lot of space on the 4.5GB drive that is /user and put all the sources in a very reasonable place. Kudos, dude.

Well... now I think I've seen it all. I found a bug in CIA today regarding the loading of a BMP file - basically, I had the red and blue bytes for each pixel reversed. How this showed up, however, is a tale to tell...

I have been trying to come up with an effective filter for the nuclear matter of the cells on the 7561/Her2 sample - you can see this on my CIA homepage. Basically, I can easily get the stained material, but the docs also want the number of cells, and their size to do a membrane comparison. So we've been trying to nail down what to look for. Well... today I hit on a big problem - the filter seemed to be acting exactly opposite to what I intended. I had computed HSL components of several RGB values, and the fat should have a high Hue, but was being nocked out by the high Hue filter. This was very troubling.

I checked the threshold calculations - they were right... then I checked the filters... they were right, and finally I got specific coordinates from GIMP and used them. I found the RGB values were wrong, and within five minutes, I realized that they were wrong in a very specific way - red and blue were transposed. Once I fixed it in the code the filter acted correctly. It was amazing... the data was bad in just such a way as to make the results look as if I had the binary filter in the wrong direction. How incredibly odd.

Now that I have the BMP reader working properly, I'm going back to look at some of the runs I've done to see if the Hue-based filters that didn't seem to work now work because the data is more in line. Preliminary results show that it's looking good and we could have made a big break-through today. What a trip!

This morning I checked the Ximian web site to see what the status of Red Carpet was. Red Carpet is their maintenance tool for Linux that keeps your distributions up to date, along with Ximian GNOME and Red Carpet itself. Their previous work, Helix GNOME Updater, is an excellent tool that I've used for a long time. The initial release of Red Carpet had the problem of enforcing dependencies in Red Hat, and because of a foolish oversight in the making of the PostgreSQL 7.02 RPMs, I had to install them as --nodeps, which lead to problems. The update that I obtained today fixes that and several other things. Excellent work by the Ximian folks.

This afternoon I found out that the RealPlayer7 on my Linux machines has expired. So I go to Real Networks and see that they have finally started supporting Linux as well as they support Windows. There's a Linux RPM on their site for RealPlayer8 - the latest version. That's incredible news! So I get it, install it on my laptop, and test it. Looks good. Plays the content. I'd say I'm pretty happy now.

I upgrade my other Linux box, and then look at the Solaris/SPARC version. Uh oh... now we're getting more like the old Real Networks. They haven't updated the player for SPARC - it's still RealPlayer7, and the news from their support groups is that they have upgraded the UltraSPARC version, but they haven't upgraded the SPARC version. This is making me nervous because Real has the annoying habit of making their code expire, and that means that I need to see a new version of RealPlayer7 or 8 for SPARC really soon.

I then checked on the Irix situation, and again, if you run 6.5, they have RealPlayer 8. The problem is that it only supports the MIPS4 architecture - so barney is going to be out of luck there as well. It appears that Real is supporting the latest hardware eventhough older hardware is still very much in use. If I try to run the Real Player 7 for Irix 6.3, I get messages saying that it's can't decode certain content, and in reality, it doesn't decode any content at all.

So it seems as though sparky and barney are not considered to be viable hardware platforms by Real Networks. Well... I certainly understand their position, but I think they are as wrong as they can be. They need to embrace the Unix community to hedge against Microsoft and MediaPlayer. If they support Unix well then they have the ability to claim better interoperability, which is what a lot of web sites are going for. Look at Flash plugins... they're trying to support Unix, and it's done pretty well by them.

Anyway, the bottom line is that my Linux machines have RealPlayer8 and sparky has a lingering RealPlayer7 while barney doesn't have a functioning player. Gosh! I wish it were better supported.