What's it all about, Alfie?

An interesting thing happened yesterday that I, unfortunately, did not catch in time to correct. It turned out that the Weather Channel's web site wasn't updated at 7 AM and the previous day's data was still there. So... my INSERT failed. That's really pretty good news. The problem was that had I retried it a few minutes later, it probably would have worked. So what I did was put in the same kind of retrying I had in the beginning of the code when no HTML could be found, and this should work much better. It's interesting how this code is evolving - partly because of the moving target of web page scraping but also as error conditions arise how they are handled. It's interesting.

I also took the time to update sparky with the latest patches from Sun. Now that I have the script working pretty well, it's really pretty easy to do this and just watch for errors. The most common problem is that Sun doesn't allow free downloading of some of their patches - you have to be on their maintenance program. There is an occaisional patch that doesn't fit the hardware profile, but I haven't spent a lot of time trying to figure out how many of these are really applicable. I suppose I could, but that's not a high priority to me right now.

I put in a few new things into CIA this morning for Joel. I'm not sure if this is total folly or good, hard work. I guess if Joel pulls this off, then it'll be good hard work, if he doesn't then it's all been folly. Hard to say right now, but I'm not betting anything on it.

Anyway, the changes are pretty simple - count the number of pixels in the image that beat the filter and a few localization #define constructs so I don't have to do anything once I move the code to the ISP that's hosting it. Nothing major, but it's nice to not have to hassle with the relocation and Joel is convinced that this is the thing that's going to make it all happen. More power to him...

While reading up on my newsgroups this morning an interesting thread came across my path in one of the SGI newsgroups. It was talking about the GTK+ themes and how the default GTK look-and-feel is so much different than the Irix default window manager. Well... as it turns out, someone at SGI decided to take it upon themselves to make a GTK theme that made the GTK apps - gEdit, gimp, etc. look like they fit in the Irix window manager. The theme engine is called Buffy and has several pre-built themes corresponding to the different color schemes in Irix's 4Dwm. Interesting...

Now I'm no look-and-feel bigot. I like a lot of different UIs - in fact, there probably isn't one that I dislike, but I have to agree that the GTK apps just seem to stick out - and that's not optimal. So I decided to give this a try. Very interesting and very easy to get to work. Simply build it and then put the appropriate theme gtkrc in your home directory as .gtkrc and it's ready to roll.

No functionality change here, just a little more consistent look for the apps that use GTK. Nice.

I put in a new filter into CIA for Joel yesterday... basically, his idea is to have a user pick a representative pixel and then based on some quantization of that pixel (Hue, Red, etc.) create a filter based on that quantization centered around the value for that pixel. Nothing that earth-shaking, but it's certainly a nice addition to CIA. I put it in, and sent the code up to the web site where, it appears, Joel runs his data through. Interesting.

With the problems CIA is having, I'm really wondering if I need to hide it on the CVSweb display. I mean really... who's looking at this stuff? I've suggested it to more than a dozen people, but it doesn't seem to matter. I'd show the web site I built for Leslie, but that contains sensative information that I think we shouldn't have running loose in the world. But CIA is a good example of C and I might just open it up...

Well... I finally got all the information I needed from WebPipe (and Leslie) to get their web site up and running on the production host. What a wonderful group of folks! They have top-notch servers, darn good response time and the price is just too good to be true. With the database information I built the database and set the constants on the server-side PHP scripts to the proper values and Voila! it worked like a charm.

Well... I did notice a few style problems that were easy enough to fix, and I finally did a proper job of making sure all the references to North Hollow Farm were in the singular, but that's something I should have done a long time ago. Better now than wait for this to go live and have to worry about fixing them while it's live.

More good news is that the site seems to work fine with the counter from SoVer.net - which was a concern for me. I'm sure that WebPipe would have something to handle the page hits, but it's nice that we don't have to hassle with that for the time being. Lucky break we cought on that one.

I also mailed off to someone who sells SGI hardware a question I have about the maximum RAM I can fit into my Indigo². I have read conflicting reports, and I wanted to get the straight story from a good supplier. SGI says the max is 384MB, while I've read it can go to 640MB. If this is the case, then I can certainly see getting a memory upgrade from 256MB to 640MB to make things a lot faster. Right now, Netscape and CVS are a lot slower than I want, and I'm looking for something to give them a boost.

Then again... it's not going to be anything like the Blackbird x4 I get in a few weeks...

I started the process of moving naturalmeat.com from Cydian to WebPipe by going to Network Solutions (their registrar) and starting the move. I changed the technical contact to Leslie and then told her what to do with the email Network Solutions would be sending her. So it's on the way!

Leslie has checked on the web site, and likes it, so it looks like we're ready to go. Once the domain is changed, I'll go in to the constants and put naturalmeat.com in place of the WebPipe tag, but that's only cosmetic. The WebPipe site is up and ready to go, and I've done 98% of the work myself. No need to worry about this anymore... I can certainly handle it.

This morning in the shower I got an interesting little idea that I wish I could have done. I was going to get the First Chicago stock prices for as long as I was there - and then get the BankOne prices after the merger. Then I was going to highlight the ups and downs with events in my career and notice the correlations. As I recall, when bad things happened to me the stock took a tumble :). OK... maybe not every time, but often enough, and sometimes quite dramatically.

Anyway, this was thwarted by the lack of historical records on First Chicago stock prices on the web. Dissappointing. I saw it all in my head... nice little comic prose with an honest historical account of the stock price... Ahh well... too bad. But it still leaves a smile on my face.

This morning we had a problem with the weather scraping off the Weather Channel site. It seems they have done something that makes the wget 1.5.x fail on the grab. Interestingly, the getPage program I wrote based on the GNOME libghhtp library works fine. Wow... I'm impressed. Well... the latest wget 1.6 works fine and so for sparky I've simply removed the old wget I got from Sun Freeware and grabbed the source for wget 1.6 from the GNU archives. I built it and it works fine. On tux I've simply gone back to using my version and it works fine.

Interesting... very interesting.

On a somewhat related note I've been working for a while on a new feature for CIA. When I 'paint' a region I use a recursive function that does the job. Well... in some cases the recursion migh go very, very deep. On the CellAnalysis web site machines it causes a core dump. So, what I wanted to do was to un-'dig' the routine and make it an iterative one. Very similar to a loop for factorial as opposed to recursion. Well... it didn't quite work out that way, but almost as nice. The trick was to implement a looping algorithm that utilized a stack to save the points I was visiting, and in doing that we achieved what was desired - minimal system stack usage.

Interestingly enough, the routine is faster in that there aren't the overhead calls, and it uses less memory since only the relavent values are on the stack, so it's a pretty good win all around. Sure, the complexity goes up a little, but that's a reasonable trade-off for not having to worry about stack implosion.

Today I did almost nothing other than to design an icon on Irix for AbiWord. I wanted to help the SGI Freeware guys a little, and when I think I can do an icon for a program, I usually do and send them the files. It's nothing special, but it's what I try to do to help.

But then I had to deal with the ISP for CellAnalysis. What a goofy bunch of guys. I tell them to get the code for FFTW and build it. There are a lot of optimizations that are done in the build to get the fastest implementation possible on the target hardware. It's a good thing to so. They didn't. They got a set of RPM libraries, which didn't include the header files. So I can't build and they have to go back to find the right RPM with the includes. What a goofy situation.

I need something else to do... I'm waiting for people to do things and I need to keep my mind busy so I don't think about the job search.

UPDATE: - they installed the headers after I asked for them. Then I was able to build CIA with the FFT smoothing of the sample data. I emailed Joel to let him know that I'd done this, and what still didn't work at CellAnalysis (no PNG headers -> no PNG output from Ploticus) but that's not critical at this point. In the end, they got it right.

Today I spent some time migrating the CIA Uploading and processing code to PHP 4.04pl1 because there were differences that I hadn't taken into account in 4.04, and I wanted to get the code working again as Joel was having so much trouble getting his PHP code to work. So I did it. I haven't liked that BMP files don't display properly in Netscape, but that's another story - and in fact, I created a wonderful work-around in CIA that solves yet another problem - BMP file size.

You see, the original scripts planned on having only JPEG files. But they are lossy and BMPs are not. So CIA can read either and convert from one to another. So if you give CIA a BMP file it will generate a JPEG file so that the scripts can use it for presentation. The fact is that JPEGs are a lot smaller than BMPs, so even if they send a BMP we want to show them a JPEG. It'll display faster, and that's important.

So now no matter what is uploaded, a JPEG is generated for viewing purposes. This is a good step forward. Additionally, the comment Joel made about problems in PHP 4.04pl1 are not true - at least not generally true. I have use the PHP 4.04pl1 source and built it and it shows no signs of adding additional lines to the beginning of a file. None at all.

On the syrup front, it looks like my cousin will be going with Online Merchant Gold a package that does a complete eCommerce site for you for a nominal set-up fee and a recurring $70/mo with 4.95% of each sale. This is a reasonable set-up because it offers the Certificate and SSL which would be at least $35/mo and then there's the support you get and the database and software.... so it's a reasonable deal. They do all the card processing sending you orders and checks. Sounds nice.

I also got the easyDNS working with my domain TheManFromSPUD.com and converted all my machines to this new domain name. Very nice to have everything more stable and working well. I haven't tried sendmail, but now that I have DNS, I'll be able to do that if I so choose.

I've also taken the time to update all the web pages to remove the IP addresses and put in the machine names with the domain so I don't have to worry about updating those pages if/when the IP addresses from MediaOne change. I have to say that this is one of the better things I've done in the office for a while. It's nice and clean and cheap, and it's great to know that people can find my machines no matter where they are. Cool!

UPDATE: there's another eCommerce solution that Leslie is considering. Their present web hoster has a simple eCommerce option that might just do the trick. She's checking into it to see how they do card processing and shipping, but it sounds pretty interesting for the price.

Well... it's been a heck of a couple of days. I should have written more, but I was just so busy getting things working I didn't have time. Here's what I've been up to: I have built a development eCommerce site with PHP4 and PostgreSQL on tux for North Hollow Farm. It's very simple - you see the entire catalog (which for them isn't much). You select the quantities you want and go to the next page. On this page you check the order items and enter your shipping and billing information - you're in SSL now, and stay that way for the remainder. The next page is a complete order summary, and if you hit continue on this one you are presented with an order confirmation sheet. Your order is in the database, and an email has been sent to the Farm indicating a new order.

It's very simple, very clean, and very nice. I can't imagine where it'll break because it doesn't do a lot of the more complex eCommerce processing steps:

• Card Processing - not a chance, but it's certainly possible if I get hooked up with a card processing gateway. Since Leslie puts the transactions into QuickBooks and it clears them, this is not a bad thing not to have. Sure, if I was Amazon, then I'd need an automated system, but this isn't Amazon.
• UPS Tools - this might be nice, and it's certainly the next thing to look into, but what this would give us is more accurate shipping information and times. The problem is it's all in XML, and while my PHP4 has XML compiled, I'm not sure about the others. Also, there's getting into it and figuring out what they need, and what they provide. It's certainly on the wish list, but it's not required today.

All in all, I'd have to say that this is one of the better pieces of coding that I've done lately. It's nicely laid out, it's fast, simple, and there's almost no possibility that any reasonable web surfer can make a mistake or get lost in it. I've put in tons of well-worded error messages, and even added an administrative side to view orders and update the order status. It's pretty nice.

Now for some well deserved rest and let Leslie have a go at it. Tomorrow I have an interview and will be looking into the XML in PHP4 routines to try and get a handle on the UPS tools.

Yesterday I received yet another notice about an ISP that allowed MySQL/PHP/Apache services, but not PostgreSQL. I wanted to find out what I didn't know about MySQL so I started looking into it on barney which has it installed.

Well... I certainly know why they like it. The entire database is file-based. This makes for easy accounting - doesn't it. Since an ISP wants to limit the entire diskspace per client, it makes sense that a database that's file-based is a good thing. Well... the fact is, PostgreSQL is also file-based, but the ownership of the files is that of the process running the database. Is it possible to still calculate the total used space? Yes. But it's more difficult, that's true. So now it's clear why. Let's see what it costs us.

PostgreSQL is very much like any of the high-end commercial packages. Here are a few things that MySQL doesn't have that PostgreSQL does:

• Foreign Keys - this is almost hard to believe. They even say in their documentation that this is a good thing as FKs are difficult to deal with. Now I'm no database snob, but I can't believe that anyone who maintains business critical data thinks this. FKs are the key to referential integrity and without that you have no guarantees that the data you have is related to the key components for it's understanding. Imagine an order item with $1,000 without an order to go with it? Do you assume it's bad data, or do you try and find the customer that's missing a $1,000 item, or find where your receivables are? This is the kind of thinking that I simply can't believe they didn't do.
• Stored Procedures - if you can parse SQL - which they do, then you should be able to parse scripts. If you can do that you can handle stored procedures. There's nothing to it. While it may not be the most efficient language to use, a few additions to straight SQL make it very reasonable for SPs. I'm really amazed that they don't consider this to be a useful addition. But looking at their attitude towards FKs, this doesn't surprise me.
• Transactions - for a database to be SQL-based, I'm amazed that such a fundamental concept can be thoughtfully left out. While I can certainly understand that there are plenty of applications that don't need transactions, there really aren't many that are business critical that don't. This then places the target market for MySQL squarely at the web server, text logs, non-business critical database market. There really isn't any other way to look at all these omissions other than to realize that there is a different market for this database.
• Triggers - this is a little easier to believe because a lot has to go into handling triggers. You ought to have a transaction to have a trigger so you can roll it back if the trigger senses that something's gone wrong.

Combine this with the fact that PostgreSQL is every bit as fast as MySQL and you're in the position of choosing a solid, complete database versus a table/record database. I can't even image why someone you willingly choose MySQL. I can understand why certain shops don't offer PostgreSQL, but that may be as much education as anything else. We'll have to see.

I got word from the ISP for North Hollow Farm that the account was set up for the shopping cart experience. I went and checked it out. I also checked into the UPS tools like address verification, time tracking and package pricing and services. There is good news and bad news about each, I'm afraid.

Most importantly, the shopping cart is reasonably well done with just enough customization to be nice, while still making it wasy for a new person to use. This will be important because there's a lot of data to load/type in, and while I can physically do it, she might as well do it because she knows it, and the maintenance practice will do her good. The bad news is that the order flow is fixed, and there's no way to interject the UPS tools into the flow if they aren't done by the software manufacturer. This isn't really surprising to me, but it's unfortunate.

On the UPS software, even if we could have interrupted the order flow, there's a problem with the UPS tools that I hadn't counted on - XML. The documentation said that the tools were available in HTML and XML forms, but that's only true for the most simplistic of tools. If we needed to address verification or time tracking, then we'd have to have XML, and that means CGI at least, and that's a server configuration issue. Not a pretty sight, I promise you. So... it's probably just as well that the order flow can't be modified - I'm not sure we could have modified it anyway.

This brings me to an idea... I wonder if I should just do it myself and then see how it compares. The issue really isn't anything other than what's best for Leslie. I can't really do card processing, but I can get an interface and try... I haven't done SSL/HTTPS, but I can find out about it and try... I guess it comes down to having the time to try versus doing something else.

I think I'll wait and see what Leslie says. Depending on me for code is not necessarily in her best interests.