What's it all about, Alfie?

Today I was trying to look at the chat interface of one of my apps that uses CKit and has a chat interface. I wanted to make sure that we'd be able to enable logging of individual price updates during the day because some of the support staff in London were getting a little panicked because they're used to this level of logging in another app I've written and frankly, they've come to depend on it.

So I was checking and realized that the chat interface was offline. Odd. It should work. Then it hit me... there was a change I'd made to the chat interface not so long ago (July 23, 2008: Lots of Trouble with MindAlign These Days). But I couldn't remember it. Thank goodness for WordPress! I found the entry where I'd changed the logic on the getReply() method on CKIRCProtocol. I'd had a problem of an infinite loop where I wasn't checking for an error when nothing was returned from the socket. So I changed the code to look like this:

    // now read up to the "\n" NEWLINE that the IRC server sends
    if (!error) {
        retval = mCommPort.readUpToNEWLINE();
        /*
         * It's possible that the data is empty - but the only way for
         * that to be acceptable is for a timeout to have occured. So,
         * if the data is empty and a timeout *didn't* occur, then we
         * need to disconnect this guy and the next pass through, we'll
         * be able to connect again and set things up properly - we
         * hope.
         */
        if (retval.empty() && (errno != ERR_READ_TIMEOUT)) {
            disconnect();
        }
    }

but that left me in the case today where the return code wasn't an error or a timeout - but in interrupt. What to do? Well... the point of an interrupt is that something caused it to bail, but it's not a problem of the socket per-se. So I needed to really change the code to read:

    // now read up to the "\n" NEWLINE that the IRC server sends
    if (!error) {
        retval = mCommPort.readUpToNEWLINE();
        /*
         * It's possible that the data is empty - but the only way for
         * that to be acceptable is for a timeout to have occured. So,
         * if the data is empty and a timeout *didn't* occur, then we
         * need to disconnect this guy and the next pass through, we'll
         * be able to connect again and set things up properly - we
         * hope.
         */
        if (retval.empty() && (errno == ERR_READ_ERROR)) {
            disconnect();
        }
    }

so that I'm sure to only bail when there's an error. Timeouts and interrupts don't count. This fixed up the problem and I got it out to the effect apps. Interesting little thing that turned up on linux. Solaris didn't have this issue.

After I got the WebDAV server working on my OS X 10.4 machine, the next thing was to get SSL working with a self-signed certificate so that I could do secure WebDAV to the box. I've done this on Leopard (10.5) and so doing it on 10.4 was not too bad, but there were a few wrinkles.

First, follow the directions for creating the certificate files in this Mac OS X Hints hint. Don't mess with the httpf.conf file as we're going to do something a little more 10.5-like there. But make the cert, and the private key and the request and then sign it all.

Then in the /etc/httpd/users/ directory use the following file for ssl.conf - theirs is missing a few things.

  #
  # This is the SSL config goodies
  #
  LoadModule ssl_module         libexec/httpd/libssl.so
  AddModule mod_ssl.c

  <IfModule mod_ssl.c>
    Listen 80
    Listen 443
    # Some MIME-types for downloading Certificates and CRLs
    AddType application/x-x509-ca-cert .crt
    AddType application/x-pkcs7-crl .crl

    # inintial Directives for SSL

    # enable SSLv3 but not SSLv2
    SSLProtocol all -SSLv2
    SSLPassPhraseDialog builtin
    SSLSessionCache dbm:/var/run/ssl_scache
    SSLSessionCacheTimeout 300
    SSLMutex file:/var/run/ssl_mutex
    SSLRandomSeed startup builtin
    SSLLog /var/log/httpd/ssl_engine_log            
    SSLLogLevel info

    <VirtualHost _default_:443>
      SSLEngine on
      DocumentRoot "/Library/WebServer/Documents"
      ServerName MACHINE
      ServerAdmin you@yourplace.com
      ErrorLog /var/log/httpd/error_log
      TransferLog /var/log/httpd/access_log

      SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
      SSLCertificateFile /Users/YOURNAME/Documents/certs/frosty/newcert.pem
      SSLCertificateKeyFile /Users/YOURNAME/Documents/certs/MACHINE/webserver.nopass.key
      SSLCACertificateFile /Users/YOURNAME/Documents/certs/demoCA/cacert.pem
      SSLCARevocationPath /Users/YOURNAME/Documents/certs/demoCA/crl

      <FilesMatch "\.(cgi|shtml|phtml|php)$">
        SSLOptions +StdEnvVars
      </FilesMatch>
      <Directory "/Library/WebServer/CGI-Executables">
        SSLOptions +StdEnvVars
      </Directory>

      # correction for browsers that don't always handle SSL connections well
      SetEnvIf User-Agent ".*MSIE.*" \
        nokeepalive ssl-unclean-shutdown \
        downgrade-1.0 force-response-1.0
    </VirtualHost>
  </IfModule>

Where YOURNAME is the login you put the certificates under, and MACHINE is the name of the machine directory that you put the specific certs in. It's pretty close to what he had, but there are a few differences and it's important differences.

With this, I have https: and WebDAV over SSL for the box. Nothing horribly hard about this, but it's nice to have it all working now. Just took a little time.

With all the interest in using WebDAV as a network storage protocol for things like dotMac and applications like Skitch, I wondered if it would be possible to get WebDAV working on my older Mac server sitting in my office at home. I have heard it's easy if you have 10.5 (Leopard) but I'm not really ready for that, and it was more for the interest of getting it going on the server and not for any real pressing need. If I had a pressing need, then I'd have it going on Leopard, as it's got everything ready to go. This was going to be interesting.

What I found was that there was a lot of people looking into this in the pre-10.5 days, and it wasn't really all that successful in the general sense. Several folks reported they got something working, but they had to disable all the security measures (like encrypted passwords) and that was simply unacceptable to me.

But I got it close working. The WebDAV service was running in the stock Apache from Apple, and I could get it to authenticate with security, and I even saw the files in the root of the WebDAV service. The problem was that I could not do anything with them. I ended up with complete WebDAV functionality.

I could not read them, or update them, or save files... it was like only a limited subset of the functionality I needed was in the WebDAV client that shipped with 10.4.x. And to that I guess I'm not terribly surprised. It would have been really nice, but I'm not surprised that in order to have this really work I'm going to have to be working with Apache2 and the WebDAV module there. [Really, it turned out to be a configuration problem that I really needed to look at the Apache logs to figure out. There was also a configuration problem on the client I was using (Cyberduck) and I needed to figure out the server before I could figure out the client. But I got both.]

But on the off-chance it's possible to get this finished, here's what I did. First, you need to edit the /etc/httpd/httpd.conf and where you see the lines:

  ...
  #LoadModule digest_module          libexec/httpd/mod_digest.so
  ...
  #LoadModule dav_module             libexec/httpd/libdav.so
  ...
  #AddModule mod_digest.c
  ...
  #AddModule mod_dav.c
  ...

edit them to remove the comments so that they become:

  ...
  LoadModule digest_module          libexec/httpd/mod_digest.so
  ...
  LoadModule dav_module             libexec/httpd/libdav.so
  ...
  AddModule mod_digest.c
  ...
  AddModule mod_dav.c
  ...

then, after the block of config that looks like this:

  <IfModule mod_dir.c>
      DirectoryIndex index.html index.htm index.php
  </IfModule>

add in the configuration for the WebDAV server component:

  #
  # This is for WebDAV
  #
  <IfModule mod_dav.c>
      DAVLockDB /Library/WebServer/WebDAV/Dav_lock
      Alias /webdav /usr/local/davroot
      DavMinTimeout 600
      <Directory /usr/local/davroot>
          Dav On
          AllowOverride None
          Options FollowSymLinks Indexes
          AuthType Digest
          AuthDigestFile /Library/WebServer/WebDAV.passwd
          AuthName "frosty"
          <LimitExcept GET HEAD OPTIONS>
              require user drbob
          </LimitExcept>
          <Limit GET HEAD OPTIONS>
              require valid-user
          </Limit>
      </Directory>
  </IfModule>

This ends the editing of the /etc/httpd/httpd.conf file, so you can save it. What will become important later is the location of the directories and files, and the AuthName as that will be used in the password creation command, below.

At this point, it's time to make the location of the 'root' of the WebDAV server. I've chose to hang it off /usr/local, but you can put it anywhere:

  cd /usr/local
  sudo mkdir davroot
  sudo chown -R www:www davroot

and we also need to make a place for the lock file:

  cd /Library/WebServer
  sudo mkdir WebDAV
  sudo touch WebDAV/Dav_lock
  sudo chown -R www:www WebDAV

We're getting close. The next thing to do is to create the password file for the user(s) mentioned in the configuration, above. The AuthName is used here, as is the user drbob as it appears above. To make the password file:

  sudo htdigest -c WebDAV.passwd frosty drbob

Finally, we're ready to restart Apache:

  sudo apachectl graceful

At this time, on Mac OS X 10.4.x (Tiger), the user will be able to login securely to the URL http://machine/webdav/, and they will be able to see the contents of /usr/local/davroot but they will not be able to save things there, nor will they [and] be able to see, modify, or [and] use anything that they see. It's almost like the initial protocol is supported, but the more advanced stuff the clients have updated, but the server has not. [Everything works as advertised.]

Like I said... if it gets serious, I'll put this on my Leopard box and be done with it. But there's nothing pushing me right now, and I may come back to this and try to see what I need to do to fix it. Hard to tell. But that's what I've done.

UPDATE: Holy Cow! I decided to look at the Apache error and access logs and I figured out the problem. The Alias line above needed to not have the trailing slashes and they needed to match. I removed it from the first argument and restarted and all of a sudden, things started working! I'm amazed. Transmit works, Safari works, the Finder works... everything seems to work just fine. I need to set up SSL on frosty, but that should be pretty easy and when that's done I'll have a secure channel to my home WebDAV! How cool is that?!

Cyberduck still tries to assume it's using Basic authentication, and I can see that, it's not trying Digest, but the WebDAV server understands that and deals with it nicely. It's amazing. Wow.

This morning I was checking on a few remote sites and decided to pull up Cyberduck to see how it was doing with the relative speed of access to my home network. When it launches, Cyberduck uses the Sparkle framework to check for updates, and there was a new version released - 3.0.2. Of course, I got it.

Now I have paid for Transmit, and I'll still use it from time to time, but I have to say that I like the minimalistic GUI layout of Cyberduck more. No unnecessary border, no wasted screen real-estate - that's what I like. But with 3.0.2 they've added a default Bookmarks view. All of a sudden, it's not as clean as it used to be.

There's a menu item to toggle the bookmarks on and off - that's nice, but you can't really get rid of the bar right below the toolbar. You used to be able to just have the toolbar and the list of files - very nice and clean. You drag things out and that downloads them, you drop them in and it uploads them. What could be simpler?

I've looked around the program and I can't find a way to remove the second toolbar. Maybe they'll get requests to make it optional - maybe even from me. If they can remove it I'll be a lot happier. I just want a simple interface... nothing fancy or anything I don't need.

After reading several posts in the Cyberduck forums, it's clear that I'm not the only one that thinks this is not a step in the right direction. So I'll bide my time. I'm guessing that there's going to be a compromise here about how the bookmarks are handled. For now, If I get rid of the toolbar, I have only the one I can't get rid of, and that's a reasonable number for me to deal with. Still... too bad.

This morning I read that there was an update to Scribbles - the new drawing app I'm looking at, to v1.2.3. The release notes said something about a bug in one of the brushes, which is nice to know they are paying attention and listening to users. I was able to pick up a nice wireless Mighty Mouse for my MacBook Pro, so I'm hoping to really start to use Scribbles soon, but it's going to depend on the task(s) at had as well. I've got to have something I need to do to really test it out.

I was talking to a few guys around the office today - specifically about how they had not passed a test in their series of certification exams that several more people need to take around this place. I've written about the evils of these vendor exams - written by people that aren't professional educators, graded in the most cursory manner - computerized for the most part, and the resulting problems this makes for people in the workplace that need these certifications for their jobs.

But that's not what I'm writing about now. No, this is something I hadn't realized I'd picked up along the way to where I am today. I'm talking about understanding the difference between "testing" and "fact checking".

What I mean by the first is that anyone, at any time can make a test for someone. We do it all the time as parents ("Did you knock this over?"), we do this as developers ("Is this the problem with the system?") and as companies - in the form of these certifications. They aren't fair, don't really concern themselves with "right" and "wrong" - they are questions, challenges, and tasks that we must overcome to someone else's satisfaction in order to achieve what they have promised we'd receive.

"Fact checking" is all about the "right" and "wrong" - Can a 3kg spherical lead ball dropped from 1m crack a 1cm thick piece of glass? That kind of stuff. It's all something that that is by it's very nature objective. And for the most part, dispassionate.

"Testing" is all about someone's passion - I guarantee it. The person that wrote the material the test is based on... the instructor... the test's author - someone in this chain cares a great deal about this test. And as such, it's not about you, the test taker to tell them what's "right" and "wrong" - they already believe they have a firmer grasp on that than you - which is why you're taking the test and they are administering it.

What surprised me today was that I knew all this, and my co-workers didn't. Then I got to thinking about it and I realized where it all came from - grad school. I have to hand it to grad school, it really taught me this lesson. You sit in a class of less than a dozen grad students, and the professor knows you all by name. He's spending a lot of time trying to make this material understandable to you all, and so there's a lot he's got invested in the exams (tests). You need to make sure that when you take the tests, you give him what he's looking for. Period.

Maybe the university is an easy place to learn this, I don't know. But I do know that I have learned it, and I don't think of myself as cynical on this point. I think of myself as recognizing the concerns, issues, and feelings of the people involved. There's nothing to gain by making the test so simple it's useless as a gauge of the material retained. It's also equally useless to make it so hard that it's nearly impossible to pass - no one will pass it and therefore it'll have no value.

So there's a compromise there - an optimization of sorts. Again, grad school helps me there more than I initially realized. It's taken me 20 years, but I'm really starting to look back fondly to that place. Who knows... maybe we'll take a road trip and see the campus.

Last week I saw an interesting video making the rounds which included a few comments about the legal ownership of data available on the web. For instance, flickr is nice, but in order to put one of my pictures uploaded there on my blog, I have to make a link to the flickr page with that image.

Makes sense from flickr's point of view - it's how they pay for things. But it brings into question - Who's data is is anyway? I mean really... if they can make that rule, and Yahoo's music service can go dark, then what guarantees are there that my images uploaded to flickr aren't going to disappear? Or find their way into someone's photo gallery?

Nothing. Absolutely nothing.

This is one of the reasons I've tended to stay away from these "repository" sites - or hosted source control like SourceForge, hosted Subversion, etc. You never know what your rights are, and what theirs are. Even if you don't plan on having something of value there, are you opening yourself up if you put something up there, later decide to remove it and then try to claim it's really "yours"?

I'm certain there are some cases and some sites where you're out of luck the second the first upload it complete. It's theirs. and nothing you can do is going to change that. You just loose.

With the fact that license/usage agreements can include language that essentially says "subject to change without notice", you can't even be assured that you'll be alerted if the rules of usage change while your data is hosted there. It's almost impossible for the non-lawyer to know what they legally can do and can't do. And saying "Hey, it's a picture of me" doesn't count for squat in a courtroom.

So it's tough for me to see what the best course of action here is. I've decided not to post a lot of things to flickr even though it's a decent service, and could be really useful, but it's that like to the flickr page that causes me to pause.

Skitch is different - they might very well go away, and that would be bad, but I'd be willing to pay got Skitch usage/storage. No question. I might be willing to pay for flickr if they relaxed the rules, but I don't think I can trust that they will.

For the most part, I'm using HostMonster, and that I know I own the data. It costs more, but it's worth it. It's my space and my stuff. I've thought about the dotMac and MobileMe, and if they stabilize it out, I might be willing to go that route, but again, it has to be clear that it's mine for me to really depend on it.

I know someday there's going to be a shake-out and these "free" services are either going to have to charge or be a lot more up-front about the ownership issues. But until that day comes, you have to be personally careful. Better safe than sorry.

I've been in a lot of different business situations in my life - at the bottom, in the middle, at the top, big place, small place, conservative place, educational institution - and still I have to say that I'm amazed when I run into a boss that seems to need to have his people agree with him on a course of action.

Case in point: several years ago I was building an interface to a new ticker source from a group in the larger bank. There was an issue about what data to retain, and I thought that the existing model was fine, but a manager (not mine) in the Shop decided that we needed to retain more - specifically, an additional price.

He explained it and while I understood that I would be forced to do it, I wasn't about to agree that it was a good idea - it wasn't. But that didn't matter. If I had wanted to stay in a position where I was the one making the decisions, I'd have stayed there. I moved for a reason, and this was one of the things I knew I'd have to live with, and I was fine with it. The problem was, he wanted me to think it was a good idea. He just kept at it, and at it to get me to see this as a good idea.

Finally, my manager said "Give it up... Bob's going to do it, but he's not going to agree it's a good idea." He understood. Good.

Today was right along these same lines - even the same manager was involved. They asked me to coordinate all the development of a particular feature. I got all that tested and coordinated the two other guys and they said their stuff was done and tested. So I told one user to kick the tires and see what they thought. Interestingly, there was a difference in the two ways this package was executing the code, so there was a difference to the user. Nothing big, in my mind - we'll get it fixed as soon as the vendor's rep gets back to us about how to make them work the same.

But this manager told everyone we hadn't completed testing on the additions and to hold off testing. Seemed a bit excessive to me, but he's the manager, so I simply wrote back - 'Hey, we tested it, didn't find this, so we'll fix it.' And then the "please agree with me" chain started.

Over the next hour, about half a dozen emails went back and forth about the true state of the testing. I tried to explain that we did test it. But if he wants to be the final tester before anything is shown to the users, then just say so. Yet he continued to want to get me to agree that we hadn't tested it properly.

OK guy, get over it. You want to be liked, then be liked. You want to be the Boss, then be the Boss. But don't expect to be able to be the Boss and have everyone like you. You asked someone (me) to make the call, and then didn't like the call. OK, make the call yourself, but don't expect me to say I made a bad call... I just made one you wouldn't make.

Yikes... just be a bloody manager and say what you want.

I like to stay up with the Camino project as well as Firefox, but I'm not yet convinced that the Camino Team is building a better browser than Safari or Firefox. I still use Safari for 90% of my work, but when that fails, I pull up Firefox that's always running and pull up the page. It's been a really workable solution for me so far.

But I have to admire the Camino team... they're still at it. So I check each new release to see if there's a better browser there. Getting better, but it's still not Firefox.

I've been working with a system today that has nearly driven me to drink with it's hidden features. I know each complex system will have them, but when you are paying millions of dollars for a serious system, you should have as few of these hidden gotchas as possible. I submit that the higher the quality of the product, the fewer the number of hidden gotchas it has.

Take a good messaging system like IBM MQ or Sun ONE MQ. Both have serious price tags, both do a serious job, and while I may not like the configuration of one, or the other, I have to say that there are so few hidden gotchas in these two products as to really make them almost perfect. Ideal? No... but they do exactly what they say with no ambiguity and very little learning curve.

In comparison to the product I'm working with. They have a feature where you can define supplemental fields to the existing fields (attributes) to the stock objects in their system. You can add strings, doubles, dates, etc. Seems reasonably flexible, until you have to actually have to use them.

Say you have an object that has a built-in value (attribute) called 'date', and you want to add a supplemental field for 'expiration'. You can define the 'expiration' field just fine - as a date, even. You can then use the fields like:

  print obj.date
  print obj.expiration    // <- WRONG!!

You cannot "just use" the field you've defined even though the system has all the data it needs to work this out. No, you have to read it through a special method:

  print obj.read_supp('expiration')

and writing is even worse. You have to scan through all the supplemental fields on the type of object and see if there's the one you want, and then scan the object itself for this 'id' as you can't look it up on the name.

In short, this is an enormous pain, and there's even more. Even if you define a supplemental field as a double, you have to store it as a string! All supplemental values are going to be strings. The only advantage the 'type' has in the definition is the input checking in the GUI. It's a joke. I'm stunned.

In a good product there aren't any gotchas like these. Things work in a similar and symmetric manner. Sure, it takes more work on the part of the developers, but that's what it takes to have quality software - work. I'm continually amazed that this product sells any copies at all. (which is why I'm not referring to it by name, clearly)

If I"m lucky, this will die soon enough.