What's it all about, Alfie?

Today I spent a good chunk of the day creating a page in my web app from two other pages. It's a request I received from a user that wants to see both views of the data, but wants to pack it in on the display - screen real estate being what it is on most desks. What I kept thinking while I was doing this is: Why isn't this easier?.

I'd love to see nib-ware for the web. Something where I can build a self-contained "widget" that I can drop into pages with a few personalizations. I suppose it's possible with the Google Visualization Widget toolkit, but I didn't think of that until just now. Maybe I'll look into it. What did come to mind was Interface Builder nib-ware: drop-in boxes of functionality that can be customized and then used over and over. The low-level details are all handled in the implementation of the widget.

This gets back to the idea of having an Interface Builder for HTML/JavaScript/AJAX systems in the first place. Something where I can create functions (or at least stubs of functions) and then hook the outlets and actions up so that a clicking of a checkbox fires of some action like hiding an element, or clicking on a button grabs the value of a textbox and adds it to a list... stuff that you can do in IB, but can't really do it HTML.

Of course, the trick is that the rendering is the biggie. That's where all these JavaScript and HTML toolkits and frameworks come in. I guess it's getting there - slowly, but it'd sure be nice to be able to use something like IB for this. It sure would make it a lot easier.

Well... I've been doing a little playing with Coda, and while I have to say it's very nice, and it's got a ton of flexibility. I think it's the tool I've been looking for to do the larger scale, faster AJAX building and deployment. You can put the CSS for a web page in the page, which is quite nice, but you can also put it in a general CSS file, and share it among many pages. Nice flexibility.

Of course, with preview is Safari (WebKit), so the JavaScript is working in the preview, and while it's not exactly like Safari because I can't set my default fonts in the preview pane, it's probably a lot more representative of the typical user than I've been thinking my Safari and Firefox sessions are. For example, in Google Chrome, by WordPress blog looks very different - primarily in the font selection. I'm not sure which is right, because Safari, Firefox and Camino are all looking the same and it's only Chrome that's odd. Maybe it's just Chrome.

Anyway, it's probably a decent thing that the preview is very generic, as it'll force the proper use of CSS. That's important for a whole host of reasons, not the least of which is that these days you have to have CSS or the page looks horrible on just about any browser.

So I was thinking that with Coda, I had a really nice tool to build the pages. But then there's the complete object-oriented development tools. The plus there is that I have the Google DataTable in Java, and JavaScript, and that serialization is done very nicely, but in general, I really need to think about building up some JSON serialization tools for things like Maps and Lists in Java. It's not a complete 1:1 mapping, but there are a few things I could write that would make it a lot easier to serialize out some standard Java datatypes to the JavaScript world, and that would make it faster to build pages.

Still... writing the Java code isn't the long pole for AJAX development for me. It's more the initial building of the HTML page based on all the ways of putting it together. All the different tags and structures are more than a little challenging on multiple browsers, but it's something I'm getting used to, and in time, I feel, it's something that will come as second nature. But for now, it's the one thing that would benefit from a builder tool.

In the end, it's still one of the most enjoyable development environments I've coded in in a while. Glad that I have the tools to do it easily.

Adobe recently admitted to a nasty zero-day exploit in their Flash player that's already making the rounds of the nasty web sites. This, combined with my current need for Flash in my work, means that when Adobe announced a Beta of Flash player 10.1, I got it. No reason to be careless about attacks, and might as well get the latest and greatest Flash.

I'm still up in the air about Flash. In general, I don't like it on the web. Everyone uses it far far too much for animations and ads. It's annoying. Also, I've developed a little in it. Not the best platform I've seen, but not the worst. Yet still... the Google Visualization widgets are in Flash, and I really like them. So it's really Google that's saving my belief in Flash.

Today I spent quite a bit of time putting together a new page in my web app. I had a model to work off of, but there were enough new things I needed to work in that I got to stretch a little artistic muscle - thankfully, I didn't pull anything. But it got me thinking how much I'd like to have a layout editor that is sort of like the WebKit Inspector panel - in reverse.

I spent the majority of the time messing with the HTML to get the layout looking the way I wanted. I had to make it look nice in FireFox and WebKit (Google Chrome), which wasn't horrible, but it did take a lot more time than glueing it all together with JavaScript and the servlet backend. So it's got me to thinking a few thoughts.

First, I'm very interested in hitting Coda again and seeing how they handle the CSS. That could make it a lot faster right there. There's no way to do web design on multiple platforms without CSS. I can't imagine it. So... faster CSS means faster development. I'd like to hit Coda harder and see how it handles more complex pages.

Second, I'm interested in the idea that there might be a more accurate preview system. Maybe this is Coda, I'm not exactly sure, but I'm thinking that spending 15 sec. editing JavaScript followed by a minute of redeploying it, refreshing the web browser, checking it, going to the other box, repeating the process, all adds up. If the preview was more like the Interface Builder preview, then I'd know it was wired up and working exactly as I expected.

Third, I can see the great value in the JavaScript Frameworks like DoJo, etc. and now the buzz coming out about Apple's private framework for making iPhone web apps. These all make things nicer, but I'm looking for one to emerge as the standard - at least for the Mac community. There's too much to handle on a low-level to be doing this all over from scratch each time. Cross-browser compatibility among the biggies here. I just haven't taken the plunge - yet.

I guess that's about it for today. It's been a lot of fun because in general, once you get the HTML hammered out (the view), the controller (JavaScript) and model (servlets) are not too bad. Not a bad way to develop an app, really.

I have been running some tests all morning, and they'll continue throughout the day, but as I'm watching these tests and gathering data, it occurred to me that I wanted to know which of my data injectors was giving me the most data, and by how much. It's not hard to get the counts with SQL:

  SELECT portfolio, COUNT(*) AS hits
    FROM PortfolioData
   WHERE acquired > '2009-12-15'
  GROUP BY portfolio
  ORDER BY hits DESC

and I get a table that looks a lot like this:

...and I've truncated the table because the effect I was looking for is clear - there are a few of the portfolios that are contributing the vast majority of the rows to the table. The problem is, I can't see the real percentage each contributes to the total. And while I've shown the top 10, there are really more than 30, so it's not easy to do the percentage calculation in my head.

I googled a bit and came up with a really simple solution: include the subquery as the divisor in the select statement:

  SELECT portfolio, COUNT(*) AS hits,
         COUNT(*)*100.0/(SELECT COUNT(*)
                           FROM PortfolioData
                          WHERE acquired > '2009-12-15') AS percentage
    FROM PortfolioData
   WHERE acquired > '2009-12-15'
  GROUP BY portfolio
  ORDER BY hits DESC

and to this I get what I was looking for:

With this, I can now see that my top 4 (out of more than 30) contribute more than 54% of the rows in the table. That's significant, and it's nice to know. If I throttle back these four, I have a great deal of control over the total number of rows inserted in a day.

Cool.

Well, this morning I went to have a look at frosty, the iMac G3 I have in my home office that I use for hosting my CVS repos, Git repos, web services (including SSL and WebDAV). Well... it wasn't good. The drive had died, and rather than mess with getting another 160GB drive and building it up again, I decided to bring a Mac Mini that used to be the kid's computer out of retirement, get a new 500GB drive for it, get a monitor, and install Snow Leopard on it to make it up to date and a little easier to maintain.

What follows is the details of all the little things I had to do to get all the services running on this Mini. It's not too bad, but it's different enough from the old 10.3 install on the iMac that it warrants detailing the differences.

Getting the Base Tools

The first thing I needed to do after installing Snow Leopard on the new 500GB drive was to use the Time Machine drive from my iMac to pull in the basic apps and my accounts to the new box. Interestingly enough, this seemed to pull over the Developer Tools (Xcode, etc.) but upon closer inspection, come critical command-line tools like rlog and rcsdiff, so I had to install Xcode 3.2 from the Snow Leopard disk, and then use Software Updates to get it to Xcode 3.2.1. This got me the command-line CVS tools as well as the RCS tools I needed for CVSweb.

The next thing I needed to do was to update the box with the version of Git from the Google Code project. This was currently at 1.6.5.2. I have since seen that it's at 1.6.5.5, but so it goes... these minor updates come fast and furious. I'm sure in another few weeks, I'll update to the latest version at the Google Code site, but for now, this was the latest, and more than sufficient.

CVS pserver

The first thing I needed to get going was my CVS repository - with the pserver running on the box. This is just so critical to all the code I have it had to be the first thing I got working.

The first step wasn't too bad. I had good backups of the CVSROOT directory - which just happened to be /usr/local/CVSroot in my old server. So I put it back there, and made sure that the CVSROOT environment variable were defined on my account on the new box, and then start to work on getting the pserver going. Since all my old experience had been with xinetd, and I knew that wasn't on Snow Leopard, I needed to create a launchd configuration file for the CVS pserver.

Taking the example I had to PostgreSQL, I came up with the following that I placed in /Library/LaunchDaemons/cvspserver.plist:

  <?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
  <dict>
      <key>Disabled</key>
      <false/>
      <key>GroupName</key>
      <string>wheel</string>
      <key>InitGroups</key>
      <true/>
      <key>Label</key>
      <string>com.apple.cvspserver</string>
      <key>UserName</key>
      <string>root</string>
      <key>Program</key>
      <string>/usr/bin/cvs</string>
      <key>ProgramArguments</key>
      <array>
          <string>/usr/bin/cvs</string>
          <string>-f</string>
          <string>--allow-root=/usr/local/CVSroot</string>
          <string>pserver</string>
      </array>
      <key>Sockets</key>
      <dict>
          <key>Listeners</key>
          <dict>
              <key>SockPassive</key>
              <true/>
              <key>SockServiceName</key>
              <string>cvspserver</string>
              <key>SockType</key>
              <string>SOCK_STREAM</string>
          </dict>
      </dict>
      <key>inetdCompatibility</key>
      <dict>
          <key>Wait</key>
          <false/>
      </dict>
  </dict>
  </plist>

Of course, the /usr/local/CVSroot is supposed to be the location of the CVS repository, and it just happens to be mine, but if you have something different, use it. I've seen examples where folks create a 'CVS' user, and then use the directory /Users/cvs/CVSROOT, or something like that, so it's all in a "user", which does make it easier to move things around.

For me, it's Old School - the old BSD/Solaris/linux background that makes me think in terms of services and not necessarily users. But to each his own.

Once this is done, load it up with:

  sudo launchctl load /Library/LaunchDaemons/cvspserver.plist

and then you should be good to go.

CVSweb

The key to getting CVSweb working is to make sure that the cvsweb.cgi file is correctly configured. Therein lies a tale.

The original CVSweb I had was for NT. There was a great CVS pserver for NT that I used for a very long time. It was wonderful. It allowed me to use a 180MHz Pentium II with 144MB of RAM and three disk drives totaling less than 30GB to be a CVS pserver, a Sybase Database server, an Apache server, and a few other little things. It was really impressive. So I had the CVSweb configured for NT.

Then I moved it to the iMac G3, and had to convert it to Mac OS X. No big deal, just a bunch of trial and error. Finding odd paths, improper commands, etc. and fixing them one by one. Not really hard, but not trivial, either. The problem was I didn't save it! So I had to do it all again. Thankfully, this time, /Library/WebServer/CGI-Executables/ is backed up with TimeMachine so I won't loose it again.

There's a configuration file that needed to be put in a logical place for the CGI script to pick it up. I mistakenly put it in /etc/apache2/other/ directory as the default name - cvsweb.conf. The problem is that the main httpd.conf has an include of all the *.conf files in other/ and as a consequence, it was trying to read the CVSweb configuration file as an Apache2 configuration file. That was an interesting development. In the end, I placed the cvsweb.conf file in /etc/apache2 directory with all the other high-level config files.

In the end, it was a simple CGI file and a few images to throw in the /Library/WebServer/Documents/Images/ directory. Not bad, just took a little time.

Git Using Gitosis

The next thing I wanted to get going was Git service for my Git repos. This needed to support the two methods of access: the git@git.themanfromspud.com method and the web service (git://) method. The first was basic configuration with a git account, which I had as a backup from the iMac G3. So I just had to make the account on the new machine, un-tar the backup from the external drive, and then it's ready to go. Well... almost.

The PATH for the git user needed to include the path to all the git executables. For the Google Groups install, that's /usr/local/git/bin/, and once I added that, things were a lot better.

Made sure the SSH service is turned 'on' from within System Preferences and I could:

  git clone git@git.themanfromspud.com:project.git

To get the git-daemon going, I needed to make another launchd config file, and this one I called /Library/LaunchDaemons/git.plist and it contained:

  <?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
  <dict>
      <key>Disabled</key>
      <false/>
      <key>Label</key>
      <string>com.apple.git</string>
      <key>UserName</key>
      <string>git</string>
      <key>GroupName</key>
      <string>_www</string>
      <key>Program</key>
      <string>/usr/local/git/bin/git</string>
      <key>ProgramArguments</key>
      <array>
          <string>/usr/local/git/bin/git</string>
          <string>daemon</string>
          <string>--base-path=/Users/git/repositories/</string>
          <string>--export-all</string>
          <string>--inetd</string>
      </array>
      <key>Sockets</key>
      <dict>
          <key>Listeners</key>
          <dict>
              <key>SockPassive</key>
              <true/>
              <key>SockServiceName</key>
              <string>git</string>
              <key>SockType</key>
              <string>SOCK_STREAM</string>
          </dict>
      </dict>
      <key>inetdCompatibility</key>
      <dict>
          <key>Wait</key>
          <false/>
      </dict>
  </dict>
  </plist>

and once it's loaded with:

  sudo launchctl load /Library/LaunchDaemons/git.plist

it's ready to go. You can then clone a repo with:

  git clone git://git.themanfromspud.com/project.git

GitWeb

GitWeb is a little different in that it's in the Git source package. In order to get it, I ended up pulling down the 1.6.5.2 source tarball from the source:

  curl -o git.tar.gz http://www.kernel.org/pub/software/scm/git/git-1.6.5.2.tar.gz
  cd git-1.6.5.2
  ./configure
  make

there's no reason to install it, but it's nice to get it made, just to be sure things are working OK on the server. Then, in the git-1.6.5.2 directory, there's a gitweb directory that has the information you need to install gitweb. There's a CGI file, a Perl file, a couple of PNG images, and a CSS file. Get it all configured and installed, and it's not too bad. Just took a little time.

SSL for Apache2

Getting SSL working on Apache2 was something I wanted, and badly, but it wasn't critical. I had everything I needed, and now I was onto the icing on the cake. SSL isn't critical, but it's something I've used in the past for building other sites, so I wanted to have it on this rebuilt server. Thankfully, this article about getting it going on Leopard (10.5) is still accurate enough to get me home.

I crated a 100 year certificate, sure, it's not signed by anyone other than me, but that's good enough for me. It's all working and that was a nice load off my mind.

PostgreSQL Support for Apple's PHP and Apache2

I decided a while back to stop using Marc's PHP builds - they're just too infrequent, and Apple ships a good PHP build, it just doesn't have PostgreSQL support in it. Thankfully, I had this solved already for my Intel iMac, and there was very little I needed to do to get this working. Having all these detailed instructions in the blog is a really nice thing.

PostgreSQL

When I moved off Marc's PostgreSQL builds, the one that I found that was the most successful for me was the KyngChaos wiki build. This has the 64-bit and 32-bit versions in the same binaries, and that's great for the libraries as well as the database engine itself. I decided that it'd be worth it to get PostgreSQL working on this guy - even if I didn't do a lot of heavy lifting with it, it'd be nice to have. Hey... if I need a bigger machine at a later date, I'll just get a Mac Pro and be done with it.

The launchd config file I used was placed in /Library/LaunchDaemons/org.postgresql.postgres.plist, and contains:

  <?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
  <dict>
      <key>Label</key>
          <string>PostgreSQL</string>
      <key>UserName</key>
          <string>postgres</string>
      <key>RunAtLoad</key>
          <true/>
      <key>EnvironmentVariables</key>
      <dict>
          <key>PGDATA</key>
              <string>/usr/local/pgsql/data</string>
      </dict>
      <key>ProgramArguments</key>
      <array>
          <string>/usr/local/pgsql/bin/postgres</string>
          <string>-e</string>
          <string>-i</string>
      </array>
      <key>StandardOutPath</key>
          <string>/usr/local/pgsql/var/logfile</string>
      <key>StandardErrorPath</key>
          <string>/usr/local/pgsql/var/logfile</string>
      <key>ServiceDescription</key>
          <string>PostgreSQL Server</string>
  </dict>
  </plist>

and again, once it's loaded with:

  sudo launchctl load /Library/LaunchDaemons/git.plist

it's ready to go. You can check the databases with:

  psql -l

WebDAV with SSL

The last thing I wanted to get going was WebDAV with SSL. I found several articles about WebDAV on Snow Leopard as it's pretty much built-in. All we need to do is to configure it. The one I used as my primary reference was this guy, and it's close - but not exact. The problem is that the config is using Basic authentication and I wanted to use Digest, as the Apple file suggests. In general, I copied a lot more of Apple's example config than I did the article, but it helped put a lot of it in perspective.

The /etc/apache/extra/httpd-dav.conf I got working was:

  DavLockDB "/Library/WebServer/WebDAV/DavLock"
 
  Alias /webdav /usr/local/davroot
  DavMinTimeout 600
  <Directory /usr/local/davroot>
      Dav On
 
      Order Allow,Deny
      Allow from all
 
      AuthType Digest
      AuthName frosty
 
      AuthUserFile "/Library/WebServer/WebDAV.passwd"
      AuthDigestProvider file
 
      <LimitExcept GET HEAD OPTIONS>
          require user drbob
      </limitExcept>
      <Limit GET HEAD OPTIONS>
          require valid-user
      </Limit>
  </Directory>

and then created the necessary password file as stated in the article. In the end, Cyberduck didn't work because it only understands Basic authentication. Transmit understands the Digest authentication, and so that's working just fine. I'm going to see if there's something I can do to get Cyberduck working, but even if I don't it's better to be hitting it with non-plaintext passwords than passing any passwords in the clear. Ever.

All Said and Done

When it all was said and done, including a multi-hour initial TimeMachine backup, I had a new server that had everything I needed and didn't loose a thing from the dead machine. Nice.

When I added a bunch of additional values to my web app, I noticed that the updating of the Google AnnotatedTimeLine was taking a lot longer than before. I mean it was pausing for a good 3 sec. before updating the view. I looked at the CPU usage, and it was all in the Flex component. It was bad. Very bad. So I knew I had to do something about it.

In the original version of the page, I had code that was run when the data was received and the graph redrawn. This was a standard hook for the AnnotatedTimeLine:

  /**
   * This function is called when the ATL is done updating the graph
   * from the data and the "draw()" method.
   */
  function graphReady() {
    // hide all the unchecked data sets
    for (var i = 0; i < portfolioChecks.length; ++i) {
      if (!portfolioChecks[i].checked) {
        updateBackgroundVisibility(portfolioNames[i], false);
      }
    }
    // ...finish up with more processing
  }
 
  /**
   * This is called to update the visibility of the named portfolio
   * to the provided state in the background graph.
   */
  function updateBackgroundVisibility(name, state) {
    var colCnt = graphData.getNumberOfColumns();
    // I have to find the column name in the table
    for (var i = 1; i < colCnt; ++i) {
      if (graphData.getColumnLabel(i) == name) {
        // the dataset number is one less than the column number
        if (state) {
          chart[bg].showDataColumns(i-1);
        } else {
          chart[bg].hideDataColumns(i-1);
        }
      }
    }
  }

When I originally built the code, I wanted to have something that allowed me to change the visibility of the data sets in the graph(s) either way, and that's really nice. But what I didn't expect was the performance penalty I would pay for such a design.

As it turns out, there's a form of the hideDataColumns() method on the ATL that allows the user to send an array of dataset indexes. I didn't know what to expect, but I thought it had to be better than this, so I recoded this as:

  /**
   * This function is called when the ATL is done updating the graph
   * from the data and the "draw()" method.
   */
  function graphReady() {
    // hide all the unchecked data sets
    var cols = [];
    for (var i = 0; i < portfolioChecks.length; ++i) {
      if (!portfolioChecks[i].checked) {
        cols.push(i-i);
      }
    }
    chart[bg].hideDataColumns(cols);
    // ...finish up with more processing
  }

Without the call to updateBackgroundVisibility(), I knew it'd be a little faster - no need to look up all the column headers, etc. But I didn't expect to see what I saw.

The resulting code took the refresh time from 2-3 sec. to under the blink of an eye. Really. It was a slight flicker, but that's about it. Amazing. In retrospect, it makes sense... I was individually hiding about 30 columns each update. That's a lot. To do it right, there had to be some type of lock, update, refresh/redraw, and then unlock. All that added up. Big time.

It's taught me a big lesson about the efficiency of Google's code: if in doubt, look for a better way, there's probably one there, or you can always ask the visualization team. I know I've learned my lesson.

Today I found another bug in the Google Visualizations AnnotatedTimeLine widget. Basically, if you set the graph setting legendPosition to sameRow, the legend at the top of the widget will start on the same row as the date/time of the point you're currently highlighting. If you have it set to newRow, the legend will start on the line below the date/time. The sameRow looks like this:

and the newRow looks like this:

What you can see is that on the sameRow, the legend starts out right, but it never wraps to the next line. On the newRow version, it wraps nicely, but you loose a complete row, and in the case of large legends, that row is important.

So I posted a question to the Visualization group and got this answer:

Hi,

Please open a feature request from the link at the left side menu, and we will try to get to this.

Regards,
VizGuy

So that's exactly what I did. I'm hoping that they get to this as soon as possible.

I'm not a big Flash fan, in fact, I use ClickToFlash to keep from seeing it displayed on most of my web usage, but there is one notable exception: the Google Visualization widgets. I use these extensively in my web work to assist in the visualization of the data. In order to run these guys, I need Flash. So it makes sense to keep up with Flash for this reason alone.

This morning, Adobe updated the Flash player for Mac OS X to 10.0.42.34, and I needed to pick it up. Not thrilled, but until Google moves from Flash to something else, this is what I have to do.

This morning I noticed that BBEdit 9.3.1 was out with an impressive list of fixes and features for a minor release update. Had to get that, I use it every single day.