What's it all about, Alfie?

I've been working on a web app that's a very heavy JavaScript page, and as such, I've had to force users to run it in Google Chrome as that's the only browser that has a decent JavaScript Engine (on Windows) that can handle the load. It doesn't hurt that I'm using a lot of the Google AJAX libraries, and so they are sort-of made for each other.

This morning I took a whack at the new Google Chrome Frame. This slick IE plug-in is the complete Google Chrome environment in a plug-in frame for IE. This means that pages that run in Chrome will run in the Chrome Frame without modification.

That's big news. I was stunned.

So I decided to try it out. First, you need to put the meta tag in your pages to tell IE to use the Chrome Frame plugin:

  < meta http-equiv="X-UA-Compatible" content="chrome=1" />

That, in itself, will make things work, but if you wanted to verify that the plug-in was installed, then you can add the following to your page to do the check and redirect the user to the Google Chrome Frame download page if it's not installed. First, you need to load the script in the head of the page:

  < script type="text/javascript"
  src="http://ajax.googleapis.com/ajax/libs/chrome-frame/1/CFInstall.min.js" ></script>

Then in the body of the page, you need to define a div that will be the target of the download prompt frame:

  < div id="chrome_check" />

just make the id unique and you'll be fine.

Next, have the JavaScript code in the initialization of your page:

  CFInstall.check({ node: "chrome_check" });

The CFInstall.check() function checks to see if Chrome Frame is installed, and if not, will display at the div indicated by the node property a download page hosted by Google. The user can then download Chrome Frame and that's it. It's ready to go.

The docs on Chrome Frame say that the page should automatically reload, but I haven't found that to be the case. It could be just me, I've only tried it once, but even that isn't bad. It's something that allows users to send links in emails and have their default browser (IE) render the project's pages in a Chrome environment.

Super sweet.

I'm looking at the plans for my next project, and the guy I'll be working with side-by-side is pretty much set on C++ for his project, and since it feeds into mine, I really would need a spectacular reason not to pick the same thing. So I started looking at the current version of the project I'll be updating and seeing what it had that I might find difficult to get my hands on in C++. The key components I could come up with were:

• Good Database Tools - with SQLAPI++, there's no problem there. We can connect to MS SQL Server which is the "standard" at the shop.
• In-Memory Database - it might be very nice to have something like H2 in C++ so that should I choose to go down that path, it would be as easy to use as an external database, but much faster.
• Web Server - the current project does a lot of serving of requests via 29 West and http requests (it's a Tomcat web app). If I"m going to make the transition as simple and easy as possible, then I need to be able to handle http requests.

When I started looking at the in-memory database, I realized that SQLite can be used to create in-memory databases, and since it's also supported by SQLAPI++, it's a perfect solution for what I might need. You only need to open a connection the the special file named :memory: and the connection returned will be to a blank, in-memory database. Close the connection, however, and the database goes away.

Additionally, if you open another connection to the same special file, you'll get a completely different in-memory database. This is really interesting in that it allows the user to create as many unique in-memory databases as they wish. Unfortunately, it's not as nice as H2 in that the connection can be set to stay open, but it's a reasonable compromise for an in-memory database.

Even if I didn't use the in-memory database, the SQLite filesystem database is strong enough to not corrupt the file through power outage, and it's very fast for a single system to hold data - so should I want to use it to hold configuration details or parameters, it's certainly more than up to the task.

So I think I have more than enough on the database front for this project in C++. But the real question was the web server interface. I needed to have an embeddable web server that was going to be reasonably fast, small, easy to interface to, and handle all the things I needed. I had to hit Google for this one.

After a lot of poking around, I found several that might fit the bill. The most interesting is Lighttpd - pronounced "Lighty" by the author, and it's goal was to be able to handle 10,000 connections on a single server. Admirable goal, and it's being used by some heavy hitters out there. The big problem seems to be that it's a complete web server with CGI capability. That means that I'd need to create something to use as CGI and then put that into the web server. Not a lot different from using Apache, except the loading capable with the server. So I kept looking.

I found a GNU project: libmicrohttpd which is a C library that you can embed and gives you tons of the goodies you'd need - and far more than I'd need in this application. It's in C, which means it's going to need to be wrapped at least a bit, but it's a strong possibility in my mind because it's a GNU project, and they tend to be pretty bug free.

Probably the most promising candidate seems to be mongoose hosted on Google Code. It appears to be a nice C/C++ library that runs on a ton of platforms and handles all the things I need a web server to do. Additionally, it boasts a simple, clean API to plug into for request processing.

After looking at mongoose, I think I have enough to get started - technically, on this project. It's not necessarily ideal, but I have to say that if I ported my Google DataTable code from Java to C++ and put it in CKit, I think there's not a lot I couldn't do in the existing project. The real problem is that there's a ton I want to do in the new project that's a lot more complex than what's being done in the current project that I'm still going to need to think about it.

I just don't want to get to the point that I have to re-write huge chunks of functionality from one language to C++ in order to get things working. It's a waste of time and a maintenance nightmare.

This morning I decided it's been almost a year since I've gotten the latest FORTRAN compiler from the HPC for Mac OS X web site. They typically update ever couple of months, and with the release of Snow Leopard, I was in for a wonderful treat - they had a Snow Leopard version that generated 64-bit executables. This is fantastic news!

I downloaded the package, updated my uninstall script based on the tarball's contents, and then gave it a whirl.

  peabody{drbob}275: gfortran --version
  GNU Fortran (GCC) 4.5.0 20090910 (experimental)
  Copyright (C) 2009 Free Software Foundation, Inc.
 
  GNU Fortran comes with NO WARRANTY, to the extent permitted by law.
  You may redistribute copies of GNU Fortran
  under the terms of the GNU General Public License.
  For more information about these matters, see the file named COPYING
 
  peabody{drbob}276: 

Sweet.

I love that they build this package with auto-vectorization and OpenMP built in. This is just one of the most wonderful treats of life: to be able to work with such a wonderfully simple programming language on such wonderfully complex problems. Face it - there's no GUI for FORTRAN, it's standard out. Period. That means that you're not going to be making a cool web browser in it. It's not meant to do that. It's meant to push bits around and solve problems like never before.

There's no doubt that FORTRAN was one of the things that drew me into computers back in the 70's. It's just that incredible. I feel incredible joy in being able to use it. Thanks to the HPC on Mac OS X guys for making my day.

I've heard a lot about ClickToFlash, and I have stayed away from it so far because I didn't think it was that bad. I mean sure, a few stolen cycles here and there, but I didn't think it was a really serious problem. But still... I've seen Liza's computer. She's always going to very heavy Flash sites and she could really probably use this.

So I decided to get it and give it a spin. It comes highly recommended, so it's not junk. We'll see if I don't like this better. Certainly, it's something to try with Liza.

Well... today I was able to get the moving median-based filter into my web app without too much trouble. There was a bit of grief in that I hadn't planned on having multiple data filters on the alert inputs, but that's the way you need to code - make it easy and simple, and then if you have to re-factor code, then that's when you do it. In the beginning, there was no way to know - nor reason to believe, that I was going to need multiple data scrubbers. It took me a grand total of less than an hour to add the other filter, but then about another hour to debug it due to BigDecimal versus Double issues when dealing with the output of the H2 database.

I also added a few new features to the app in order to make it a little more nimble and lightweight for those users that are reporting that the app is a little too slow and cumbersome. I can understand it - they have loaded machines and this is just one of the many things they run. So I added a feature to just show the last hour as opposed to the entire day. This will really help if all you're interested in is the recent past. Might really help.

Other than that, it's been a pretty quiet day. No major updates to anything, just nice, solid, steady progress on the app. Not a bad day at all.

I've written a system of alerts for a web app that I've been working on, and one of the problems - as it always is, is that of bad data. Specifically, false positives and false negatives. If we have an alert system, then we need to be sure that when there's a reason to alert, we alert the right people. A data entry error is not necessarily a reason for alert. It might clear up in a minute, and if so, we shouldn't alert folks telling them that the sky is falling.

So there's some element of signal processing in this work. How to filter out the bad data points and keep the good. I've been working with some interesting algorithms as well as some pretty straight-forward ones. One of the simpler ones is a simple n-point moving average. I have this on the input of the data to the alert system to suppress the spikes but there were some very nasty problems associated with that algorithm.

Probably the biggest was that it really doesn't filter the bad data 'out', as much as diffuses it. Take a large negative spike with a small trailing bit. An 11-point moving average would look like this:

The problem is that the bad data starts at 7:25:34, but it effects the moving average for a much larger area simply due to it's magnitude with respect to the "signal" it's in. There's not a lot a moving average can do to smooth this out. Sure, other filters can, but I also need something that's going to be fast, so I can't be computing an FFT to get a frequency-domain based filter for this data.

As I was thinking about this problem and started thinking about the other simple statistical functions that were in Numbers (the program I was using to look at and play with the data. In a flash, the median hit me, and I started to run with it.

What if I did a 5-point median, and then did an 11-point average on that? Oh... that looked good. What about a straight 11-point median? See for yourself:

This is a ton better. While it can still effect the output, bad data has to exist for at least half the window in order to effect it at all. This is a pretty fair situation as if it's that bad, then someone needs to know about it.

Next week, I'll look at putting this into the code.

I didn't know that VelaClock wasn't Snow Leopard compatible but today I noticed that VelaClock 2.2 was out and they mentioned that it was now Snow Leopard compatible. Of course, I had to get it.

I was working at home today and needed to get into a Windows box from home. I fired up the Cisco VPN from within Shimo (incredibly app), and then fired up CoRD. I noticed there was an update to 0.5.1, so I got it.

I have to say this is what I like about folks that write code for Macs. It's stellar. Superlative UI, clean, simple, well thought out. It was a joy to use. I almost didn't mind that I was connecting to a Windows box when it looked this nice.

I got what I needed done quickly, and things shut down as gently as could be. What a wonderful experience. Loved it.

Today I wanted to upgrade my PostgreSQL install to 8.4.1 (from 8.3.x) based on the KyngChaos package I'd just started using. It's as much a test of the upgrade process as it is about any 8.4.x feature in PostgreSQL. It's really not all that hard, just a few steps in the right order.

First, open up a Terminal.app window and become the user postgres:

  su - postgres

then, dump the database. Assuming you're in the postgres user's home directory - /Users/postgres, you can do something like:

  /usr/local/pgsql/bin/pg_dumpall > postgresql_20090930.dump

and the dump will be tagged with the date. Easy.

Next, turn off the existing PostgreSQL engine, and set set the old install aside:

  sudo launchctl unload /Library/LaunchDaemons/org.postgresql.postgres.plist
  cd /usr/local
  sudo mv pgsql pgsql_old

Next, install the new package. If things are going well, the new engine will be started by the installer. As long as the launchd file exists, you should be OK.

Next, load the databases. Back in the postgres user's Terminal window:

  /usr/local/pgsql/bin/psql -d template1 -f postgresql_20090930.dump

Check that the databases are back and functioning. Then it's just a matter of a little cleanup:

  sudo rm -rf /usr/local/pgsql_old

For the last several days I haven't written any posts because I've been deep into this codebase that I inherited and don't really like much. OK, that's being polite. But a few interesting things happened in these last few days that make me step back and try to see the forest for the trees. Maybe it's just because I think I'm done, and I'm in one of those euphoric states of mind that occurs when incredible pain is removed, and it appears that you're in incredible happiness... and then again, maybe it is honest happiness.

The Code

I'm not going to apologize for my take on the code. It is really horrible. Yeah, it's really that bad. It's not the worst code I've ever seen, but had it been even a little worse than it is, I would have said that it never would have worked for the purposes at hand, and a re-write would have been required. It was passible, but only barely.

So what's the upshot of this? Well... I'm stuck with it. I've been in this spot several times before with other projects. The code is so convoluted that only a very few (typically only one) person can go in and make changes. The process is hard, painful to do, and error-prone. It's a disaster waiting to happen. Consequently, it's something that has to be done very carefully.

But it can be done.

I'm trying to take away that last point: It can be done. It's not impossible, just very, very difficult. In that sense, I'm not really upset about the code, or having to work with it. It's something else entirely that I'm upset with.

The Timescale

I'm all for deadlines. I don't mind them as long as everyone knows them. If it's an unreasonable deadline, then when it's made, the statement of it's unreasonable-ness will be made, and expectations can be set. It's never an all-or-nothing proposition, but as long as everyone knows where they stand, it's fair. Something that's been happening in the last few weeks is that I'm getting the very real sense that there's a deadline, or timescale that I'm not being made aware of. There is a very real pressure to complete things as fast as possible - without regards to duplicating work and carefully removing bugs in the code.

Such might be the case, if there were a deadline that I weren't aware of. Something like "Mean Guy over there is saying our group doesn't work hard, and I told him we'd be done with this by the end of the month." I'm all for sprinting for a goal, but I need to know the goal, and that we're sprinting. Otherwise, I get the feeling that I'm being horse-whipped with things that make no sense.

• Sending the app to testing with users when we know some of the numbers are bad -- why not just wait until the numbers are right and then show it to them?
• Changing the Test configuration in the middle of testing for an hour so someone can see the numbers -- again, you're making this a wasted hour because I'll have to revert back to the consistent testing configuration to really be sure I have everything.

I don't say that there's not a reason for this - only that I'm totally unaware of what that reason is, and being in the dark, working as hard as I do, is not a comfortable feeling. I get to feeling very manipulated.

I'm not such a prima donna that I don't think all decisions have to come to (or through) me, I just want to be told what the decisions are when they effect me. It's about feeling that you know where you stand.

The Support System

This week I've had a real shock to my traditional support system. I have a few friends that I've worked with for years, who now work different places, and I chat to them pretty much every single day. It's not like we're texting teenagers, it's more like an informal way to communicate if we happened to be further apart than a shout.

I had been having problems with this project and was venting to these guys, and basically got slapped back down by both of them. One, said he had problems as bad with his, and didn't think it warranted the grumbling noises that I was taken to task for by management. The other is now a manager, and he said he wouldn't stand for that.

Interesting.

While I understand their points of view, and probably even agree with them (now), at the time, they were "keepin' it real" by smacking me down. Needed? Maybe. Helpful? Not in the least.

The problem was, as I realize now in retrospect, that I was mad at more than those asking me to do this, I was mad at myself for not putting my foot down and requiring some sense of timeframe, or setting a schedule, or at least not letting them push me to answer deliverables every hour (no joke). Asserting myself at work is something that I have a hard time with, especially when I'm new to a position. I don't feel comfortable saying "No" to even unreasonable requests. The word "possible" kills me.

So while I needed support, and didn't get it, I needed perspective more.

Lessons Learned

So after all this, what have I learned? I think there are a few things that I should work hard on:

• Expressing my beliefs about the cost of certain decisions - heck, maybe if I'd told them exactly what the cost was going to be, they'd have said "Nah, it was just an idea".
• Sticking to my beliefs and not being a doormat - there's nothing professional about being walked over by peers or management. If you are a professional, and in your professional opinion the plan has mistakes or omissions, then you should feel that it's possible to express them professionally.
• Know when to shut up - obvious, but necessary.

I'm sure there's more, but for now, this will get me a long way towards not getting into a situation like this again.