What's it all about, Alfie?

I've heard a lot about ClickToFlash, and I have stayed away from it so far because I didn't think it was that bad. I mean sure, a few stolen cycles here and there, but I didn't think it was a really serious problem. But still... I've seen Liza's computer. She's always going to very heavy Flash sites and she could really probably use this.

So I decided to get it and give it a spin. It comes highly recommended, so it's not junk. We'll see if I don't like this better. Certainly, it's something to try with Liza.

Well... today I was able to get the moving median-based filter into my web app without too much trouble. There was a bit of grief in that I hadn't planned on having multiple data filters on the alert inputs, but that's the way you need to code - make it easy and simple, and then if you have to re-factor code, then that's when you do it. In the beginning, there was no way to know - nor reason to believe, that I was going to need multiple data scrubbers. It took me a grand total of less than an hour to add the other filter, but then about another hour to debug it due to BigDecimal versus Double issues when dealing with the output of the H2 database.

I also added a few new features to the app in order to make it a little more nimble and lightweight for those users that are reporting that the app is a little too slow and cumbersome. I can understand it - they have loaded machines and this is just one of the many things they run. So I added a feature to just show the last hour as opposed to the entire day. This will really help if all you're interested in is the recent past. Might really help.

Other than that, it's been a pretty quiet day. No major updates to anything, just nice, solid, steady progress on the app. Not a bad day at all.

I've written a system of alerts for a web app that I've been working on, and one of the problems - as it always is, is that of bad data. Specifically, false positives and false negatives. If we have an alert system, then we need to be sure that when there's a reason to alert, we alert the right people. A data entry error is not necessarily a reason for alert. It might clear up in a minute, and if so, we shouldn't alert folks telling them that the sky is falling.

So there's some element of signal processing in this work. How to filter out the bad data points and keep the good. I've been working with some interesting algorithms as well as some pretty straight-forward ones. One of the simpler ones is a simple n-point moving average. I have this on the input of the data to the alert system to suppress the spikes but there were some very nasty problems associated with that algorithm.

Probably the biggest was that it really doesn't filter the bad data 'out', as much as diffuses it. Take a large negative spike with a small trailing bit. An 11-point moving average would look like this:

The problem is that the bad data starts at 7:25:34, but it effects the moving average for a much larger area simply due to it's magnitude with respect to the "signal" it's in. There's not a lot a moving average can do to smooth this out. Sure, other filters can, but I also need something that's going to be fast, so I can't be computing an FFT to get a frequency-domain based filter for this data.

As I was thinking about this problem and started thinking about the other simple statistical functions that were in Numbers (the program I was using to look at and play with the data. In a flash, the median hit me, and I started to run with it.

What if I did a 5-point median, and then did an 11-point average on that? Oh... that looked good. What about a straight 11-point median? See for yourself:

This is a ton better. While it can still effect the output, bad data has to exist for at least half the window in order to effect it at all. This is a pretty fair situation as if it's that bad, then someone needs to know about it.

Next week, I'll look at putting this into the code.

I didn't know that VelaClock wasn't Snow Leopard compatible but today I noticed that VelaClock 2.2 was out and they mentioned that it was now Snow Leopard compatible. Of course, I had to get it.

I was working at home today and needed to get into a Windows box from home. I fired up the Cisco VPN from within Shimo (incredibly app), and then fired up CoRD. I noticed there was an update to 0.5.1, so I got it.

I have to say this is what I like about folks that write code for Macs. It's stellar. Superlative UI, clean, simple, well thought out. It was a joy to use. I almost didn't mind that I was connecting to a Windows box when it looked this nice.

I got what I needed done quickly, and things shut down as gently as could be. What a wonderful experience. Loved it.