What's it all about, Alfie?

I've written a system of alerts for a web app that I've been working on, and one of the problems - as it always is, is that of bad data. Specifically, false positives and false negatives. If we have an alert system, then we need to be sure that when there's a reason to alert, we alert the right people. A data entry error is not necessarily a reason for alert. It might clear up in a minute, and if so, we shouldn't alert folks telling them that the sky is falling.

So there's some element of signal processing in this work. How to filter out the bad data points and keep the good. I've been working with some interesting algorithms as well as some pretty straight-forward ones. One of the simpler ones is a simple n-point moving average. I have this on the input of the data to the alert system to suppress the spikes but there were some very nasty problems associated with that algorithm.

Probably the biggest was that it really doesn't filter the bad data 'out', as much as diffuses it. Take a large negative spike with a small trailing bit. An 11-point moving average would look like this:

The problem is that the bad data starts at 7:25:34, but it effects the moving average for a much larger area simply due to it's magnitude with respect to the "signal" it's in. There's not a lot a moving average can do to smooth this out. Sure, other filters can, but I also need something that's going to be fast, so I can't be computing an FFT to get a frequency-domain based filter for this data.

As I was thinking about this problem and started thinking about the other simple statistical functions that were in Numbers (the program I was using to look at and play with the data. In a flash, the median hit me, and I started to run with it.

What if I did a 5-point median, and then did an 11-point average on that? Oh... that looked good. What about a straight 11-point median? See for yourself:

This is a ton better. While it can still effect the output, bad data has to exist for at least half the window in order to effect it at all. This is a pretty fair situation as if it's that bad, then someone needs to know about it.

Next week, I'll look at putting this into the code.

This entry was posted on Friday, October 2nd, 2009 at 4:28 pm and is filed under Coding, Cube Life. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.