What's it all about, Alfie?

I've written a system of alerts for a web app that I've been working on, and one of the problems - as it always is, is that of bad data. Specifically, false positives and false negatives. If we have an alert system, then we need to be sure that when there's a reason to alert, we alert the right people. A data entry error is not necessarily a reason for alert. It might clear up in a minute, and if so, we shouldn't alert folks telling them that the sky is falling.

So there's some element of signal processing in this work. How to filter out the bad data points and keep the good. I've been working with some interesting algorithms as well as some pretty straight-forward ones. One of the simpler ones is a simple n-point moving average. I have this on the input of the data to the alert system to suppress the spikes but there were some very nasty problems associated with that algorithm.

Probably the biggest was that it really doesn't filter the bad data 'out', as much as diffuses it. Take a large negative spike with a small trailing bit. An 11-point moving average would look like this:

The problem is that the bad data starts at 7:25:34, but it effects the moving average for a much larger area simply due to it's magnitude with respect to the "signal" it's in. There's not a lot a moving average can do to smooth this out. Sure, other filters can, but I also need something that's going to be fast, so I can't be computing an FFT to get a frequency-domain based filter for this data.

As I was thinking about this problem and started thinking about the other simple statistical functions that were in Numbers (the program I was using to look at and play with the data. In a flash, the median hit me, and I started to run with it.

What if I did a 5-point median, and then did an 11-point average on that? Oh... that looked good. What about a straight 11-point median? See for yourself:

This is a ton better. While it can still effect the output, bad data has to exist for at least half the window in order to effect it at all. This is a pretty fair situation as if it's that bad, then someone needs to know about it.

Next week, I'll look at putting this into the code.

I was working at home today and needed to get into a Windows box from home. I fired up the Cisco VPN from within Shimo (incredibly app), and then fired up CoRD. I noticed there was an update to 0.5.1, so I got it.

I have to say this is what I like about folks that write code for Macs. It's stellar. Superlative UI, clean, simple, well thought out. It was a joy to use. I almost didn't mind that I was connecting to a Windows box when it looked this nice.

I got what I needed done quickly, and things shut down as gently as could be. What a wonderful experience. Loved it.

For the last several days I haven't written any posts because I've been deep into this codebase that I inherited and don't really like much. OK, that's being polite. But a few interesting things happened in these last few days that make me step back and try to see the forest for the trees. Maybe it's just because I think I'm done, and I'm in one of those euphoric states of mind that occurs when incredible pain is removed, and it appears that you're in incredible happiness... and then again, maybe it is honest happiness.

The Code

I'm not going to apologize for my take on the code. It is really horrible. Yeah, it's really that bad. It's not the worst code I've ever seen, but had it been even a little worse than it is, I would have said that it never would have worked for the purposes at hand, and a re-write would have been required. It was passible, but only barely.

So what's the upshot of this? Well... I'm stuck with it. I've been in this spot several times before with other projects. The code is so convoluted that only a very few (typically only one) person can go in and make changes. The process is hard, painful to do, and error-prone. It's a disaster waiting to happen. Consequently, it's something that has to be done very carefully.

But it can be done.

I'm trying to take away that last point: It can be done. It's not impossible, just very, very difficult. In that sense, I'm not really upset about the code, or having to work with it. It's something else entirely that I'm upset with.

The Timescale

I'm all for deadlines. I don't mind them as long as everyone knows them. If it's an unreasonable deadline, then when it's made, the statement of it's unreasonable-ness will be made, and expectations can be set. It's never an all-or-nothing proposition, but as long as everyone knows where they stand, it's fair. Something that's been happening in the last few weeks is that I'm getting the very real sense that there's a deadline, or timescale that I'm not being made aware of. There is a very real pressure to complete things as fast as possible - without regards to duplicating work and carefully removing bugs in the code.

Such might be the case, if there were a deadline that I weren't aware of. Something like "Mean Guy over there is saying our group doesn't work hard, and I told him we'd be done with this by the end of the month." I'm all for sprinting for a goal, but I need to know the goal, and that we're sprinting. Otherwise, I get the feeling that I'm being horse-whipped with things that make no sense.

• Sending the app to testing with users when we know some of the numbers are bad -- why not just wait until the numbers are right and then show it to them?
• Changing the Test configuration in the middle of testing for an hour so someone can see the numbers -- again, you're making this a wasted hour because I'll have to revert back to the consistent testing configuration to really be sure I have everything.

I don't say that there's not a reason for this - only that I'm totally unaware of what that reason is, and being in the dark, working as hard as I do, is not a comfortable feeling. I get to feeling very manipulated.

I'm not such a prima donna that I don't think all decisions have to come to (or through) me, I just want to be told what the decisions are when they effect me. It's about feeling that you know where you stand.

The Support System

This week I've had a real shock to my traditional support system. I have a few friends that I've worked with for years, who now work different places, and I chat to them pretty much every single day. It's not like we're texting teenagers, it's more like an informal way to communicate if we happened to be further apart than a shout.

I had been having problems with this project and was venting to these guys, and basically got slapped back down by both of them. One, said he had problems as bad with his, and didn't think it warranted the grumbling noises that I was taken to task for by management. The other is now a manager, and he said he wouldn't stand for that.

Interesting.

While I understand their points of view, and probably even agree with them (now), at the time, they were "keepin' it real" by smacking me down. Needed? Maybe. Helpful? Not in the least.

The problem was, as I realize now in retrospect, that I was mad at more than those asking me to do this, I was mad at myself for not putting my foot down and requiring some sense of timeframe, or setting a schedule, or at least not letting them push me to answer deliverables every hour (no joke). Asserting myself at work is something that I have a hard time with, especially when I'm new to a position. I don't feel comfortable saying "No" to even unreasonable requests. The word "possible" kills me.

So while I needed support, and didn't get it, I needed perspective more.

Lessons Learned

So after all this, what have I learned? I think there are a few things that I should work hard on:

• Expressing my beliefs about the cost of certain decisions - heck, maybe if I'd told them exactly what the cost was going to be, they'd have said "Nah, it was just an idea".
• Sticking to my beliefs and not being a doormat - there's nothing professional about being walked over by peers or management. If you are a professional, and in your professional opinion the plan has mistakes or omissions, then you should feel that it's possible to express them professionally.
• Know when to shut up - obvious, but necessary.

I'm sure there's more, but for now, this will get me a long way towards not getting into a situation like this again.

Today I realized something that's been bothering me for weeks. It's all about management's perceived need to rush something through the process, and what actually the business need is. Specifically, when you're not building a fix, or a critical feature that's time-dependent, then what's the bloody rush? Why should I find 30 mins right now to show something to a group of users when I'm in the middle of fixing something in the code and tomorrow will do?

This is all about management's need to set it's own, self-imposed goals, and has nothing to do with reality or the development process. In fact, it's counter to that process as a whole. I've been told to show something that I know is wrong in spots because management doesn't want to wait until it's fixed in an hour or so, worrying that we'll loose an entire day. So what?

This isn't business critical. This isn't time-critical. This isn't something they have been clamoring for. This is something that is an enhancement to the existing functionality and because the QA process is very lengthy, management doesn't want to wait an hour it doesn't have to.

Can you say Crack Monkey? Yes, I knew you could.

This makes me exceptionally angry about some management decisions, such as these. If you think I'm the one that can get the job done, and if you trust me to do it right, and quickly, why on earth are you so "ants in the pants" about a release? This is in-house development, folks. This isn't releasing the next version of the OS, or meeting a hard deadline. This is a simple matter of a manager losing touch with what's the best way to manage people.

I've got to remember that I'm angry at the decision, not the person. It's been happening for several weeks, and it's only just now that I've realized it. I need to say "No" more. Just plain "No".

I'm as angry at myself as I am at him.

Today was one of those days that, I believe, defines what kind of person you are. Are you a person that can knuckle down and do what's needed even though every fiber of your being is screaming that it's the wrong thing to do? Where is the individual and the team defined in your mind? How much can the team request as sacrifice from the individual? It's interesting questions because we all think we know where we sit on the questions, but it's really when push comes to shove that we learn where we really sit.

Today I thought I was done with a very distasteful bit of coding. To call it a 'bit' is to call Lake Michigan a 'pond'. It was a week of 11-hour days of the most intense, horriffic coding that I've done in a very long time. But, like ripping a band-aid off, once it was done, I could move on to nicer things.

Yet, as soon as I was done I got several requests from folks "...yeah, I wasn't right on that, it needs to be like this..." and "Great! Now add this". In the end, I received in a 4 hour period enough work to keep me busy for the rest of this week, and possibly into the next.

To add to this insult, some of the requests were for changes in the Flex client part of the project. While I've done some Flex coding, it's nothing like what I needed to do the request, and I said so. The original author that handed me this project said "Sure, it's doable". I wanted to scream at him then and there: Shut up unless you are going to do the change! But I didn't. I'm already too "gruff" for the office.

So I'm back at the bottom. It was as if all the work didn't matter - worse it was because of all the work that I got more. I wanted to sit and cry - or maybe just plain leave. I had been telling management that this system was a pain to work in... like it taking more than 5,000 lines of java code to add less than a dozen fields to the system... it's a monster and it's not getting any easier to work with given all the additions.

So to hear that I had effectively brought down more work in this horrible project upon myself was simply more than I could bear. I was more than a little angry at the original developer who, like Pilot, had washed his hands of his horrible incarnation. If only...

Anyway, I had a bunch more work to do, so laying it out in my notes, I dug in and started working. What I found was that the requests had generated a ton of new work. Amazing. They think these things are "simple", or "easy", and in a good system, they would be, but this isn't a good system. It's not even "reasonable." It's crummy - and that's being kind.

So, now I'm down... then I'm kicked... and it's not getting any better anytime soon. I still need to do all this work, and then I'm sure there's going to be add-ons after that. In a very real sense, I've learned that my skills are like anything of value: a double-edged sword, and I'm feeling the cuts very deeply today.

Because I can do this, I'm told to do this.

Regardless of my warnings, advice, professional judgement... none of that matters because a few days of my time is considered insignificant and the results fantastic. Who cares if I want to scream every minute of every hour while I'm doing it?

Gotta love the "bottom line".

Well... everything costs. Pretty darn true. Not a lot in life is free, and that which is most of the time only seems free at the time. So it's not at all surprising what happened today.

Ripping the Band-Aid

I'm a Band-Aid Ripper. When I have a task that I know I'm not going to like, I try to steel myself to the pain, and then just rip. Get it done. Get over it. Move on. There's no reason to draw out a painful process to make it long and painful. It's enough that it's painful. Rip it off.

In all honesty, I didn't use to be this way. I was scared - plain and simple. I hoped, like all kids, I guess, that if you do it slowly, there won't be any pain. All the pain, you hope, is in the ripping. But that's not the case, is it? Nope. In most cases, that band-aid is stuck, and it's going to be painful to get it off. So it's just up to you to take it off on your terms.

So I've been working for the last several days on adding a few calculated values to this application that I have inherited at my new job. It's not what I'd have written. In fact, I look at it and I think there's just about nothing I think that was done right. It's a mess. Plain and simple. Even the original author has called it "crap", "a hack", and not once or twice.

But I'm able to make it work.

So there's no need to rewrite it. It works. So when I had to work on it, I knew I had to attack it... get into it, get it done. The longer I stayed working on it, the angrier I was going to be. But why?

To give you an idea, to add the 10 or so calculated values I had to add over 5,000 lines of code - totaling nearly 500,000 characters. Just to add a few fields. The problem is in the way it's structured - lots of interfaces and objects - but all functional in nature and none really encapsulating the reporting behavior. This means that when I have a report that looks at a certain variable trading during the day, there's going to be an opening value, a trading value and a net value. Rather than create a object for this, there are three values passed around the code.

Add to that, we might need to difference these, and we get nine values: open, trading, net, and then initial, final, difference. If I add 10 variables, then I'm really adding 90 arguments to many methods.

It's not good. It just wasn't designed properly for what it was doing. But I got it done.

How it Looks to Others

After I got this all done, I had a meeting with my manager. Basically, he believed that there were two ways to go about doing this:

• re-work the code to add in the values... grumbling and grousing while you do it... giving off the impression that you're capable, but not a guy anyone wants to work with - possibly someone that's just a pain.
• re-work the code to add in the values - but this time be nice about it. Everyone wins. You're happier, no one gets the wrong impression. Win-Win.

I wanted to say that Fantasy Island is a nice place to live, and I'd love to visit him there, but I didn't. What I said was that I'd try harder to keep my mouth shut, and not give off the wrong impression.

But is it really the wrong impression, or just an unpopular impression? I have no doubt that I'd love to be happy when working with all the code I have to work with. I don't enjoy getting upset at code, or developers, but when I have to extend code and the number of arguments to a method exceeds 255, then there's something terrribly wrong with the code and it needs to be fixed.

But if it's not outwardly broken, why fix it? There's no business motivation for it. It's working, after all, it's just a complete mess.

So I'm left with a manager thinking I'm making the active decision to be unhappy when I'm just responding to someone doing a horrible job and being held up as a paragon of development virtue.

Where Do I Really Fit?

I'm left thinking about the experience and wondering if Liza's not spot on when she says this happens with every place I work. That my expectations of others are far too high for mortals, and while it's less than I demand of myself, the vast majority of people simply fall far short of my standards, and these problems are bound to arise. I don't think I'm that demanding. But I can certainly see her points.

I'm in an industry where developers are paid a ton of money. I mean a ton. If you're making in the six-figures annually, then by golly, you had better be giving it your all every day - or at least more days than not. That's a ton of money, and you need to be worth it. There is no free ride.

So I do get a little miffed at these guys making this money and not feeling compelled to fix up their messes. Far too many developers write something and hand it off never to think about it again. Where's the pride in what you have created? Where the dedication to your craft that you'd hand off such crap and call it "done"? Clearly I come from a different time. Which leads me to the ultimate question: Where do I belong?

I'm coming back to the thought that I need to be on my own. Writing software that meets my standards. It's not easy, and consulting is the easiest way to make the transition, but that leads to a lot of consulting and not a lot of your own code getting written. No, I'm thinking it's time to start planning on getting out on my own. I've toyed with the idea for a long time, but never really had a motivation to make any real, concrete steps. Maybe this is showing me that I need to go out on my own.

There's the money. That's #1. I know what I need to do -- write code, push it, and get some sales going. After a few apps making $50k/yr, I'll be able to think about going it alone. I know what I need to do, I just need to do it.

Today was both a great joy and a major bummer. I thought I was done with the adding of new calculated values to this application I inherited, so in that moment, I was really happy. It had been a hard several days to get the values worked in at each level, but hard work pays off, right? I thought so too.

And after I sent out the email on what additions were now in the system, I was reminded of one I had forgotten.

Crud.

I had to go back to the code and work in another variable. Thankfully, I had the data in the system, so all I had to do was to work it through the reporting system. But even that was going to take time. Ugh.

Well... I knew this wasn't a one-day job... in fact, it's liable to take a few more, but today was one of those days I didn't want to talk to anyone - I just wanted to get through these changes and get something out. It's maddening when someone wants to talk to you in this mode. I know they don't realize it, but this is like ripping off a band-aid - just do it and then it's done.

Yeah, today has been a day when I've been barely able to keep my head above water, and had to spend a ton of time working on the feature set expansion for this unruly app that I inherited. It should belong on the Daily WTF but if I sent it in, I'm afraid that I'd be giving myself away and that's no good. So I dove deep into the code and hammered on it as hard as I could for the day.

Today I started the laborious process of adding more than a dozen new calculated fields to this somewhat shaky web app that I've inherited. It's working, and that's the best and worst of it, really. If it weren't working, given all the improvements that I've made to the system, then there would be a greater push to replace it with something that's really much better. But it's running. Yet it's barely running, and that means that working on it is exceptionally uncomfortable.

But it's something I have to do, and so today I started the multi-day process of adding more calculated values to the system. It's going to take a few days to get it all done, but as long as I'm slow and methodical about the additions, I'll be able to do this pass nearly error-free, which is going to make it a lot easier.

Not fun, but no one asked me if it was going to be fun.