What's it all about, Alfie?

This morning I decided it's been almost a year since I've gotten the latest FORTRAN compiler from the HPC for Mac OS X web site. They typically update ever couple of months, and with the release of Snow Leopard, I was in for a wonderful treat - they had a Snow Leopard version that generated 64-bit executables. This is fantastic news!

I downloaded the package, updated my uninstall script based on the tarball's contents, and then gave it a whirl.

  peabody{drbob}275: gfortran --version
  GNU Fortran (GCC) 4.5.0 20090910 (experimental)
  Copyright (C) 2009 Free Software Foundation, Inc.
 
  GNU Fortran comes with NO WARRANTY, to the extent permitted by law.
  You may redistribute copies of GNU Fortran
  under the terms of the GNU General Public License.
  For more information about these matters, see the file named COPYING
 
  peabody{drbob}276: 

Sweet.

I love that they build this package with auto-vectorization and OpenMP built in. This is just one of the most wonderful treats of life: to be able to work with such a wonderfully simple programming language on such wonderfully complex problems. Face it - there's no GUI for FORTRAN, it's standard out. Period. That means that you're not going to be making a cool web browser in it. It's not meant to do that. It's meant to push bits around and solve problems like never before.

There's no doubt that FORTRAN was one of the things that drew me into computers back in the 70's. It's just that incredible. I feel incredible joy in being able to use it. Thanks to the HPC on Mac OS X guys for making my day.

Well... today I was able to get the moving median-based filter into my web app without too much trouble. There was a bit of grief in that I hadn't planned on having multiple data filters on the alert inputs, but that's the way you need to code - make it easy and simple, and then if you have to re-factor code, then that's when you do it. In the beginning, there was no way to know - nor reason to believe, that I was going to need multiple data scrubbers. It took me a grand total of less than an hour to add the other filter, but then about another hour to debug it due to BigDecimal versus Double issues when dealing with the output of the H2 database.

I also added a few new features to the app in order to make it a little more nimble and lightweight for those users that are reporting that the app is a little too slow and cumbersome. I can understand it - they have loaded machines and this is just one of the many things they run. So I added a feature to just show the last hour as opposed to the entire day. This will really help if all you're interested in is the recent past. Might really help.

Other than that, it's been a pretty quiet day. No major updates to anything, just nice, solid, steady progress on the app. Not a bad day at all.

I've written a system of alerts for a web app that I've been working on, and one of the problems - as it always is, is that of bad data. Specifically, false positives and false negatives. If we have an alert system, then we need to be sure that when there's a reason to alert, we alert the right people. A data entry error is not necessarily a reason for alert. It might clear up in a minute, and if so, we shouldn't alert folks telling them that the sky is falling.

So there's some element of signal processing in this work. How to filter out the bad data points and keep the good. I've been working with some interesting algorithms as well as some pretty straight-forward ones. One of the simpler ones is a simple n-point moving average. I have this on the input of the data to the alert system to suppress the spikes but there were some very nasty problems associated with that algorithm.

Probably the biggest was that it really doesn't filter the bad data 'out', as much as diffuses it. Take a large negative spike with a small trailing bit. An 11-point moving average would look like this:

The problem is that the bad data starts at 7:25:34, but it effects the moving average for a much larger area simply due to it's magnitude with respect to the "signal" it's in. There's not a lot a moving average can do to smooth this out. Sure, other filters can, but I also need something that's going to be fast, so I can't be computing an FFT to get a frequency-domain based filter for this data.

As I was thinking about this problem and started thinking about the other simple statistical functions that were in Numbers (the program I was using to look at and play with the data. In a flash, the median hit me, and I started to run with it.

What if I did a 5-point median, and then did an 11-point average on that? Oh... that looked good. What about a straight 11-point median? See for yourself:

This is a ton better. While it can still effect the output, bad data has to exist for at least half the window in order to effect it at all. This is a pretty fair situation as if it's that bad, then someone needs to know about it.

Next week, I'll look at putting this into the code.

For the last several days I haven't written any posts because I've been deep into this codebase that I inherited and don't really like much. OK, that's being polite. But a few interesting things happened in these last few days that make me step back and try to see the forest for the trees. Maybe it's just because I think I'm done, and I'm in one of those euphoric states of mind that occurs when incredible pain is removed, and it appears that you're in incredible happiness... and then again, maybe it is honest happiness.

The Code

I'm not going to apologize for my take on the code. It is really horrible. Yeah, it's really that bad. It's not the worst code I've ever seen, but had it been even a little worse than it is, I would have said that it never would have worked for the purposes at hand, and a re-write would have been required. It was passible, but only barely.

So what's the upshot of this? Well... I'm stuck with it. I've been in this spot several times before with other projects. The code is so convoluted that only a very few (typically only one) person can go in and make changes. The process is hard, painful to do, and error-prone. It's a disaster waiting to happen. Consequently, it's something that has to be done very carefully.

But it can be done.

I'm trying to take away that last point: It can be done. It's not impossible, just very, very difficult. In that sense, I'm not really upset about the code, or having to work with it. It's something else entirely that I'm upset with.

The Timescale

I'm all for deadlines. I don't mind them as long as everyone knows them. If it's an unreasonable deadline, then when it's made, the statement of it's unreasonable-ness will be made, and expectations can be set. It's never an all-or-nothing proposition, but as long as everyone knows where they stand, it's fair. Something that's been happening in the last few weeks is that I'm getting the very real sense that there's a deadline, or timescale that I'm not being made aware of. There is a very real pressure to complete things as fast as possible - without regards to duplicating work and carefully removing bugs in the code.

Such might be the case, if there were a deadline that I weren't aware of. Something like "Mean Guy over there is saying our group doesn't work hard, and I told him we'd be done with this by the end of the month." I'm all for sprinting for a goal, but I need to know the goal, and that we're sprinting. Otherwise, I get the feeling that I'm being horse-whipped with things that make no sense.

• Sending the app to testing with users when we know some of the numbers are bad -- why not just wait until the numbers are right and then show it to them?
• Changing the Test configuration in the middle of testing for an hour so someone can see the numbers -- again, you're making this a wasted hour because I'll have to revert back to the consistent testing configuration to really be sure I have everything.

I don't say that there's not a reason for this - only that I'm totally unaware of what that reason is, and being in the dark, working as hard as I do, is not a comfortable feeling. I get to feeling very manipulated.

I'm not such a prima donna that I don't think all decisions have to come to (or through) me, I just want to be told what the decisions are when they effect me. It's about feeling that you know where you stand.

The Support System

This week I've had a real shock to my traditional support system. I have a few friends that I've worked with for years, who now work different places, and I chat to them pretty much every single day. It's not like we're texting teenagers, it's more like an informal way to communicate if we happened to be further apart than a shout.

I had been having problems with this project and was venting to these guys, and basically got slapped back down by both of them. One, said he had problems as bad with his, and didn't think it warranted the grumbling noises that I was taken to task for by management. The other is now a manager, and he said he wouldn't stand for that.

Interesting.

While I understand their points of view, and probably even agree with them (now), at the time, they were "keepin' it real" by smacking me down. Needed? Maybe. Helpful? Not in the least.

The problem was, as I realize now in retrospect, that I was mad at more than those asking me to do this, I was mad at myself for not putting my foot down and requiring some sense of timeframe, or setting a schedule, or at least not letting them push me to answer deliverables every hour (no joke). Asserting myself at work is something that I have a hard time with, especially when I'm new to a position. I don't feel comfortable saying "No" to even unreasonable requests. The word "possible" kills me.

So while I needed support, and didn't get it, I needed perspective more.

Lessons Learned

So after all this, what have I learned? I think there are a few things that I should work hard on:

• Expressing my beliefs about the cost of certain decisions - heck, maybe if I'd told them exactly what the cost was going to be, they'd have said "Nah, it was just an idea".
• Sticking to my beliefs and not being a doormat - there's nothing professional about being walked over by peers or management. If you are a professional, and in your professional opinion the plan has mistakes or omissions, then you should feel that it's possible to express them professionally.
• Know when to shut up - obvious, but necessary.

I'm sure there's more, but for now, this will get me a long way towards not getting into a situation like this again.

Today I realized something that's been bothering me for weeks. It's all about management's perceived need to rush something through the process, and what actually the business need is. Specifically, when you're not building a fix, or a critical feature that's time-dependent, then what's the bloody rush? Why should I find 30 mins right now to show something to a group of users when I'm in the middle of fixing something in the code and tomorrow will do?

This is all about management's need to set it's own, self-imposed goals, and has nothing to do with reality or the development process. In fact, it's counter to that process as a whole. I've been told to show something that I know is wrong in spots because management doesn't want to wait until it's fixed in an hour or so, worrying that we'll loose an entire day. So what?

This isn't business critical. This isn't time-critical. This isn't something they have been clamoring for. This is something that is an enhancement to the existing functionality and because the QA process is very lengthy, management doesn't want to wait an hour it doesn't have to.

Can you say Crack Monkey? Yes, I knew you could.

This makes me exceptionally angry about some management decisions, such as these. If you think I'm the one that can get the job done, and if you trust me to do it right, and quickly, why on earth are you so "ants in the pants" about a release? This is in-house development, folks. This isn't releasing the next version of the OS, or meeting a hard deadline. This is a simple matter of a manager losing touch with what's the best way to manage people.

I've got to remember that I'm angry at the decision, not the person. It's been happening for several weeks, and it's only just now that I've realized it. I need to say "No" more. Just plain "No".

I'm as angry at myself as I am at him.

I was looking at a few git-related things this morning and noticed that the open source Mac OS X git GUI client, GitX, was now at version 0.7.1 which was a nice jump from the last version I'd gotten. In this version, it's picking up the user's images from Address Book, like Mail.app, which is really neat - nothing fancy, but neat.

Also, there are a lot of fixes and features in this new version. Very nice indeed.

While I can certainly deal with the installation of Mac OS X apps, I have to say that when I see an Installer package, I cringe slightly. You never know what they are doing, and if it's not from Apple, the reasons for putting things outside the app bundle are really minimal. Sure, utilities, web add-ons, they make sense. But apps shouldn't. Period.

Then the problem is the delivery mechanism. Is it a DMG file and then the user has to drag it out of the mounted disk image into the usable drive space someplace? Sure, symlinks on the DMG make this a little easier to drag it into the Applications folder, but still, that's a pain.

Then there's the Zip file where the application is unpacked into the Downloads folder. Not bad, but again, it has to be manually moved to a better place. Better, but still not ideal.

Then there's the Move on Startup idea. Delicious Library (among others, I'm sure) has this feature where if the application is launched from a DMG image or the Downloads folder, it asks the user if they want to move the app to the Applications folder. This one-time step, along with Sparkle, is about the most fool-proof way to deliver an application. Make it a Zip, let them move it or launch it right from the Downloads folder, and then move it into it's final spot. Sweet.

It gets even better because there's a GitHub project for this behavior so anyone can make use of it: LetsMove. I haven't dug into it, but it's probably not all that hard to detect the location and the tricky part will be the move. But still... there it is. I'll have to make use of that (and Sparkle) in my projects that I take public.

Today was one of those days that, I believe, defines what kind of person you are. Are you a person that can knuckle down and do what's needed even though every fiber of your being is screaming that it's the wrong thing to do? Where is the individual and the team defined in your mind? How much can the team request as sacrifice from the individual? It's interesting questions because we all think we know where we sit on the questions, but it's really when push comes to shove that we learn where we really sit.

Today I thought I was done with a very distasteful bit of coding. To call it a 'bit' is to call Lake Michigan a 'pond'. It was a week of 11-hour days of the most intense, horriffic coding that I've done in a very long time. But, like ripping a band-aid off, once it was done, I could move on to nicer things.

Yet, as soon as I was done I got several requests from folks "...yeah, I wasn't right on that, it needs to be like this..." and "Great! Now add this". In the end, I received in a 4 hour period enough work to keep me busy for the rest of this week, and possibly into the next.

To add to this insult, some of the requests were for changes in the Flex client part of the project. While I've done some Flex coding, it's nothing like what I needed to do the request, and I said so. The original author that handed me this project said "Sure, it's doable". I wanted to scream at him then and there: Shut up unless you are going to do the change! But I didn't. I'm already too "gruff" for the office.

So I'm back at the bottom. It was as if all the work didn't matter - worse it was because of all the work that I got more. I wanted to sit and cry - or maybe just plain leave. I had been telling management that this system was a pain to work in... like it taking more than 5,000 lines of java code to add less than a dozen fields to the system... it's a monster and it's not getting any easier to work with given all the additions.

So to hear that I had effectively brought down more work in this horrible project upon myself was simply more than I could bear. I was more than a little angry at the original developer who, like Pilot, had washed his hands of his horrible incarnation. If only...

Anyway, I had a bunch more work to do, so laying it out in my notes, I dug in and started working. What I found was that the requests had generated a ton of new work. Amazing. They think these things are "simple", or "easy", and in a good system, they would be, but this isn't a good system. It's not even "reasonable." It's crummy - and that's being kind.

So, now I'm down... then I'm kicked... and it's not getting any better anytime soon. I still need to do all this work, and then I'm sure there's going to be add-ons after that. In a very real sense, I've learned that my skills are like anything of value: a double-edged sword, and I'm feeling the cuts very deeply today.

Because I can do this, I'm told to do this.

Regardless of my warnings, advice, professional judgement... none of that matters because a few days of my time is considered insignificant and the results fantastic. Who cares if I want to scream every minute of every hour while I'm doing it?

Gotta love the "bottom line".

With the problems I had getting PostgreSQL support in PHP, and the desire to take control of my toolset, I went looking for a 64-bit version of PostgreSQL rather than resort to building it myself. What I found was that William Kyngesburye (a.k.a. KyngChaos) maintains a dual-build of PostgreSQL - i386 and x86_64 that's even Snow Leopard compatible! Marc seems to be out of it, and even the EnterpriseDB guys have no plans for building a 64-bit version.

While I don't need a 64-bit version per se, I do need the 64-bit libraries and this was an easy way to get them.

The upgrade was pretty easy - dump the database, move/remove the old install, and then follow the package installer. It should "just work", but it was missing the launchd file. I found something close to it on the net, and just fiddled the parameters to make it work:

  <?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
  <dict>
    <key>Label</key>
      <string>PostgreSQL</string>
    <key>UserName</key>
      <string>postgres</string>
    <key>RunAtLoad</key>
      <true/>
    <key>EnvironmentVariables</key>
      <dict>
        <key>PGDATA</key>
        <string>/usr/local/pgsql/data</string>
      </dict>
    <key>ProgramArguments</key>
      <array>
        <string>/usr/local/pgsql/bin/postgres</string>
        <string>-e</string>
        <string>-i</string>
      </array>
    <key>StandardOutPath</key>
      <string>/usr/local/pgsql/var/logfile</string>
    <key>StandardErrorPath</key>
      <string>/usr/local/pgsql/var/logfile</string>
    <key>ServiceDescription</key>
      <string>PostgreSQL Server</string>
  </dict>
  </plist>

Once I created that file as /Library/LaunchDaemons/org.postgresql.postgres.plist, I was ready to go. The instructions in the package work perfectly for starting it up, but now that I have the file there, the installer is supposed to restart PostgreSQL.

Once I had this installed, it was a simple matter of rebuilding the pgsql.so file using the provided libraries and it worked like a champ!

Now I feel I can move forward with this guy's PostgreSQL packages and making the PHP library each time I upgrade. Not a bad place to be.

[9/20] UPDATE: well, when trying to upgrade the PostgreSQL on my 20-inch iMac with 2GB RAM, I got a ton of errors about the SHMMAX value not being large enough to complete the operation. Basically, PostgreSQL wasn't able to get the shared memory to complete the operations. I did a bit of googling and found this post about setting the SHMMAX value.

When I set the SHMMAX sufficiently high, I was able to get the initdb command to work, and then I had to set these parameters in the startup by creating /etc/sysctl.conf which is read by the 10.4+ kernel on initialization. The file contains:

  kern.sysv.shmmax=1073741824
  kern.sysv.shmmin=1
  kern.sysv.shmmni=32
  kern.sysv.shmseg=8
  kern.sysv.shmall=1073741824

and after a reboot, I was able to finish the migration by loading up the database and things were working perfectly. I then compiled the PHP PostgreSQL extension and added it to the config and all was working quite nicely.

Quite an adventure!

Well... everything costs. Pretty darn true. Not a lot in life is free, and that which is most of the time only seems free at the time. So it's not at all surprising what happened today.

Ripping the Band-Aid

I'm a Band-Aid Ripper. When I have a task that I know I'm not going to like, I try to steel myself to the pain, and then just rip. Get it done. Get over it. Move on. There's no reason to draw out a painful process to make it long and painful. It's enough that it's painful. Rip it off.

In all honesty, I didn't use to be this way. I was scared - plain and simple. I hoped, like all kids, I guess, that if you do it slowly, there won't be any pain. All the pain, you hope, is in the ripping. But that's not the case, is it? Nope. In most cases, that band-aid is stuck, and it's going to be painful to get it off. So it's just up to you to take it off on your terms.

So I've been working for the last several days on adding a few calculated values to this application that I have inherited at my new job. It's not what I'd have written. In fact, I look at it and I think there's just about nothing I think that was done right. It's a mess. Plain and simple. Even the original author has called it "crap", "a hack", and not once or twice.

But I'm able to make it work.

So there's no need to rewrite it. It works. So when I had to work on it, I knew I had to attack it... get into it, get it done. The longer I stayed working on it, the angrier I was going to be. But why?

To give you an idea, to add the 10 or so calculated values I had to add over 5,000 lines of code - totaling nearly 500,000 characters. Just to add a few fields. The problem is in the way it's structured - lots of interfaces and objects - but all functional in nature and none really encapsulating the reporting behavior. This means that when I have a report that looks at a certain variable trading during the day, there's going to be an opening value, a trading value and a net value. Rather than create a object for this, there are three values passed around the code.

Add to that, we might need to difference these, and we get nine values: open, trading, net, and then initial, final, difference. If I add 10 variables, then I'm really adding 90 arguments to many methods.

It's not good. It just wasn't designed properly for what it was doing. But I got it done.

How it Looks to Others

After I got this all done, I had a meeting with my manager. Basically, he believed that there were two ways to go about doing this:

• re-work the code to add in the values... grumbling and grousing while you do it... giving off the impression that you're capable, but not a guy anyone wants to work with - possibly someone that's just a pain.
• re-work the code to add in the values - but this time be nice about it. Everyone wins. You're happier, no one gets the wrong impression. Win-Win.

I wanted to say that Fantasy Island is a nice place to live, and I'd love to visit him there, but I didn't. What I said was that I'd try harder to keep my mouth shut, and not give off the wrong impression.

But is it really the wrong impression, or just an unpopular impression? I have no doubt that I'd love to be happy when working with all the code I have to work with. I don't enjoy getting upset at code, or developers, but when I have to extend code and the number of arguments to a method exceeds 255, then there's something terrribly wrong with the code and it needs to be fixed.

But if it's not outwardly broken, why fix it? There's no business motivation for it. It's working, after all, it's just a complete mess.

So I'm left with a manager thinking I'm making the active decision to be unhappy when I'm just responding to someone doing a horrible job and being held up as a paragon of development virtue.

Where Do I Really Fit?

I'm left thinking about the experience and wondering if Liza's not spot on when she says this happens with every place I work. That my expectations of others are far too high for mortals, and while it's less than I demand of myself, the vast majority of people simply fall far short of my standards, and these problems are bound to arise. I don't think I'm that demanding. But I can certainly see her points.

I'm in an industry where developers are paid a ton of money. I mean a ton. If you're making in the six-figures annually, then by golly, you had better be giving it your all every day - or at least more days than not. That's a ton of money, and you need to be worth it. There is no free ride.

So I do get a little miffed at these guys making this money and not feeling compelled to fix up their messes. Far too many developers write something and hand it off never to think about it again. Where's the pride in what you have created? Where the dedication to your craft that you'd hand off such crap and call it "done"? Clearly I come from a different time. Which leads me to the ultimate question: Where do I belong?

I'm coming back to the thought that I need to be on my own. Writing software that meets my standards. It's not easy, and consulting is the easiest way to make the transition, but that leads to a lot of consulting and not a lot of your own code getting written. No, I'm thinking it's time to start planning on getting out on my own. I've toyed with the idea for a long time, but never really had a motivation to make any real, concrete steps. Maybe this is showing me that I need to go out on my own.

There's the money. That's #1. I know what I need to do -- write code, push it, and get some sales going. After a few apps making $50k/yr, I'll be able to think about going it alone. I know what I need to do, I just need to do it.