What's it all about, Alfie?

This morning one of my servers started sending out prices of nan, which is clearly not the right thing to do. This caused my downstream systems to stop ticking, and that caused a lot of production issues. First off, I had to find the problem, and patch the downstream systems so that they would be effected by nan values anymore. For the most part, they were protected, but from this one server it was far far too optimistic. So I had to batten that hatch down and get those changes into production as soon as possible. Then I could turn my attention to why?

The why? turned out to be a volatility curve that was primarily short-term, and rising significantly at the end - coupled with a far-term warrant that ended up with a very large volatility at it's expiration due to the interpolation of the curve. The fix was to note what the problem was and then to put into the code a configurable parameter that would 'cap' the volatility at expiration and not allow the options to be valued if the volatility exceeded a given cap. This works great, but I tell you... there's nothing like coding a fix for a system with 50 users breathing down your neck. It makes you make sure you're right - and fast.

Certainly one of the more interesting things I've ever worked on is the Business Continuity Planning or BCP for short. It's called a ton of things in different shops, but it's always the same thing - What happens if the building is wiped out?

Where I'm at now, we have sites around the world, so the obvious BCP plan is to make one of the other sites the BCP site for the main production facilities. The easiest way to do this is to have the secondary site active at all times then switching over is as simple as redirecting a few clients, etc. So, that's what I'm doing. Now, I've known that this needed to be done for months and months, but the management never really gave me a lot of time to do it. Now, with audits looming in the future, it seems that now is the perfect time.

The interesting things I'd like to put into this BCP plan are really more like hot fail-over plans so that we can tell that the plan will be easy to execute because we can see that the hardware and software are functioning every day as they are intended to. However, that means that I need to do a lot of work to get things to this point. That's been my day today. Nothing fancy - just a lot of details that need to be looked into, and done to get the systems up in the other locations. It's certainly not glamorous, but it's what needs to be done.

Today has been a good day for coding... I've fixed several issues with the Server and they will test tomorrow and then deploy, and I've seen the new MacBooks from Apple. I have to say, the hype on the rumor sites this morning played this up a lot bigger than it turned out - yes, the speed is nice, and the drive size increases are nice, but really, it's a little move and not something that's really earth-shaking.

It was fun to watch the diggnation podcast today. Those guys are so like how I used to act with my friends that it's always worth a good laugh. The stories are sometimes disturbing, but that's life. Also, the weekly News from Lake Woebegone podcast came out today and that's another giggle.

Pretty calm day... coding... giggling... not bad at all.

Sunday night... all's quiet... then comes the ring. It's work... things are down... please help.

It's not a great way to start the week. Last night I got called because things weren't working as they should. I come to find out that the linux servers have been rebooted without letting me know. This is a serious problem as the processes that run on most of them can't be in the chkconfig profile to start on startup. So I was trying to fix one thing, and found that another process on another machine wasn't running and that was the source of the problem, etc. It was a mess.

When I finally got things under control this morning for production, I counted seventeen, that's 17, machines had been rebooted and not a word on the planned outage, or a phone call on the work done so that I can get things going again in an orderly fashion. So I had to send an email to the Unix Sys Admins asking why I wasn't told anything. No reason, no excuse. Just my bad luck, the answer.

Next time, they are going to call me. I manage apps on 29 servers around the globe and they all depend in one way on each other. I can't be in the dark like this on a regular basis. No fun at all.

I work with some funny people, and by that I'm not talking only about the laugh-out-loud funny, I'm talking about the what-on-earth-are-you-doing funny. The other day I was talking to someone about some design changes they were interested in making. While I had nothing against the changes, I saw the opportunity to help this person see beyond their initial belief that the design was broken and see how to fix it. Nothing major, and the change might have been really good, but it was a nice to see if this person could see into the problem and see how to fix it.

So, I ask if they see how to fix this. No response. So I explained my idea, and say that it's not necessary, but just a different way of doing the same thing. I'm told it's not the same thing... it's adding another thread. Hmmm... No, it's not, and therein starts my explanation that it's basically the same thing at a high level. Instead I get an argument and insults.

You'd think that someone making as much as these folks would have the common sense that even if they didn't have the intelligence to know the right answer, they'd have the common sense to not insult someone in a professional setting.

I've learned all I need to know about these people. Funny, they probably were trying to enlighten me on the threading issue, but they told me so much more.

I was reading the web this morning and real a great article at CNET about SGI filing Chapter 11. The guy got it right on target - it's about the passion and belief that you're working for something that will change the world. There's something that reminds me of grad school when you're working on something that you're convinced is going to set the world on fire. It's sad to see SGI not survive. I'll never get rid of my SGI, that's for sure.

OK, I'm the first person to admit that I enjoy the fact that I understand many (but certainly not all) of the systems in use at work. It makes it easier to understand where the ones I build fit in and what jobs are best done by other systems. But there's a dark, ugly side of this... the 1:30 am phone call.

In the past week I've been called three times at 1:30 am to help the operators figure out problems they are having with systems. Every time, it was an upstream system that had the problem, it was just that when it got to my system it was clearly apparent. So they called me. Unfortunately, all I was able to do was to point them to the systems that had problems - which they already knew from other sources. So I'm not really helping a lot, but I'm certainly wide awake and that's no fun.

I have put together web pages to assist in figuring out what system is at fault, and what might be done to clear up the problem. Of course, these aren't foolproof documents as different kinds of problems keep popping up and we have to adapt to the new data sources, etc. But it's a darn good start, and far more than most developers give their support staff. But if they don't read it, it's not going to help them, and it's not going to stop the 1:30 am phone call.

So now I can tell them to read the web page and call me back if it isn't solved. But that's still not stopping the calls. Once they are used to me having the answers for them they are going to skim the docs and call me. It's a little disappointing and very tiring.

I've done it again. I can't for the life of me tell you why I do it other than I'm a Dad, and when it comes to what's right and what's easy I try to teach my kids that it's the right thing that's worth doing. Always.

So I'm trying to suggest that things at work change. Not a lot. Just make us look at what we're doing. Stop and look at things. That's all. But I've been accused of being afraid that my job would be eliminated by the success of this one project. Even if this project was perfect... even if it did everything it was capable of, and then some, I wouldn't be worried. Why? Because I've talked to the management and they have me working on more things every day. I'm in no risk. Yet everything I say is going to be discounted by the people working on the project because of what they feel are my actual motivations.

So in a way, it's a sad statement of paranoia. They feel certain that I can't be trying to help them so everything I say is tossed on the floor. They don't stand a chance of getting better without some help, but they don't want to listen to me.

Still, because I'm a Dad, and I'd tell my kids to do the right thing I'm going to keep trying. Wasting my time, probably.

It's been said a million times: It's not whether you win or loose, it's how you play the game., and that's a great thing to tell kids to get them to realize that in life there are many times when we all can't win. It's a good lesson to know. It's a good lesson to learn. If I can extend it a bit, I'd say (for software development): It's not whether it runs or not, it's how the code was built..

I've been involved with many development efforts over the years and the without exception the phrase disposable code was universally inaccurate in every place it was used. The only code that I've ever called disposable code is that which I deleted right after I ran it just once. Everything else is going to get used a few times, and if it's used in a trusted capacity, then it's not disposable. It's got to be dependable.

Currently, I'm working with a group of software vendors whose product is fed through a Java interface. This Java interface doesn't have any package structure to it to help resolve the namespace issues that might arise. It was also delivered as a zip file of .java and .class files - no JAR file. Now there's no reason that these two things need to be in-place. Java certainly allows for package-less class creation, and the CLASSPATH can include directories of class files, but what vendor in their right mind would deliver their interface in that manner?

Oh, sure... I've written test classes and 'Hello World' apps without either, but those aren't the interface to my company's flagship product. They are simple little test apps that don't get any further than the development box. When things go into production - even for in-house development, there's always a package path, and they are most often delivered in JARs. The latter is bumped to an always when the code goes outside the company.

Yet I'm sitting here looking at the opposite. I'm stunned. The marketing types spun a good story, and while we (thankfully) haven't purchased the product (yet), it's these kinds of suprises that make me want to say "No matter how nice it seems, it's not built well under the covers. It'll break and we'll be left holding the bag."

I'm not sure how this is going to turn out. I'm hoping that when I ask for code samples, and any software engineering practices they have or use, that these things will stick out as incredible anomalies and the rest of their product will have been created with much greater care. But I'm not betting on it.

Well... dealing with boredom is certainly one of the more difficult things I've had to deal with lately. When you've designed and built large, serious systems, it's hard to stay mentally challenged doing web pages. Especially when they are all basically the same - some data source feeding a graph or a table. Yet it's really important to feed and care for your Creativity.

I'm getting the impression that I'm not doing a good enough job of taking care of that part of myself that is the Creative side, and it's costing me more and more each day. I believe what I do is basically creative, that the job of building systems is as much art as anything else. Yes, there's certainly a lot of science, but there's a big difference between only using science and using art as well.

But like any tool that's not used daily, creativity will get rusty and you need to exercise it for those times you're really going to need it. Seems obvious when you think about it in the same terms as any other skill, but I'll admit to having neglected by creative side of late.

So I need to do something about that. Don't quite know what it is, but it certainly doesn't have to be something to do with programming. I built a fence a few weeks ago and it was a lot of fun and exercised those same creative forces/muscles which felt very good. But I can't keep building fences - they're just too darn big. I need something that's portable.

So I'm going to go on the hunt for something that's more reasonable and in the end serves the same purpose. I've searched osx.freshmeat.net and haven't found anything - yet, but I have a feeling I'll do some more searching before I give up. I've really enjoyed working on the OpenSource projects that I've worked on in the past, and they always needed more help than they had.

If that doesn't work out, I'll go back to some of the projects I've set down in the past and pick them up again. Sure, they're not the most interesting things to do, but it's something and exercise is like that - it's something.