What's it all about, Alfie?

I've been having a lot of problems lately with my Sun box at work. It's getting a lot of NFS errors, and on Solaris, that means the box locks up. If I unplugged the network, waited a minute, and plugged it back in, it would come back for a while. But it'd go away again. The Unix guys tried a new NIC in the box - no good... they tried different cables and ports on the switch... no good, finally we ordered a new box thinking that this was a bug we weren't going to find. Thankfully, I was wrong.

Yesterday, I looked at another box under my desk and noticed that it was on the same subnet as the troubled box. I decided to swap the network cables completely eliminating the network from the picture. Lo and behold... today my linux box (the one that I swapped the cable with) died. Yup... no pings. Never happened before in all the years it's been running for me. But it did today. So I called the network guys and told them of the problem. They found problems in the switch - maybe they weren't there before, don't know. But they found problems and they moved me to another port on the switch. Service was restored.

This was a super simple test that I should have looked into many months ago. There was no need to order the new box, but we'll keep it on hand as a spare, just in case. This test was able to completely isolate the network problem with a working machine and that's all it took. Now we know.

Whew!

This is great news! I wanted to try the latest beta of Firefox 3 - Beta 2 as I've been reading pretty nice things about it. So I go to get the source to compile it on my Fedora Core 5 x86_64 box and realize that I'm missing cairo-gtk, and don't feel like tracking it down right now. The reason I built 2.0.x from scratch was that the tarball of firefox 2.0.x for linux simply didn't run on my x86_64 box in 32-bit more. Simply would not work. I looked at a lot of things to try and get it to work, but no luck.

So I think Hey, I'll give it another try! and downloaded the linux 32-bit code for Firefox 3.0b2. To my complete amazement it worked! I was stunned. I then was able to link in the 32-bit Java plugin just as nice as you please, and Bingo! I'm running Firefox distribution on my 64-bit machine. This is really great news.

I'm glad they have done this. It makes it a ton easier for me to stay up to date with the fixes, etc.

I'd like to think that I'm not a fool. I know that capitalism is all about the smart, strong, clever, industrious, getting more from their efforts than others. It's Darwinism in business. So I'm not one to expect a lot of day-to-day patting on the back from the people I work with - or for. Every now and then a Thank You would be nice, and I get them often enough for me at the Shop. Yet today something happened that really upset me - and something I foresaw about two and a half years ago.

The CTO decided that he needed to have a Harvard Business School Org Chart for his part of the Shop. Why? Because he didn't want to have someone they are interviewing for his replacement look at the way things are now, and say "What kind of horrible manager are you?" Which to me begs the question of "I'm leaving, what do I care what you think?" or maybe even "The kind of horrible manager that instills great loyalty and affection in his troops, what kind are you that needs a Harvard Business School Org Chart to run an organization?" But I digress.

This Harvard Business School Org Chart acknowledges the alignment of the business into three pillars, and correctly identifies the technology leaders for each pillar. The org chat then takes a left turn there into the land of what might be, but isn't, and says that the pillar I'm in is then divided into three groups: development, production, and something else I can't remember for now so I'll call it 'candy canes'. Further, the head of development - because we have to have a head of an org chart box, is an old friend of the CTO and head of the pillar.

First, when they brought in this old friend two and a half years ago, I told my wife this was going to happen. When the going got tough, and they laid off tons of folks, they asked me to handle all the infrastructure of the pillar. Understandable, and I did it. Now they have a little breathing room and they want to put me back into a little box as if that's where I always was in the first place. Not so, Mr. Magoo.

The business isn't aligned like this. I do not sit completely in development - or production, or candy canes. I cover all three - and they want me to. They count on the fact that I cover all three and yet when they come to the Harvard Business School Org Chart, they don't want to draw it as it is - they want to draw it as they wished it were.

The truth is that we are aligned more by application in the pillars than by anything else. That makes the most sense in a small organization - who's responsible for what? That means doing everything for that project - soup to nuts. That's how it's really run. Putting in these mythical org charts about how they think it should be is just the caving into pressure to try t make a place that's unconventional look conventional.

Trust me, you may want to make this place look conventional, but the people in it are far from conventional, and like it that way.

So I was asked what I thought about this new org chart, and I told him it trivialized my accomplishments and daily contribution and sacrifice. He didn't expect me to react so personally, but when I reminded him that I take everything personally, and that's part of the reason I'm so good as my job, he knew I was right. This is a sham. I've got nothing against the guy that's now the Head of Development... he's a decent guy. He's just not my manager in any sense of the word. And putting it on paper in a Visio diagram is not going to make it so.

In fact, they don't want me to change how I do things. They want to leave things working just like they are - they just want a new org chart. Which comes back to the appearances factor. Trying to look like a cow when you're a horse. Silly. Dumb. And ultimately, disrespectful of those people that work very hard for you to make you a success.

Today was the first of eight days of "training" for a package that the shop has purchased at great expense and very little long-term thought. This training is required because they require that you are a certified developer in order to get technical support from them. It makes sense... why answer a bunch of silly questions from users when you can force questions through developers that have gone through your "classes" and at least have a working knowledge of the product. That's the theory, at least.

The practice is something else. An entire day spent on SQL - or a derivative of it close enough to be considered SQL, and where it differs, they should have just changed the syntax to make it look like real SQL. Imagine that they changed the inequality operator from '!=' to '^=' - just to be different. Why? No idea. The guy teaching the class had no idea either. But there it is.

There are a few changes - no inner selects, you have to use temp tables, but that's not hard. But we spent hours there doing queries that we'd never do in practice without the schema right at our fingertips. It was educational for about the first hour, and after that, it was just moe of the same with a lot of waiting while we flipped through slides.

I realize this might not be this guy's main job, but he's representing the vendor to us, the customer, and when his response to a statement from use like "That's a bug, it should be fixed" is "That's the way it is, it's in the docs" he's missing what we in the biz call the Big Picture. I'm sure he's a nice guy, but come on! When your customers are telling you this is a bug, then you at least say, "Hey, I understand, and I've tried, but they aren't budging". And then when we say "Hey, an error message to that effect would be nice", you agree and talk to the product developers. You don't giggle at us, like you have so many times before and then say that's the way it is.

So for the first time in over a decade of class work, I got a headache from this class. That's bad. I can't believe I have seven more days of this. It's got to get better, or I've got to go through it quicker.

I want to believe. I really do. I want to believe that people are basically good and honest and I want to believe that they will look out for each other - at least a bit, in a jam. I want to believe that people are honest and have honor, and work in a mode where they instill these characteristics in others.

I'm honestly shocked and hurt when this doesn't happen. More so when it doesn't happen around me with people I have known for years.

Some will say that I'm not learning from the historical evidence available to me. And I'd have to agree that they are right. If I look at the historical evidence, key members of the management team here have proven themselves to be somewhere between having selective memories and outright deception for their own ends. Yet I still want to believe that this time they are acting honorably and honestly.

Silly me.

So here's the story of the week... the shop has decided to buy this package without really technically diving into it's capabilities to see if it'll do all we need (not all we want, all we need), and so now, millions of dollars later, they need to ensure success. No surprise there... I've seen it happen time and again. Typically, this is where I come in. They come to me and say "It's got to work. Make it work." And I make it work - as best it possibly can. There are times that it hasn't been 100%, but it's always been good enough to declare success, and that's all the management wants.

So this time we're poised again to do the same thing. Then yesterday I get into a late meeting where it's now clear that the work I would be doing on the new version of the product has to be hamstrung by the fact that it's got to work with the old version that I had been told would never be deployed here. Hmmm... I see the selective memory popping up.

So now we have to deploy this old version that isn't even half of what we need but we have to do it because the guy really running the project says we have to. I can see his point of view - minimize the changes in the migration from old to new, but I disagree that the minimization of the changes should take precedence over the fulfillment of the potential of the new version. After all, we know the old version is not nearly what we can live with, why force us to live in it's limitations?

So it's now clear that I will not have the freedom I need to make sure this project succeeds. In a way, I've been given my freedom by that one simple meeting. Oh, I think the management team hasn't figured that out yet, but if not now, they'll see it soon when I say "Hey, I'm sorry, I can't do that, based on the schedule and work outlined by the project manager." Then it'll hit them that without power, responsibility is an impossibility.

As an aside, these interactions make me hold my management in contempt. If they don't want me to do a certain job, then just say so from the onset. If they do, then have the honor to do so, or tell me that things are changing and realize that they can't hold me to any sense of responsibility or a project that they themselves will not give me power to control.

Times like this I want another job. I know it happens everywhere, and when it happens where I've been, I've wanted to leave those jobs as well. I like to work. Period. Ask me to do a job, and I do it. Try to con me into some impossible situation and see if I'm dumb enough to take the bait and try to make this looming train wreck miss is not respectful nor honorable.

Nothing is perfect. Period.

Sometimes we're lucky, and some times we make our own luck. Sometimes.

I was just sitting here thinking how much I really enjoy the position I have today. I really do enjoy this industry, not that I'd have ever thought I would - from the outside looking in, but once in (thanks to a good friend) I have found it to be one of really interesting people, very challenging problems and more than enough variation to keep me interested for more than a decade now. Pretty lucky, I'd say.

It's funny to me to see how the most captivating parts of this job are the ones that take me back to the work I did in grad school - numbers. Data. Lots of it. The market data is very interesting to me. Getting it from the provider, making it faster, better, easier to use - that's a blast. Getting it into a simulation/analytic engine is also a ton of fun. Seeing all the numbers roll around from the market data all the way through the various systems to show up finally at the client is a lot of fun. Number plumber - that's me. Kick in the pants.

It's important to realize that after all the work I've done that has nothing to do with my degree, it's neat to realize that I really have come back to my 'roots'. I guess you gravitate to those things you really love doing. Sweet.

Today I spent a lot of time on migrating off the data source that we're trying to move off of, and on to the one that's got historical pricing data that we can use without the associated rules and regulations of the original source. This meant doing a lot in the market data server to make this other data source look as close to the original as possible, while still taking advantage of the few things it was doing better than the original.

One of the more interesting things is the server's respecting the field names that are used by the client in it's requests. If a user is using the Perl interface, for instance, and they ask for 'close' or 'Close', they don't want to get data back that has the field name of 'CLOSE'. That makes it hard for them to match up what they sent with what they received.

True, we could make it so that they were required to ask for 'CLOSE', and then all would be OK, but that's a little too restrictive for me. So I had to go into the server and fix up the way this data provider was handling the field names. Then there were the changes that needed to be made to the cache to hold the returned data - again, 'Close', 'close', and 'CLOSE' all should cache to the same values, so we had to put in case-independence in the code there as well.

In the end, I got all the pieces working and was able to move two more things off the old provider and onto Fusion. It's now up to all the other users of the old data provider to change their code and migrate from the old to the new. It's not going to make a lot of people happy, but then again, there's no way they were going to pay $1 million to get this data. No way at all.

In the recent weeks one of our data vendors has contacted us about our use of their API. Specifically, they created this API for themselves, as their product runs on it, but they wanted to make it possible for users of their product to also get at this data in a programmatic manner. It was a nice thought. We looked at this and decided that it would be even nicer if the general users of this data didn't have to individually write to the API. Maybe even add a cache to the data received so that for each day, a single piece of data would only be fetched from the provider once - then after that, subsequent requests would hit the local cache. It's an excellent idea, a wonderful product, and now they are interested in charging us more than $1,000,000/year because of the data we get from this.

The reason for this is that this vendor decided to un-bundle the API from the product they sell that uses this API. They decided this because, of course, people were using it and they saw it as an additional revenue stream for their company. I can't blame them, it's capitalism as it's best - make something someone wants and then charge for it... if they pay, then you have a winner, if not, then you are out the work to make the product. So they came to us and gave us a pricing model for the data we're getting and it averages out to a little over a million dollars a year.

Given that the bundled product we got originally is about $2,700/month - this represents a large increase, and it's likely that the business will simply not see that this data is worth seven-figures a year. Again, that's their job - cost/benefit analysis... it's a necessary part of all businesses.

The problem comes in that it's not this easy - the vendor isn't just offering us an upgrade path, they are also (at the same time), holding a gun to our heads. If any of the trading business requires this data to function (as some do), then turning it off is simply not an option. Migration is the key, but migration to what? We'll have to look into the different competitors of data and look to see what each has and the cost, etc. Then work that into the systems one by one and then we'll be ready to turn off the high-priced original vendor. It's more than an afternoon of work.

The other side of the problem is that the management here has known about this issue for years. Yes, years. Each time they talk down the vendor from the ledge, and they agree to let us continue to use the data under the existing contract. When we start to get too much data from them, we find alternatives and migrate a few applications from that one source to another under controlled conditions and hitting the easiest, biggest offenders first. Makes a lot of sense. Usage of the data drops, they are happy, and life goes on.

But not now. No, they are convinced that the vendor is out to get the full price and lock down all the data. The problem with the lock-down stance is that there's no way to really do it unless we shut down all usage of the data. Imagine, I use the data in a spreadsheet. I save that spreadsheet on a network drive that others look at. That's redistribution of the data and they won't allow that. The only people that can see the data are the people they license to see the data - no exceptions. And if there's a possibility that someone else might see the data, then that has to be shut down. Their latest position is very restrictive, and to follow it to the letter will mean you really can't use the data for anything "saved" in any way, shape or form.

Clearly, we need a new source for data, but management is taking the attitude that we need to "spread out" the usage and then pay for what we really use - sticking to the letter of the license. But there won't be any data that we can use to the letter of the license as it's a server, and therefore might serve up data to something that possibly makes it available to someone not intended to see it.

It's dumb, but I'm not going to fight it. If they want to shut it down, fine. They aren't going to - not without a replacement, and when that replacement arrives we'll simply slot it into the server alongside the old, restrictive one. It's going to be a mess, but they seem to have given up on talking to the vendor on this. Maybe it's not possible, but I can't imagine a vendor that would take nothing over something - which is what they'll get if we shut this down. But then again, they could be trying to shut it down on their end, and this is their way of doing it. Vendors have been known to do dumber things.

Last night I had a release of the server and in the release package I thought the shell scripts were included. Now, it's obvious what's happened, but I really thought they were included. Then, while walking to the train, and with no hopes of going back and fixing it without being home late, I went ahead and got on the train. Then I got the call. Yup, what I had thought about was the case - the deployment package did not include the shell scripts and there was a critical path change in this release. Crud!

It was easy enough to fix when I got home, but the problem was that the opening greeks were going to be missing because of the failures in the calculation nodes. So I had to reload all the underlyings with positioned options. This took about 40 mins, but could be done without having to take the server down. Since the Hong Kong day was already in full swing this was the far better idea.

As I was doing this by hand, I was thinking that it would be nice to have a little program that would do this for me. Well, this morning when I came in I started doing a little digging and sure enough, I had written a tool to do just this many moons ago. I had just forgotten about it until I had the time to dig into the possibility. Next time, I'll remember.

My point is that while I know it just makes me look more human (so says my wife) it really is terribly embarrassing to make mistakes like this. Sure, it was easily fixed and I fixed it, but the fact that I made the mistake was what bugs me. Deep down, I knew there was a problem because it came to me like a flash on the way to the train. So there was something there trying to tell me what I was doing, but I was sure it was going to be OK.

Wishful thinking.

I need to lighten up a bit. Everyone makes mistakes and I have no problem forgiving others, I just have a hard time forgiving myself. I remember a scene from My Favorite Year when Swann (Peter O'Toole) was yelling to Benjy (Mark Linn-Baker) about going on TV live - "I'm not an actor! I'm a moviestar!" Benjy gets angry at Swann for starting to walk out and says something about how Swann has always been his hero, and heros never walk out. Swann is upset with the responsibility this places on his, as he really likes Benjy, and reacts badly by saying something to the effect of "I'm not that person! I'm just a person.". Benjy responds with the best lines in the show: "I can't use my Alan Swanns life-sized. I need them as Big as I can get them." And then the clincher: "Oh... and by the way, No one is that good an actor." Swann comes around and saves the day. In the end, seeing that the person Benjy sees in him is really there, if he just believed in himself a bit more.

I feel I need to be better and not makes those mistakes that are so easy to make. I don't want to be just ordinary. Man... the baggage we carry around from our childhoods. I've certainly got my share, and it comes out when I make mistakes like this. Double-Crud!

Things are looking a lot better today with regards to the horrible application that I was working with again today. This is the one where there are empty tables, interesting method names, and nothing in general telling us what's really going on. Today I was able to successfully track down the problem to an ancient data access layer by proving that this application requested the instrument data, but never got anything back from the service. This would normally be a simple matter of looking at the database access for the loading of the data from the appropriate database, but no... that would be far too easy.

I had to find the place where the query for this service was being built, and then in an entirely different directory (library) of this app, the data was being read in and processed. It's amazing that anyone actually understood what was going on here. Maybe there were code generation tools in the initial versions, but there aren't any now, and it's an amazing lump of horrible code to try and find out something as simple as what was asked for and what was returned.

In the end, I'm confident that it's a data problem in the database that's used to return the information for these calls. What exactly is wrong, I have no idea, but there's a group that does this all day long, and while they don't necessarily have any better idea about this than I do, at least they have time to invest in getting it tracked down. I need to be doing other things, and not spending more time on tracking down a data problem.