What's it all about, Alfie?

OK, this afternoon I was testing out my market data server and I ran across a bug. It was an annoying little thing, and I deserve to have been bitten by it for trying to beat the house. And by this I mean trying to be smarter than the OS and library builders. The original method looked something like this:

/*
 * There are times that you might want to see the numeric
 * representation of the contents of this string. These methods
 * make that easy enough to do and take care of all the dirty
 * work for us.
 */
int CKString::intValue()
{
    bool    error = false;
    int     retval = 0;
 
    /*
     * We're going to loop over each character in the String until we
     * get to the end. As we're going, we'll be building up
     * the return value. If we find a character that's not a digit
     * then we'll return what we have to this point, just as atoi()
     * would.
     */
    for (int i = 0; i < mSize; i++) {
        // see if it's not a digit
        if (!isdigit(mString[i])) {
            break;
        } else {
            // accumulate the value
            retval = retval * 10 + (mString[i] - '0');
        }
    }
 
    return error ? 0 : retval;
}

the reason I chose to roll my own atoi() was that I was thinking that the system routine did too much and all I needed was to add up the value. How easy could that be?

I should have learned from Vegas - Never bet against the House!

The bug was the string '-1'. Clearly, I hadn't allowed for the '-'. When I added that into the code, and then added in the allowance for whitespace on the leading (left) side, I was looking at something that I realized that the system call was just as efficient, probably more so. When I realized how silly it was to try and beat the house, the new version of the method became:

int CKString::intValue()
{
    int     retval = 0;
    if (mSize > 0) {
        retval = (int) strtol(mString, NULL, 10);
    }
    return retval;
}

which works just fine and is probably more efficient than I could have easily written. Lesson to self: Never try to beat the House!

I was doing my first building of java code on Leopard today and realized that jikes, the fast, Java compiler is missing. I'm guessing that with Java 1.5 as the default, and jikes not supporting 1.5 or later, at least not yet, they decided that they didn't need it.

I mean it makes sense, but it also means that all my Makefiles are going to have to change. Not the worst thing in the world, but a little disappointing as I've been using jikes since it was off IBM's alphaWorks site and I've built it on a ton of machines. Just a bit nostalgic.

Today I was checking the podcasts in iTunes and noticed that all the music in my iTunes library seemed to be missing. What I found was that the location of the iTunes library was reset to be ~/Music/iTunes directory from whatever it was in the upgrade. Now with I got the first versions of Mac OS X, the iTunes library was installed in ~/Documents/iTunes and up until this upgrade, that was preserved. Well, not this time. So I decided to clean house a little and so I copied the files from ~/Documents/iTunes to ~/Music/iTunes and then tarballed the old directory up and put it on my external drive. Then I deleted it from the old location and started running it from the new location.

Seems to be working just fine. Just seems a little odd that they didn't pick up the preferences value.

Today was the first of eight days of "training" for a package that the shop has purchased at great expense and very little long-term thought. This training is required because they require that you are a certified developer in order to get technical support from them. It makes sense... why answer a bunch of silly questions from users when you can force questions through developers that have gone through your "classes" and at least have a working knowledge of the product. That's the theory, at least.

The practice is something else. An entire day spent on SQL - or a derivative of it close enough to be considered SQL, and where it differs, they should have just changed the syntax to make it look like real SQL. Imagine that they changed the inequality operator from '!=' to '^=' - just to be different. Why? No idea. The guy teaching the class had no idea either. But there it is.

There are a few changes - no inner selects, you have to use temp tables, but that's not hard. But we spent hours there doing queries that we'd never do in practice without the schema right at our fingertips. It was educational for about the first hour, and after that, it was just moe of the same with a lot of waiting while we flipped through slides.

I realize this might not be this guy's main job, but he's representing the vendor to us, the customer, and when his response to a statement from use like "That's a bug, it should be fixed" is "That's the way it is, it's in the docs" he's missing what we in the biz call the Big Picture. I'm sure he's a nice guy, but come on! When your customers are telling you this is a bug, then you at least say, "Hey, I understand, and I've tried, but they aren't budging". And then when we say "Hey, an error message to that effect would be nice", you agree and talk to the product developers. You don't giggle at us, like you have so many times before and then say that's the way it is.

So for the first time in over a decade of class work, I got a headache from this class. That's bad. I can't believe I have seven more days of this. It's got to get better, or I've got to go through it quicker.

I want to believe. I really do. I want to believe that people are basically good and honest and I want to believe that they will look out for each other - at least a bit, in a jam. I want to believe that people are honest and have honor, and work in a mode where they instill these characteristics in others.

I'm honestly shocked and hurt when this doesn't happen. More so when it doesn't happen around me with people I have known for years.

Some will say that I'm not learning from the historical evidence available to me. And I'd have to agree that they are right. If I look at the historical evidence, key members of the management team here have proven themselves to be somewhere between having selective memories and outright deception for their own ends. Yet I still want to believe that this time they are acting honorably and honestly.

Silly me.

So here's the story of the week... the shop has decided to buy this package without really technically diving into it's capabilities to see if it'll do all we need (not all we want, all we need), and so now, millions of dollars later, they need to ensure success. No surprise there... I've seen it happen time and again. Typically, this is where I come in. They come to me and say "It's got to work. Make it work." And I make it work - as best it possibly can. There are times that it hasn't been 100%, but it's always been good enough to declare success, and that's all the management wants.

So this time we're poised again to do the same thing. Then yesterday I get into a late meeting where it's now clear that the work I would be doing on the new version of the product has to be hamstrung by the fact that it's got to work with the old version that I had been told would never be deployed here. Hmmm... I see the selective memory popping up.

So now we have to deploy this old version that isn't even half of what we need but we have to do it because the guy really running the project says we have to. I can see his point of view - minimize the changes in the migration from old to new, but I disagree that the minimization of the changes should take precedence over the fulfillment of the potential of the new version. After all, we know the old version is not nearly what we can live with, why force us to live in it's limitations?

So it's now clear that I will not have the freedom I need to make sure this project succeeds. In a way, I've been given my freedom by that one simple meeting. Oh, I think the management team hasn't figured that out yet, but if not now, they'll see it soon when I say "Hey, I'm sorry, I can't do that, based on the schedule and work outlined by the project manager." Then it'll hit them that without power, responsibility is an impossibility.

As an aside, these interactions make me hold my management in contempt. If they don't want me to do a certain job, then just say so from the onset. If they do, then have the honor to do so, or tell me that things are changing and realize that they can't hold me to any sense of responsibility or a project that they themselves will not give me power to control.

Times like this I want another job. I know it happens everywhere, and when it happens where I've been, I've wanted to leave those jobs as well. I like to work. Period. Ask me to do a job, and I do it. Try to con me into some impossible situation and see if I'm dumb enough to take the bait and try to make this looming train wreck miss is not respectful nor honorable.

I've been using XML for quite a while. I can see where it's useful, and more often than not, I can see applications of it that are horrible overuse of XML for the sake of using XML. For example, I can see XML as a configuration file with less than 100 lines in it. I have also seen XML as the transport for data streams of thousands of points. The first is a decent use of the format, the latter is stupid, plain and simple.

So I find myself again today working on a project that I haven't touched for months. It's got a ton of XML configuration - literally dozens of files and thousands of lines - some files having several thousand lines of XML each. I've heard people who are big proponents of XML say "You can have code stability while configuring your app in XML". Well... that's a true statement, but you could replace 'XML' in that statement with just about anything else - including simple key/value text files. But what seems to be the trend is that the XML files actually configure the system - hook up this feed to that processor, and add this dialog box at this point, etc. I've seen this all the way to the point that XML changes alone were responsible for adding completely new functionality to the app.

At this point, the XML is really the code. You can't call the framework of interpreting the XML and following it's rules of wiring things together the core, immutable, part of the system - and the only part needing source control, test cases, and other "project-level" attributes of a codebase, when the way in which the application works is really dictated by the XML. It's like calling the JVM the code and the compiled byte-code the "config files". Nope, baby... that's a runtime and that's the code it executes.

So don't bother saying XML is wonderful... it's OK. It's not alone, and it's not even the best implementation of the self-describing markup language. But it has it's uses. Unfortunately, I've seen far more abuses than valid uses of XML. I'm beginning to think they simply don't exist.

I was just talking to a co-worker about something totally unrelated - I think it was the history of a project here at the Shop and who's done work on it over the years... when it's changed platforms, who's done what, etc. Sort of fun because one of the guys new to the group hadn't heard this, and it had come up recently as needing some work done to it. While we were talking about this, my co-worker brought up something that had happened years ago, and didn't paint me in a very flattering light.

He had asked for a feature to be added to something I had built on top of a commercial package to make using this pack a lot easier. Anyway, this one feature he wanted added wasn't natively supported in the commercial offering. I'm sure at the time I was trying to make it easy to do those things that the package did natively, and not build in a lot of functionality that wasn't part of the original concept of the package.

Well... at the time, he didn't buy this, dug into the code and saw that it could be done - not supported natively, but possible. He tells me now, he knew then that I was bs-ing (trying to be family-safe here) him and wanted to call me on it. He went to our boss, told him he knew I was bs-ing him and asked if they should push me on it. Boss said "No, let it go", and it dropped.

Later, when someone wanted to move an application to the web using this package I added this exact feature - not because anyone directly asked, but because if we were going to migrate this one app and kill it, we needed to have this functionality somewhere else. So I did it. It was more of an add-on than I'd done in the past, and meant that the use of this feature made the code much less efficient, but it did work.

And for this, he's been carrying around this on his chest for years. He got the chance today to get it off his chest and put it squarely on mine. So, was I bs-ing?

Looking back, I know I didn't think I was - then. Now I'm not so sure what my motives were. Maybe it was as I remember it - an attempt to be a thin shell on the package to make it easier to use the native features. Maybe I just didn't want to do it so I didn't look for the way to do it at the initial time.

It makes me think about what I'm doing today. I know I've turned down requests to put certain things in my code because I felt they didn't belong there. I've suggested where they might go, and encouraged others to put them there. I've even offered to let them fork my work and add what they wanted to the fork they made. But I will stand firm when I think the idea is not right. That's just good design and appreciation that not everything belongs in a String class, for instance. You create horrible maintenance problems if you keep throwing things on a cleanly designed set of classes.

Still... it causes me to pause and think. Maybe I was too harsh. Maybe I need to give it more time. I'd like to think I'm a better person because of this, but it still feels like I just got called to the principal's office.

Nothing is perfect. Period.

Sometimes we're lucky, and some times we make our own luck. Sometimes.

I was just sitting here thinking how much I really enjoy the position I have today. I really do enjoy this industry, not that I'd have ever thought I would - from the outside looking in, but once in (thanks to a good friend) I have found it to be one of really interesting people, very challenging problems and more than enough variation to keep me interested for more than a decade now. Pretty lucky, I'd say.

It's funny to me to see how the most captivating parts of this job are the ones that take me back to the work I did in grad school - numbers. Data. Lots of it. The market data is very interesting to me. Getting it from the provider, making it faster, better, easier to use - that's a blast. Getting it into a simulation/analytic engine is also a ton of fun. Seeing all the numbers roll around from the market data all the way through the various systems to show up finally at the client is a lot of fun. Number plumber - that's me. Kick in the pants.

It's important to realize that after all the work I've done that has nothing to do with my degree, it's neat to realize that I really have come back to my 'roots'. I guess you gravitate to those things you really love doing. Sweet.

Today I spent a lot of time on migrating off the data source that we're trying to move off of, and on to the one that's got historical pricing data that we can use without the associated rules and regulations of the original source. This meant doing a lot in the market data server to make this other data source look as close to the original as possible, while still taking advantage of the few things it was doing better than the original.

One of the more interesting things is the server's respecting the field names that are used by the client in it's requests. If a user is using the Perl interface, for instance, and they ask for 'close' or 'Close', they don't want to get data back that has the field name of 'CLOSE'. That makes it hard for them to match up what they sent with what they received.

True, we could make it so that they were required to ask for 'CLOSE', and then all would be OK, but that's a little too restrictive for me. So I had to go into the server and fix up the way this data provider was handling the field names. Then there were the changes that needed to be made to the cache to hold the returned data - again, 'Close', 'close', and 'CLOSE' all should cache to the same values, so we had to put in case-independence in the code there as well.

In the end, I got all the pieces working and was able to move two more things off the old provider and onto Fusion. It's now up to all the other users of the old data provider to change their code and migrate from the old to the new. It's not going to make a lot of people happy, but then again, there's no way they were going to pay $1 million to get this data. No way at all.

In the recent weeks one of our data vendors has contacted us about our use of their API. Specifically, they created this API for themselves, as their product runs on it, but they wanted to make it possible for users of their product to also get at this data in a programmatic manner. It was a nice thought. We looked at this and decided that it would be even nicer if the general users of this data didn't have to individually write to the API. Maybe even add a cache to the data received so that for each day, a single piece of data would only be fetched from the provider once - then after that, subsequent requests would hit the local cache. It's an excellent idea, a wonderful product, and now they are interested in charging us more than $1,000,000/year because of the data we get from this.

The reason for this is that this vendor decided to un-bundle the API from the product they sell that uses this API. They decided this because, of course, people were using it and they saw it as an additional revenue stream for their company. I can't blame them, it's capitalism as it's best - make something someone wants and then charge for it... if they pay, then you have a winner, if not, then you are out the work to make the product. So they came to us and gave us a pricing model for the data we're getting and it averages out to a little over a million dollars a year.

Given that the bundled product we got originally is about $2,700/month - this represents a large increase, and it's likely that the business will simply not see that this data is worth seven-figures a year. Again, that's their job - cost/benefit analysis... it's a necessary part of all businesses.

The problem comes in that it's not this easy - the vendor isn't just offering us an upgrade path, they are also (at the same time), holding a gun to our heads. If any of the trading business requires this data to function (as some do), then turning it off is simply not an option. Migration is the key, but migration to what? We'll have to look into the different competitors of data and look to see what each has and the cost, etc. Then work that into the systems one by one and then we'll be ready to turn off the high-priced original vendor. It's more than an afternoon of work.

The other side of the problem is that the management here has known about this issue for years. Yes, years. Each time they talk down the vendor from the ledge, and they agree to let us continue to use the data under the existing contract. When we start to get too much data from them, we find alternatives and migrate a few applications from that one source to another under controlled conditions and hitting the easiest, biggest offenders first. Makes a lot of sense. Usage of the data drops, they are happy, and life goes on.

But not now. No, they are convinced that the vendor is out to get the full price and lock down all the data. The problem with the lock-down stance is that there's no way to really do it unless we shut down all usage of the data. Imagine, I use the data in a spreadsheet. I save that spreadsheet on a network drive that others look at. That's redistribution of the data and they won't allow that. The only people that can see the data are the people they license to see the data - no exceptions. And if there's a possibility that someone else might see the data, then that has to be shut down. Their latest position is very restrictive, and to follow it to the letter will mean you really can't use the data for anything "saved" in any way, shape or form.

Clearly, we need a new source for data, but management is taking the attitude that we need to "spread out" the usage and then pay for what we really use - sticking to the letter of the license. But there won't be any data that we can use to the letter of the license as it's a server, and therefore might serve up data to something that possibly makes it available to someone not intended to see it.

It's dumb, but I'm not going to fight it. If they want to shut it down, fine. They aren't going to - not without a replacement, and when that replacement arrives we'll simply slot it into the server alongside the old, restrictive one. It's going to be a mess, but they seem to have given up on talking to the vendor on this. Maybe it's not possible, but I can't imagine a vendor that would take nothing over something - which is what they'll get if we shut this down. But then again, they could be trying to shut it down on their end, and this is their way of doing it. Vendors have been known to do dumber things.