What's it all about, Alfie?

I know I shouldn't be surprised given the recent history I have had with software Vendors recently, but today was once again an eye-opener. I have known that an upgrade of a certain vendor's API was coming for many months. It took a lot of negotiations, explanations of the usage patterns, etc. to get to the point where things are all signed and executed so that we can actually get the software. Today was the day we could finally start the process of actually getting and using the new API.

It started out with the same problems I've come to expect from this vendor - the software was for Windows, Solaris, and Linux - yet in order to download it, you had to be running a Windows client app. Why? Why not just have a password-enabled web site where you can get the files? It seems that when you're charging what these cats are charging, you can invest the modest sum to get someone on your staff to figure this out. Nope. It was a sign of things to come.

After getting the packages I asked the 'installation specialist' what I needed to know during the install. What questions would be asked, etc. so that I could have all the proper data available to me before the install started so that I wouldn't have to start over or back up during the process. Also, I like to have all this documented so that when I do it on the next box, I'm ready to go and the delays will be minimal.

'No problems, it's self-explanatory' he says.

Wrong. Several times I had to email him to get the specifics of what the question is asking for - things like the version of the API to install? What's the difference in the versions? Why choose one over the other? Things like this are essential in an install app, and there's no documentation - nothing, to help.

I get him on the phone and work my way through the install questions. Then I'm passed off to an 'installation technician' that is going to assign a login and certificate for the installed software. I'm asked if I have my key fob - What key fob?

Clearly, they have forgotten to tell me I needed this, what role it plays in the software, and so on. It's a ridiculous situation. A simple text file would have explained all the steps and why each was necessary. I'm not sure if I'm the first that's asked these questions, but I'm betting not. Why I had to get as many people on the phone and have them explain these things to me I do not know. In the end, I'm not really sure that I have the real explanation, but it appears that the software is working, so I'm willing to take a few things on faith. But how easily this all would have been avoided with a simple installation guide.

Once I got that working, the production box running the old version of the software stopped working! I was not surprised. They had made so many mistakes like this in the past that it wasn't an amazing surprise that by turning on the new version on the development box, they disables the old version on the production box. I had to call back and yell that they had taken my production box offline and they needed to get it back up ASAP. 45 mins later, it was back up, and I was able to tell my users things were back to normal.

The software upgrade itself is a welcome upgrade. There are fewer libraries to link and they are all shared libs as opposed to static libs - which is nice. So it's clear that someone was thinking when they did this, but it's also clear they didn't give a lot of thought to the converting users. There were no code changes - just a recompile and re-link and we're good to go. But what an ordeal. It's amazing what some vendors call support these days.

Today I finally got the WordPress installation for my daily work journal up and going the way I wanted it. I then started back-filling the entries from my old manual HTML journal to the WordPress blog. It's slow work, but in the end, I'll have the complete, searchable, history in WordPress for management to see.

It's not that the old system was bad, it's just that this system can be used by all the developers to make a consistent journalling system for all the daily work which makes the generation of management reports far simpler. No one has to guess or search their memories to remember what they were doing when. Just throw in a paragraph on every topic worked on in a day and you're good to go. Simple.

Simple for me, anyway. In a few weeks I'll have the complete history moved and I can take that old journal offline. It'll be nice.

It came up amongst my fellow developers that having a simple blogging install on one of the machines would make it easier for them to maintain a list of what they do each day, and therefore make the creation of monthly reports a lot simpler. Seems reasonable, as I keep a simple HTML journal each day, and converting it to WordPress would be very nice. So I set out to do just that.

The first thing was identifying a machine. Since the requirements for WordPress are PHP 4.3 and MySQL 4.0, it would have been nice to have found an RHEL5 box that could be used, as RHEL5 has those in the standard install RPMs. But that was not to be. I did, however, have a good RHEL3 box that I could use. Unfortunately, the MySQL RPMs with RHEL3 were at 3.2.x - no good. So I had to do some searching.

I ended up finding what I needed at the MySQL web site and downloaded:

    MySQL-client-standard-4.1.22-0.rhel3.i386.rpm
    MySQL-devel-standard-4.1.22-0.rhel3.i386.rpm
    MySQL-server-standard-4.1.22-0.rhel3.i386.rpm
    MySQL-shared-compat-4.1.22-0.rhel3.i386.rpm

I also had to get the PHP RPMs from the CentOS site:

    php-4.3.2-40.ent.i386.rpm
    php-mysql-4.3.2-40.ent.i386.rpm
    php-pgsql-4.3.2-40.ent.i386.rpm

and then I was ready to go. Or so I thought.

The default install for MySQL places the data files in /var/lib/mysql and that worked great. The problem was that I wanted to have them in a different location. Ideally, all I'd need to do was simply move them and put in a symlink back to the /var/lib/mysql directory. But it was not that simple due to where I wanted to put the files and the rules governing placement there.

Basically, I was unable to set the user/group for the files in the location I wanted to put them based on some rules in-place by the Unix engineering team. Understandable, I just needed to find another way. The simplest way to make it work was to have the MySQL server run as another user - in this case, gomer. The trick was that it wasn't that simple to figure out all the places it needed to be changed. In the end, I was able to get it by using the following file as /etc/my.cnf - the default configuration file for MySQL:

    #
    # Point the MySQL server to this location for the databases
    #
    [mysqld]
    user=gomer
    basedir=/home/data/rooster/
    datadir=/home/data/rooster/mysql
    socket=/home/data/rooster/mysql/mysql.sock

    [mysql.server]
    user=gomer
    basedir=/home/data/rooster/

    [client]
    user=gomer
    socket=/home/data/rooster/mysql/mysql.sock

where the location of the files is /home/data/<machine>, and in this case the machine name is rooster. I chose this to make it easy to distinguish the data files for one instance from another - put the machine name in the path and you're pretty sure to not have conflicts.

With this, I could start and run the MySQL server without error. Whew! That was an important step to take. Now we need to get PHP talking to MySQL. In order to do this we need to edit /etc/php.ini and set the MySQL socket reference to:

    ; Default socket name for local MySQL connects. If empty, uses the
    ; built-in MySQL defaults.
    mysql.default_socket = /home/data/rooster/mysql/mysql.sock

and then restart the web server with:

    /etc/init.d/httpd graceful

The next thing to do is to create the user and database for WordPress to hold all it's information. Interestingly, what you need to do is to make the database first, and then assign the rights to the user (thus creating a new user in the process). To do this, we first need to log into MySQL as the root MySQL user and create the database:

    mysql -u root -p

and enter the root MySQL password. Then create the database:

    create database wp_drbob;

I've chosen the convention wp_login so that the databases are easy to tell apart. Then you need to assign right to that database to a new user:

    grant all on wp_drbob.* to drbob@localhost identified by "coder";

where the last argument is the new password for the local account. At this point you can verify the user's database is set up properly by logging out of mysql with a quit and then log in as the new user:

    mysql -u drbob -p

...and enter the new password (coder), and then have a look at all the available databases:

    mysql> show databases;
    +------------+
    | Database   |
    +------------+
    | test       |
    | wp_drbob   |
    +------------+
    2 rows in set (0.01 sec)

One little wrinkle with the PHP 4.3.2 using the MySQL 3.23.58 Client API is that the passwords in MySQL 4.1.22 are stored in a slightly different way. This means that you need to do the following in order to get the password into the old format for PHP. Log in under the root MySQL account:

    mysql -u root -p

give it the root MySQL password, and then issue to following:

    mysql> UPDATE mysql.user
        -> SET password=OLD_PASSWORD('coder')
        -> WHERE user='drbob'
        -> AND host='localhost';
    Query OK, 1 row affected (0.00 sec)

the flush this out to the system:

    mysql> flush privileges;
    Query OK, 0 rows affected (0.00 sec)

At this point we have a database and we're ready to install WordPress and configure it.

Assuming that the WordPress distribution is in a tgz file called wordpress-2.5.1.tar.gz then as the user gomer go to the location where the blogs are going to be stored and unpack the distribution. It'll be named wordpress - change the name to the user's login (our convention here again). This looks like:

    $ cd /home/data/rooster/blogs
    $ sudo su gomer
    $ tar zxvf wordpress-2.5.1.tar.gz
    $ mv wordpress drbob

at this point, you need to go into that directory, rename the file wp-config-sample.php to wp-config.php and then edit it and put in the correct values for the database connection:

    // ** MySQL settings ** //
    define('DB_NAME', 'wp_drbob');
    define('DB_USER', 'drbob');
    define('DB_PASSWORD', 'coder');
    define('DB_HOST', 'localhost');

You can then install WordPress by going to the web page: http://server/blogs/drbob/wp-admin/install.php (assuming that you have the directory symlinked into the var/www/html/ directory of the web server). This will pull up a simple page where you can set the name of the blog and the email address of the administrator. They will receive an email with the login information which they can then use and customize as they see fit.

It's certainly no easy, and a lot of the little details took me hours to figure out. But in the end, I have something that's working and it's not too terribly hard to set up another. That was the goal, and it's achieved. Good enough.

I was having lunch with a good friend of mine today and we were swapping stories of things that have been happening recently - both at home and at work, just like we always do. My friend tells me a story that I must say had me in tears I was laughing so hard. Not that the situation was itself very funny, it wasn't. It was that his delivery was wonderful.

It goes like this: The time you don't want to realize you car has no breaks is when you're flying down the highway. By then, if you have no brakes, you're in a world of trouble. Best to check the breaks before you hit 65.

That's not what happened, it's just the lack of planning management did on this project he's involved in. They are in the final stages of testing and they haven't even really stopped to ask if they have the support system in place to maintain this additional system. So much energy has been put into finishing this, yet none has really been put into seeing if it can be supported long-term. Nothing like getting there and then realizing that you can't really use it because there's no one there to hold it up.

No brakes in the car.

While everyone can claim that they were heads-down getting to the finish line, my friend recalls several conversations (quite humorously) he's had with his boss about this, and other long-term maintenance points. All the while being told that things will work out, and we'll cross that bridge when we get to it, etc.

It's sad, but funny in a way. I think we help each other by letting off some steam, and I know he gave me a wonderful laugh with this tale of woe. I just hope it does work out for him, he's too nice a guy to get really hurt by such foolishness.

I was talking to a friend today and he mentioned that these first four months of this year had been the worst he's experienced in this 3 years at the job he's at now. Everything is a give and take... good times and bad. So it's tough to know the difference between being tough and committed and over staying your usefulness. Are you being dedicated, or simply ignorant of the changing landscape? It's not an easy decision for anyone to make. We're almost always too close to the 'problem' (the job) to be able to make a good, impartial decision.

Dorey's mantra - Just keep swimming is a nice thought to keep in mind. But if you're fighting a losing battle, when does the Discretion is the better part of valor come into play? It's tough. Listening to my friend, I can see what he's saying... lots of changes, excessive management oversight (micromanagement), inadequate staff for the expected deliverables, it's nothing new, but all told, it can make a decent job downright unbearable.

But is it temporary? I can remember a lot of times in grad school where I was convinced that this was the last minute I could possibly do this. I was about to scream. But I didn't. Somehow, the spirit of Dorey saved me - Just keep swimming. It's probably not the smartest thing to do in every occasion, but it might just be the wisest in most.

I made it through grad school... maybe if my friend sticks it out, he can make it through this, too, and be better for it on the other side. Maybe it won't last much longer. Good questions, all, and not a soul on this earth can possibly answer any of them. And in the absence of any real, solid, answers, we each have to decide for ourselves.

As for me, I think I'm going with Dorey, I just need to keep swimming until a change is clear and obvious to me. Maybe it won't last four months, like my friend - maybe it will. No way to know.

I've spent most of today debugging a problem I saw in a vendor's API to a messaging system. It's not the best vendor I've ever worked with, in fact, I don't think they are even in the top 80%, but they are the vendor I have to work with at the time, and I've had to try and make the best of it. So here's the problem we ran into today.

We have a price injecting system. It gets price data over a custom socket interface (I wrote) from a price feed server and the reformats it into the format needed by the vendor's product and sends it on it's way. It's a simple transformation system. Nothing fancy. But the vendor's API is socket-based as well, and as we were to learn, not done nearly as well as mine.

When the transformer/feeder app was running in Chicago, and the price source was in Chicago and the database for the vendor's product was in New York, everything worked fine. When the feeder was in London and everything else was the same, my feeder missed the messages coming from the vendor's messaging system when I was injecting prices.

Inject prices - miss messages. Stop injecting prices - get messages. Make another test app subscribed for the same messages and it always gets them - because it injects no prices. This was getting crazy. So on a whim I decided to make use of a price source in London for the London feeder - Bingo! Now it worked. It appears that the vendor's API can't handle delays in the socket delivery from another completely different source. Yup. It's got nothing to do with the use of the vendor's API - it's the activity on a completely different socket that's effecting the vendor's API.

Note that in all this, my code is working fine. It's the vendor's that's stopped working properly. Nowhere in the documentation do they say that excessive waiting on socket communications will invalidate the delivery of messages - why should they? They probably never tested it, as they probably never had reason to. But when you charge $20 mil for something, you really ought to take a more pro-active view on things. For example, don't use a home-grown messaging buss when there are so many commercial ones that you can include in your $20 mil cost and not effect the bottom-line much.

In the end, I think I'm stable now, but there's really no way to know. They aren't going to fix this, I didn't expect them to. They took almost 2 months to fix the last bug we pointed out and that was a simple recompile with the right data type for a 64-bit version of the API. This would require real changes in how they do things, and change is not a word I'd use with this vendor.

I just have to say that I really hate the fact that it's expected that we figure this out. I'm not getting any part of that $20 mil, and yet I've saved their bacon by figuring out how to make it work in our environment. Crummy vendor.

I was talking to a friend today and he was fuming over a decision that one of the other guys he works with made. Of course, this other person really isn't in a position to make these kinds of unilateral changes, and then have other systems change as well, without at least some discussion with the lead developer (my friend), but that's what he did. It's the Better to ask forgiveness than permission style of work.

My friend tells the story that this guy needed to add a field to a RPC call, and while it was meant to be an integer, he decided to make it a string because he didn't want to get into parsing problems. The "fear of the integer" so consumed this guy that he told others down-stream of the system he was working on to use the string and then parse it as necessary. There were problems because this is clearly a dumb solution to the problem, and they had to change it to what it should have been all along - an integer. But the guy didn't want to change the RPC interface - 'leave it a string' he says. My friend was fuming.

Since this isn't happening to me, I can giggle about this to no end. There's a guy that shouldn't be developing because he's not really a developer, and doesn't really understand why you'd want to make the interface match the datatype - he's one of those guys that would make all the data in a database table two varchars() - one for the name, and the other for the value, and put everything in there. It's almost comical, if you didn't have to deal with the fallout.

So my friend has to spend the next several days unwinding all these changes and getting the right data type in there - in multiple systems, all the while cursing under his breath. Yeah... it's funny if you don't have to be involved in it.

I've been working on fixing up a script that takes a Bloomberg field definition file and generates a bunch of SQL statements to populate a database with this field definition data and it's right on the edge of really needing to be an application. It's all in bash now, but it really ought to be in perl or maybe even Java or C++. It's thousands of lines, so an application that can detect the existing data easily and only update the relevant records is going to get a lot way towards making this better. Perl could do it, but this bash scripting is just awfully limited.

Oh, I'm going to finish it, because I'm nearly done, and there's nothing in the requirements of the task that can't be done with bash and the other unix tools, it's just that had I originally known the issues with this updated file from Bloomberg, I'd have probably opted for a Perl script from the get-go.

There are spurious backslashes in the file, so I have to sed them out and make a temporary file of that. Then there are the problems with the intended primary key - this time around the file has duplicates on the primary key because it's really not meant to be unique from Bloomberg's point of view, I was just using it as a primary key because the first version of the file was unique. Silly me.

So something that should have taken five minutes is now in it's second hour as I find each of these issues in the 9000+ line file. The box it's running on is not slow, it's just there's a lot of stuff to do, and I'm not being really efficient because each time I'm blowing away all the data and regenerating it. Again, I was thinking this was 'easy'. Silly me.

The lesson in all this is that no one is right all the time, and even if you're right today, tomorrow will bring facts and circumstances totally unknown today and they will make the decision wrong. We have to be flexible and willing to see what's right and wrong and fix it - even if it means re-writing the entire process.

Today I have spent most of the day working on a problem we ran into this morning about the FX conversion of the dividend curves in my server. I had to dig into this problem for quite a while to figure out what the problem was. At first, it seemed pretty clear - the FX rate for USD/CNY was 1.0 and it should have been a tenth that. The problem was Why?

The first thing I did was to be sure that the FX rate was now not 1.0. It wasn't, but this is where the complexity comes into play - I forgot to realize that the server is a multi-machine, multi-process entity, and the FX rate I was looking at was, indeed, not 1.0, but that's not the FX rate that was being used to convert the dividend curves. The FX rate used for that was still 1.0.

After looking at this for about 10 mins., I realized what the problem was, and realized that by restarting one of the components I could have it reload it's FX rates, and this guy was the one doing the dividend curve FX mapping. In the end, it was an easy fix, but it made me realize that I needed to have a better way to have these dividend curves mapped, and so that's what I set out to do.

Interestingly enough, one of the strengths of the server is that the components are very loosely coupled. This means that they can be independently restarted and the 'whole' will not suffer. Things are re-tried, re-sent, and life goes on. Very resilient. Problem is, this means that you need to have an exceptionally good communications system to make sure that what you want done to one component will be done to all effected components. Case in point, telling the system to reload an individual FX rate.

Clearly, restarting the components is an option, but that's not very user-friendly. What I was looking for was a way to tell the components when something was changed and tell them of that change. Problem was, this would represent a significant addition to the protocol that was already in place between some of the components. Not something to do lightly. Especially, if there's an easier way to accomplish the same thing.

So I kept digging. It turned out that the only reason this one component had the FX rates was for this dividend curve mapping, and if the data coming back from the database didn't need to be mapped, then this would no longer be an issue. Idea: Make the FX conversions in the database calls. Problem: can't slow things down, so I can't add a lot of processing and I have to be careful about the FX rates I use.

Turns out, the first worry wasn't too bad. I simply looked at the original data as a 'rough cut' of the data. If any FX adjustments needed to be done, I did them en masse. This meant that most things didn't experience any slowdown, and those that did, most times the curve would be converted all at once and not a point at a time. Nice.

The last problem was a little more difficult. I was able to work towards it, and I only had to make two assumptions - that the 'latest data' for both parts of the FX conversion had to occur on the same date (good idea anyway), and that it's always the 'posted source' so that we had the best marks in use. I thought these two facts should be true, but I wanted to run it by someone who's had a few more years at the data-side of things than I. After I explained this to him, he was convinced that they were valid assumptions/rules and I then knew I could complete the FX conversion within the database.

What's the point? Well... complex systems are complex. It's in the name. Even when I've worked with this guy for years, there are parts and interactions that you might not realize and it plays tricks on what you think should happen. Then you try it and you get a different result and start to think about it, and then in a flash it comes to you that you were wrong and it was right.

Kind of like those simulations where you don't program in certain behavior, but the higher-order behavior is a direct result of the low-level rules, and so the complex behavior is displayed on your simulation. Wild stuff. Cool, but when you're trying to make a simple change and the complexity of the system is staring you right in the face, it's giggling at you. You are the one that needs to adapt.

Today I had a disagreement with my manager - he said we never agreed to a certain set of pricing rules in the risk server, and I said we did - specifically at the request of users. These types of things happen all the time for a ton of different reasons, and I'm sure he remembers it as he does because he thinks it was implied that the other things would also be involved. Specifically, when you filter out events based on the data in those events you have to be careful not to filter out too much or too little. Today we were filtering out too little, and his opinion was that he never would have agreed to the rules in place that allowed that data to get into the system.

I begged to differ. I recounted for him the meetings with the users that started the revision of the rules, and what subsequent meetings discussed, and the modifications to the rules as those series of meetings continued - until we arrived at the rules that we now use.

However, it never pays to remind someone that they mis-remembered, so I added in yet another facet to the rules to check for the data that slipped in today. It took me about two hours to get it in all the systems that it was needed, but in the end, it's a better set of rules, and will give the users a higher-quality price feed - which is something that will please everyone.

Had my life been a documentary, I'd have been able to rewind to the date in question, play for him the relevant clip(s), and there would have been no reason for a conflict because there would have been no reason to disagree. Facts make arguments moot. And yet when you start to record things - like meetings, it generally makes people feel very uncomfortable because they instantly know that it's a possibility that they will be shown their poor memories at a later time. This tends to really put a damper on the process.

So while it'd be great, it's not likely to happen. Too bad.