What's it all about, Alfie?

I know it's ranting... I know it's tilting at windmills, but darn it, these large ISPs that are not really doing the right thing for their customers are really yacking me off! The other day it was Comcast and their blocking ports and messing with responses to requests. It's frustrating that it seems the companies with the capital to provide large bandwidth are the same ones that are the most intent on keeping it to web and mail traffic. It's incredible.

Then today I'm trying to upload a few files from work via an AT&T DSL line to my HostMonster account and it starts to send but fails in the response. I know about this as it's happened before, and I can solve this problem by doing it while I'm at home - from Comcast. Now I don't know why AT&T's DSL is messing things up - and it honestly might not be AT&T - but it's upstream where I can't see the HostMonster boxes and they can't see me.

It's really amazing. In the old days, I'd have had a modem in my laptop and I'd dial into the ISP I needed to get to and things would just work. Today the speeds are faster - which is nicer, but the connectivity is probably a lot less. Trying to stretch their infrastructure to the limit to get the most customers on it before having to upgrade it is just the typical corporate way... and it's leading to regulation. They are going to find themselves looking down the business end of massive regulation if they don't clean up their act.

One of the things that I keep coming back to every so often is the simulation work I did during my Ph.D. Now that I had upgraded to 10.5.1 on my laptop, I wanted to see about getting the code going in x86_64 mode. I have had it running in 64-bit on PPC, but now that Xcode supports it, and the gfortran I use does as well, I was interested in seeing if it was going to run faster - or slower, in x86_64 mode.

The first thing I noticed was that the LINPACK routines that I had taken and hand-tuned to the problem were not working out well with the 64-bit compiler. I was getting SegFaults, and rather than mess with trying to fix those versions, I thought I'd use the BLAS and LAPACK that are bundled with Mac OS X in the Accelerate Framework. These are supposed to be optimized for the AltiVec (PPC) and SSE3 (Intel) so I was thinking that moving this way was a nice upgrade.

The code changes weren't major - primarily in the data storage going into the functions, so it only took me a few hours to fix all that up and clean up the code with a few #ifdefs to make it compile either with the LINPACK routines I built or with the LAPACK functions that came with the OS. What was major, were the results.

As I had hoped, the 64-bit version of LAPACK was faster than the 32-bit version. However, the surprise was the fact that my 32-bit hand-optimized routines were faster still. If I wanted, I'm guessing that I could update these guys to 64-bit by looking at the use of the data element sizes - that's got to be the cause of this as the logic is fine, and then I might very well have something that's faster still. What a shot in the arm! I had no idea that the modifications I had made were going to be that fast. Good for me.

UPDATE: I went into the code and found that it was a simple matter of how the integers were being passed from FORTRAN to C. By putting a simple typedef in the code:

    /*
     * Because we need to be able to build this for 32-bit and 64-bit
     * versions, I want to be able to typedef the integer here so that
     * the value coming in from the FORTRAN code matches what we will
     * use here. Without this, we'd have a mess on the conversions.
     */
    #if defined(__x86_64__) || defined(__ppc64__)
    typedef int f_int;
    #else
    typedef long int f_int;
    #endif

and replacing the long int with f_int (FORTRAN int), I was able to use the same code for both builds and the errors went away. Nice.

There's a reason I like working where I do - it's not the hours, that's for sure. It's that for the most part decisions are made by the appropriate people. That means that if there's a need for the entire organization to be involved, then the decision is handled at a very high level. However, if it's the decision to update from an unsupported product to a supported version of the same thing, then, again, for the most part the decision is handled at a low level as there's not really a lot that we're going to do about it. We're going to have to update, and what we update to is really the only question.

Today I've had a couple of things I'm working on go from being properly handled to being very improperly handled. A while back it was the ordering of hardware and the request to get the vendor's technical sales reps involved. True to my prediction, they knew nothing. They took more than two weeks to come back and tell us that they knew nothing. Total waste of time.

A few weeks ago, we wanted to get a nice messaging bus in - Tibco EMS. We knew it was going to be a ton of money, but there were a ton of reasons to spend it now and convert the Shop to Tibco. But since it wasn't going to solve a pressing problem, it was into the "project" category... complete with timelines, Visio diagrams, step-by-step proposals. The Works!

I read a wonderful quote the other day attributed to Mark Twain: The best time to fix a hole in the roof is when the sun is shining. The reality here is much more like: They only time to fix the hole in the roof is during a monsoon. I appreciate that this is not pocket change that we're talking about, but the problems facing us aren't likely to be solved in an afternoon, either. Serious problems demand serious people and serious tools. This is not something to slap together with perl and Visual Basic.

But that battle is lost. Today I was trying to get a new version of the modeling libraries into production. I try to keep a low profile - there's no real option. We have to do it, so the fewer people that know about this the better. The fewer people involved the better. More visibility almost always translates to slower decisions, more arguments, more grief, and in the end, since there's really no other option, simply a slower roll-out of the needed functionality.

I tried not to get flustered with this as it was a friend that made the critical mistake of involving too many people early on. His opinion is (was) that this would be 'no big deal', and could not have been more wrong. I appreciate what he was trying to do. He just wants to keep people up to date and he thinks it'll be a slam dunk. But when someone usually thinks that I have the nervous feeling that if they are wrong, they'll be wrong in a very big way.

So it got involved with four more people that have very little time to look at things and have very little idea of the real alternatives open to us. This means that they'll look at a difference of 0.0005 and say "That's too much - match it exactly!". But Dude! That's the old numbers... we match the new ones exactly. Let's not get bogged down about the difference on what was - let's look at what is.

But I know that's not how it's going to go. It's going to be a few days of people discussing it and trying this permutation on the data, and that input set, and this and that until they get frustrated and finally see that it's the old values that are off and give up. In the end, it'll be just as if they never got involved. They'll add nothing, because there's nothing to add. But it's something to argue over.

This the is most Dilbert-like part of my job. I really don't like it.

Today I've started really working with Leopard (10.5.1) and there are a few things that aren't really exactly how I'd like them to be. For the most part, I'm willing to let these things slide, but there are a few that are somewhat annoying and others are nice surprises.

Don't get me wrong... I know 10.5 is what I need to go to, and it's clear that there has been a ton of thought put into it, but there are just a few rough edges that I'm sure will get worked out in the coming months. I guess I'm just realizing that the critics of Leopard are, in part, right. It does seem a tad rushed. But given all that's happening, it's clear that this was easier to rush than the iPhone, so they put their limited resources where they needed them. It'll just take a while to get everything back to working really well.

SSH

One of the nicest things about Leopard is the integration of SSH. Specifically, the addition of keys and ssh-agent to the login process. In the previous versions of Mac OS X, I had to create a script for my .login and .tcshrc that starts ssh-agent if it's not already started, and if it has been started, reads the environment variables from the output of that command - stored in a file in /tmp/. This lead to problems if someone ran a new shell and the .login script didn't get the environment variables right, so it'd start a new ssh-agent and blow away all my old env vars. A pain.

Now, when you first ssh to a host a dialog box pops up asking you for your SSH pass-phrase, and if you want to store it in the KeyChain. I put it in, said "Yes", and now all my ssh sessions are authenticated properly without me having to worry about a thing. This is exceptionally nice. Saves a lot of hassles for me as I use ssh all day long.

Spaces

Prior to 10.5, I was using VirtueDesktops - an open source virtual workspace app for 10.4. It was not perfect, but it worked, and as long as it was working for me, I was happy. I knew with Spaces coming, this app was at an end, and that's the way of the world. But there are many things in VirtueDesktops that Spaces is missing. Some minor - but some major.

First off, Spaces only allows three ways to switch spaces. It's really just allowing you to choose one of three modifiers and a numeric key for going directly to a space. There are also similar modifiers to the arrow keys to move from space to space, but since I use BBEdit, the keys I would have used are already used in BBEdit, and so I had to turn that off in Spaces. Not horrible, but annoying that I can't use Fn keys as opposed to numbers.

Secondly, the automatic switching to the app upon selection. That's a nice feature - if you want it, but I don't, and the problem is that when I want another Safari window, I get thrown back to the space with Safari open, and that's very annoying. I have to make a new window there, then F8 to see all the spaces, and then drag the new window to the space I want it, and then switch back to that space. Non-ideal.

Thirdly, the animations for switching from space to space - I'd like to turn it off. You can turn off the Dock's magnification... and the window closing effect... so why not this? No reason, but you can't. This is only annoying, but when I'm moving from one space to another the last thing I need is eye-candy. Make it optional, guys.

X11

There have been a lot of things written about this already. It's not really all there, and it's primarily due to the fact of moving off the XFree86 codebase to the xorg codebase and the differences therein. It's nice that they made it start with launchd, but it needs to read the .Xdefaults properly, and there are still a lot of things that seem a little odd about it.

For instance, I know the app in the Dock is really /usr/X11/X11.app and to change the app icon, you need to change the permissions on the app bundle and then change the app's icon. I did the following:

  cd /usr/X11
  sudo chgrp -R admin X11.app
  sudo chmod 775 X11.app

I then went into the Finder and changed the icon like you would for any application, and then locked down X11 again with the following:

  sudo chmod 755 X11.app
  sudo chgrp -R wheel X11.app

A few times - right after the update, I had a few problems getting X apps to display on my MacBook Pro from my Sparc 20. The problems went away and it could have been the 10.5.1 update, but it's still a little troublesome.

Overall, however, X11 is far better in Leopard than Tiger. The launchd aspect was really a beautiful solution.

Time Machine

I'm so unsure of the effects of this, I've only got it running on one machine, and it's not a critical one. When I've seen it run for a while I'll look to turning it on for the other machines I have, but for now, this has too many unknowns for me to feel comfortable trusting right now.

Xcode

One of the really nice things that's in this release is Xcode, and GCC, in general. There are things that have changed, and while I know to expect them, it's really surprising how easily I found them.

In CKit, I use a #ifdef for defining MAXHOSTNAMELEN as it was defined on Solaris and Mac OS X easily enough, but Linux seemed to be hiding it somewhere. I put it in and ran with it. No big deal. But with 10.5.1 and Xcode 3.0, it was missing from the #includes I was using. So I used this occasion to dig a little further. It turns out that it's located in sys/param.h for all three OSs, and I was just missing it in the #include-chain on linux. No big deal. I take that out, put in the include for that file and we're good to go.

Since I do a lot of coding to x86_64 on opterons, I was very happy to see that architecture supported on the latest Xcode. While much of the code I write is specific to linux, I make significant efforts to make my CKit library as cross-platform as possible, and this certainly means 32-bit and 64-bit on Mac OS X as well as Linux and Solaris.

Today was a little mixed bag of coding issues. I had some old code to clean out in the server for option strike formatting that was causing me grief, and then after yesterday's problems with my Sun development box for the MareketData server, I decided to make sure that I could build it on another box - the other development Sun box that my co-worker has.

For the most part, it was building pretty nicely. I had to fix up a few things and move a few directories to a new partition to make space for them, but other than that it was humming along pretty nicely. Then I started noticing some stalls in the network traffic and did a little netstat -i to see that there were some in-bound errors. The network crew replaced the cables from the box to the switch and I'm hoping that will help, but I really think it might be a duplex issue on these auto-negotiation NICs and switches. We'll see.

There was also a mount point issue, and after I pointed out the problem they updated the YP maps so that after the nightly push, I'll be able to reboot the box and then get the right maps for the box. With this, I should be able to build the code on the other box. That will certainly give me a nice buffer... some breathing room, so that if my development box goes on the fritz again, we have some place to build the code for a production problem like we had yesterday.

I also added in some interesting little code to the MarketData server to allow all the symbol mappers for each of the data providers (each provider has it's own symbology, and these components map from our symbology to the native symbology of each provider's API) to change their mapping parameters. For the most part, these are database connections where I run stored procedures or SQL queries to get the native symbology components from our symbology components.

What this is going to allow me to do is to change the database connections on the fly through the chat interface of the server. I can also query the server and see what parameters are being using by each provider. This would have made the need for a rebuild unnecessary yesterday, and it makes switching as easy as typing in a few lines to the chat bot for the app. Not bad.

We also gave a presentation about the existing infrastructure and why we want to use Tibco EMS as opposed to the home-grown message brokers that we have. It was basically showing the other folks - including management, how much we depend on custom communication protocols and why it'd be a good idea to standardize on something like Tibco and get the non-standard stuff out of the shop. It's going to be a hard-sell, unfortunately, as it still represents a ton of money to do the switch. When it's important to them, they'll find the money. Until then, I can make what we have work.

Today was the first day back from the long weekend where we went to Philadelphia for Liza's marathon. It was an amazingly crazy day that had more than it's share of issues and problems that I had to overcome. Not a fun day - not by a long shot.

First, I had more of the 'NFS not found' errors on my Sun box. This was supposed to have been cleared up with the new networking and cables, but it wasn't. This was made infinitely worse today as I had problems with the production MarketData server and without my Sun box to develop/debug, I was in the dark. They came and replaced the network card in the box and reconfigured it to use that new NIC as opposed to the one on the motherboard. This took about an hour from the time I started 'yelling' about the problem and it's impact to production, so that's not too bad, but that it was out after having told them to replace the NIC weeks ago is a little troubling. But I hope this is it for this problem.

Next, there was the problem that I needed that box to solve. That was, the database that I get symbol mappings from was not returning data as it should have. If I went into SQSH, the data looked to be there, but when I went in from the program and the SQLAPI++ libraries, it wasn't. After a bit I gave up and pointed my code to the primary database (as opposed to the read-only replicant) and it worked perfectly. I had seen a few database issues this morning early, but thought they were all fixed up. I was wrong.

Later this afternoon, I saw that the read-only database was working again, and I'll switch production back at the end of the day. I try to be a good corporate citizen as much as possible and sing the read-only database is something they want me to do. I'll comply as long as it's working, but when it fails, it's back to the primary as the users have to come first.

After I got that fixed up, I had to fix up problems with my server and the EQSVal libraries. It turns out that with the most recent version of the libraries (v7.3.24b) you have to set things that were once defaults in previous versions. This isn't horrible, and the fix was simple - but figuring out what needed to be done was the problem. Thankfully, the EQSVal guys were helpful and once I generated them a test datafile they were able to tell me the issue and what to do.

After that, I had to make several changes to the web-based editor for my server. Nothing major, but there were some edge conditions that were showing up as records in lists that should not have been there. The perl processing and filtering was about as good as I could do in a single pass, so I had to add a second pass on the data to further clean it up. This will take a little bit more on the execution, but it'll keep the records cleaner and therefore make it less confusing to the users. Good things to do.

Finally, I got the most recent version of a driver from the internal price source folks at the Bank. I needed this because their mapping rules from Reuters to what we needed weren't capable of being exchange-specific enough to really hit the points we needed hit. With this update, my contact Todd is able to get the rules as specific as necessary and that's great. I'm hoping that these work out in test over the next few days so we can roll them out to production as soon as possible. These are tickers that the users have been wanting more smarts in, but haven't been able to deliver due to the limited rules on Todd's side of things.

That's about it. It's been a monster of a day. I'm looking forward to calming down and taking it easy for a bit.

Today I finally figured out the problem I have been slugging out for a few days. Actually, there were a few - none of them of my creation, but legacy issues that I have to deal with in order to get the server making a little forward progress.

The first was the fact that the Bank's valuation library, EQSVal, deals in dates that are ints and I had assumed that the original author of the conversion code had checked to make sure that the conversion was done properly. After all, the data looked right. In C/C++ the standard for converting dates to ints is to compute the number of days since 1/1/1970 - the epoch for all Unix systems. And that's what the server's conversion code was doing - converting the dates based on that reference date. Interestingly enough, the EQSVal library authors have decided to reference the same date as Microsoft Excel. Why? I can't possibly tell you because it makes no sense to me, but that's not the point, I suppose. The point is that they reference the first day of the last century - 1/1/1900. This made all my dates off by 70 years - some 25,567 days.

I saw this problem and realized that the easiest thing to do was to make a new method on MMDate called daysSince1900() so as not to confuse the other uses of the method toInt() that got the date with respect to the Unix epoch. This fixed the first of the two problems. I'm convinced that when I was trying the latest production version of the EQSVal code I was getting what they classified as operating system errors most likely from doing the reverse of what I was doing - subtracting 25,567 from the '1900' date and then using the Unix tools to then trying to convert it to a Unix time struct. Getting negative numbers might really be a mess, and could easily have been the issue.

So... one down, one to go.

The second problem I found was the quote date for the stock 'products' going into the model. The original code had the driver date - which is the quote date offset by the settle offset for options. This was a typo in the code and was leading to problems in the model. EQSVal checks the volatility curves (volatility, skew and kurtosis) to see that the first point in each curve is not before the quote date for the evaluation. Because the stock product was using the driver date, and the first point in each curve was only a business day away, it was advancing over the start of the curve and it was flagging an error. Most of the time this would not be an issue, but near the expirations this is going to happen as the volatility curves are calculated on these expirations.

With both of these problems solved, the values were calculating properly and no flags were being raised. Good enough. Not easy to find, but I'm glad I stuck with it before I left for the weekend.

UPDATE: it turns out that the EQSVal libraries allow for me to specify the date of 1/1/1970 which they use for offset. In general, then, the difference should not have mattered, had we set the offset right. For now, we'll leave it, but it's nice to know that there's a place we can control the offset itself.

There are those days that make you love the Job. There are those days that make you hate it. But for the most part, it's just that - a Job, and there's nothing particularly exciting about it and nothing particularly aggravating about it either. Today is one of those days.

I'd rather be doing something other than what I'm doing - which is debugging other group's libraries in my code without their source or support. It has to be done, because there isn't a real alternative, and it's not horrible because I have decent ways around most of the limitations I've found in their code, but it's still just plain old grunt work. Nothing fancy or glamorous, but the bread-n-butter of the developing job.

It's not fast because a significant change requires a restart which takes about 20 mins, but I try to make those significant, and therefore informative. It's going to get solved, it has to get solved. But it's not really something I'm going to put on my resume. It's just the job.

Today, actually yesterday evening, a trader had a problem with some of the data my app was showing. Now I don't generate this data - I just display it, but as is often the case, if it's connected with your app, you get at least partial blame for it not being right. I understand this and it's something that, to a large degree, I agree with. The problem comes in where the guys feeding me the data won't (or can't) fix the data coming to me in a timely manner. Then I have to get into two modes: Defensive Systems Development and someone else's debugger.

Today was one of those days.

After coming up with a plan to avoid the problem in the short-term, I set out on the task of making my system capable of filtering out the bad data from the upstream source. It wasn't something I was planning on doing today, but it wasn't bad and allowed me to add another level of flexibility to my flagship middle-tier. Not bad really. But in the middle of trying to get al these changes in for stopping and starting the flow of updates from my main collector/distributor, there were a flurry of emails about the upstream system - a database, and trying to find the problem there so that we stop it at the source.

I'm all for fixing things at the source, but I soon realized that they weren't seeing the problem. So I had to just into the fray and try and figure it out myself. I'll admit it was a tricky little problem in a stored procedure. Basically, a table which originally presumed to have only one day's worth of data in it was having, for a few hours, two days worth of data. This wasn't bad because when they changed to making it have multiple days of data, they changed the stored procedures to look at just one day's data in that table. Or so we thought.

In actuality, it was not looking at just one day's data because the where clause on the defining cursor was not limiting the primary table's data to just one day. This meant that for a time, two day's data was being used in the cursor. This would have been fine, if it not for another bug in the storied procedure that didn't limit an update statement to a single day. Now the bug is evident - two days were being looked at in the cursor, both days were being updated at the same time in the update statement, and one of those days was not really complete.

That meant that the incomplete day was getting used and updating the complete day and that was the bug. Once I figured that out, I told the manager of the project, and when I had convinced him, told him what he had to do to fix it. He then had one of his guys fix the stored procedure and we should be fine.

I still finished my enhancements to the filtering on the data sources because I think it's a good thing to have. But it's probably no longer necessary. We have it fixed at the source and that's going to be all we need.

I was messing with Vim's ctags support today on my Mac and then somehow wondered if either SubEthaEdit or BBEdit had support for ctags like Vim had. I dug into it and in fact BBEdit can support ctags, but unfortunately not the ctags that exist with Mac OS X 10.4 - you need the exuberant ctags on sourceforge.net. They talk extensively about it in Chapter 14 of the BBEdit manual. You need to download, build and install the ctags app:

    cd ctags-5.7
    ./configure --prefix=/usr/local
    make
    sudo make install

In order to have it not conflict with the existing ctags I did the following:

    cd /usr/local/bin
    sudo mv ctags ectags

I also changed the name of the man page to get that matching the new command:

    cd /usr/local/man/man.1
    sudo ctags.1 ectags.1

and to make sure this man page gets into the existing MANPATH - if you don't want to add this location to your MANPATH, you can do:

    cd /usr/local/share/man/man1
    sudo ln -s /usr/local/man/man1/ectags.1 .

Then I created a simple alias that allows me to call it simply with the right arguments for the file BBEdit needs:

    alias ectags `ectags  --excmd=number --tag-relative=no  --fields=+a+m+n+S -R'

So that in the Makefiles for BKit and CKit I can add the target:

    CTAGS = ectags  --excmd=number --tag-relative=no  --fields=+a+m+n+S -R
    ...
    tags:
        @ $(CTAGS) `pwd`/src

And this way I can then automatically pick up the tags in both projects. It's nice that CVS doesn't try to update the file called 'tags', and placing it at the top of the source tree allows BBEdit to find it - as well as Vim, if I'm into that as well.

In the end, this is a really nice little addition. I'm a little surprised that the BBEdit folks didn't include it as a part of the BBEdit distribution so that you would not have to download it and build it. I mean really they know that Mac OS X 10.4 doesn't come with it, and they are going to need it, so I can't quite figure out why they didn't include it. But they didn't. Easy enough to download and build.

UPDATE: I also found that there are 'Jump' and 'Jump Back' menu commands that I've got hot keys set up for. This makes it very easy to jump to a functional definition based on the ctags and then back to where I was, and then back to the definition. Very nice. Gotta love BBEdit for this.