What's it all about, Alfie?

Today I've started really working with Leopard (10.5.1) and there are a few things that aren't really exactly how I'd like them to be. For the most part, I'm willing to let these things slide, but there are a few that are somewhat annoying and others are nice surprises.

Don't get me wrong... I know 10.5 is what I need to go to, and it's clear that there has been a ton of thought put into it, but there are just a few rough edges that I'm sure will get worked out in the coming months. I guess I'm just realizing that the critics of Leopard are, in part, right. It does seem a tad rushed. But given all that's happening, it's clear that this was easier to rush than the iPhone, so they put their limited resources where they needed them. It'll just take a while to get everything back to working really well.

SSH

One of the nicest things about Leopard is the integration of SSH. Specifically, the addition of keys and ssh-agent to the login process. In the previous versions of Mac OS X, I had to create a script for my .login and .tcshrc that starts ssh-agent if it's not already started, and if it has been started, reads the environment variables from the output of that command - stored in a file in /tmp/. This lead to problems if someone ran a new shell and the .login script didn't get the environment variables right, so it'd start a new ssh-agent and blow away all my old env vars. A pain.

Now, when you first ssh to a host a dialog box pops up asking you for your SSH pass-phrase, and if you want to store it in the KeyChain. I put it in, said "Yes", and now all my ssh sessions are authenticated properly without me having to worry about a thing. This is exceptionally nice. Saves a lot of hassles for me as I use ssh all day long.

Spaces

Prior to 10.5, I was using VirtueDesktops - an open source virtual workspace app for 10.4. It was not perfect, but it worked, and as long as it was working for me, I was happy. I knew with Spaces coming, this app was at an end, and that's the way of the world. But there are many things in VirtueDesktops that Spaces is missing. Some minor - but some major.

First off, Spaces only allows three ways to switch spaces. It's really just allowing you to choose one of three modifiers and a numeric key for going directly to a space. There are also similar modifiers to the arrow keys to move from space to space, but since I use BBEdit, the keys I would have used are already used in BBEdit, and so I had to turn that off in Spaces. Not horrible, but annoying that I can't use Fn keys as opposed to numbers.

Secondly, the automatic switching to the app upon selection. That's a nice feature - if you want it, but I don't, and the problem is that when I want another Safari window, I get thrown back to the space with Safari open, and that's very annoying. I have to make a new window there, then F8 to see all the spaces, and then drag the new window to the space I want it, and then switch back to that space. Non-ideal.

Thirdly, the animations for switching from space to space - I'd like to turn it off. You can turn off the Dock's magnification... and the window closing effect... so why not this? No reason, but you can't. This is only annoying, but when I'm moving from one space to another the last thing I need is eye-candy. Make it optional, guys.

X11

There have been a lot of things written about this already. It's not really all there, and it's primarily due to the fact of moving off the XFree86 codebase to the xorg codebase and the differences therein. It's nice that they made it start with launchd, but it needs to read the .Xdefaults properly, and there are still a lot of things that seem a little odd about it.

For instance, I know the app in the Dock is really /usr/X11/X11.app and to change the app icon, you need to change the permissions on the app bundle and then change the app's icon. I did the following:

  cd /usr/X11
  sudo chgrp -R admin X11.app
  sudo chmod 775 X11.app

I then went into the Finder and changed the icon like you would for any application, and then locked down X11 again with the following:

  sudo chmod 755 X11.app
  sudo chgrp -R wheel X11.app

A few times - right after the update, I had a few problems getting X apps to display on my MacBook Pro from my Sparc 20. The problems went away and it could have been the 10.5.1 update, but it's still a little troublesome.

Overall, however, X11 is far better in Leopard than Tiger. The launchd aspect was really a beautiful solution.

Time Machine

I'm so unsure of the effects of this, I've only got it running on one machine, and it's not a critical one. When I've seen it run for a while I'll look to turning it on for the other machines I have, but for now, this has too many unknowns for me to feel comfortable trusting right now.

Xcode

One of the really nice things that's in this release is Xcode, and GCC, in general. There are things that have changed, and while I know to expect them, it's really surprising how easily I found them.

In CKit, I use a #ifdef for defining MAXHOSTNAMELEN as it was defined on Solaris and Mac OS X easily enough, but Linux seemed to be hiding it somewhere. I put it in and ran with it. No big deal. But with 10.5.1 and Xcode 3.0, it was missing from the #includes I was using. So I used this occasion to dig a little further. It turns out that it's located in sys/param.h for all three OSs, and I was just missing it in the #include-chain on linux. No big deal. I take that out, put in the include for that file and we're good to go.

Since I do a lot of coding to x86_64 on opterons, I was very happy to see that architecture supported on the latest Xcode. While much of the code I write is specific to linux, I make significant efforts to make my CKit library as cross-platform as possible, and this certainly means 32-bit and 64-bit on Mac OS X as well as Linux and Solaris.

This weekend I spent the time to update my machines to Mac OS X 10.5.1 - Leopard. The updates went well - both PPC and Intel, and while there were a few issues, there wasn't anything that seemed to be systemic about the problems. After the install, I immediately updated to 10.5.1 and that went well. I had to update a few applications after the long install, but those went smoothly as well.

Spaces is nice, it's not the hot-key combination that I'd like, but I can live with it. It's only going to take a little time to get used to the different key combinations. The UI is nice. It's different from Tiger, but there's nothing really horribly wrong with it, either.

There is going to be a time when some applications are not performing 100%, but that happened with Tiger (10.4) and it'll only take a little while to get them sorted out. All-in-all, it was a lot of work for five machines, but it was worth it.

UPDATE: I needed to reconfigure ssh in launchd on sherman (my G5 iMac) to make sure that it answers to port 2224 as well as 22. This is because I'm using that port to redirect it through my router.

Today was a little mixed bag of coding issues. I had some old code to clean out in the server for option strike formatting that was causing me grief, and then after yesterday's problems with my Sun development box for the MareketData server, I decided to make sure that I could build it on another box - the other development Sun box that my co-worker has.

For the most part, it was building pretty nicely. I had to fix up a few things and move a few directories to a new partition to make space for them, but other than that it was humming along pretty nicely. Then I started noticing some stalls in the network traffic and did a little netstat -i to see that there were some in-bound errors. The network crew replaced the cables from the box to the switch and I'm hoping that will help, but I really think it might be a duplex issue on these auto-negotiation NICs and switches. We'll see.

There was also a mount point issue, and after I pointed out the problem they updated the YP maps so that after the nightly push, I'll be able to reboot the box and then get the right maps for the box. With this, I should be able to build the code on the other box. That will certainly give me a nice buffer... some breathing room, so that if my development box goes on the fritz again, we have some place to build the code for a production problem like we had yesterday.

I also added in some interesting little code to the MarketData server to allow all the symbol mappers for each of the data providers (each provider has it's own symbology, and these components map from our symbology to the native symbology of each provider's API) to change their mapping parameters. For the most part, these are database connections where I run stored procedures or SQL queries to get the native symbology components from our symbology components.

What this is going to allow me to do is to change the database connections on the fly through the chat interface of the server. I can also query the server and see what parameters are being using by each provider. This would have made the need for a rebuild unnecessary yesterday, and it makes switching as easy as typing in a few lines to the chat bot for the app. Not bad.

We also gave a presentation about the existing infrastructure and why we want to use Tibco EMS as opposed to the home-grown message brokers that we have. It was basically showing the other folks - including management, how much we depend on custom communication protocols and why it'd be a good idea to standardize on something like Tibco and get the non-standard stuff out of the shop. It's going to be a hard-sell, unfortunately, as it still represents a ton of money to do the switch. When it's important to them, they'll find the money. Until then, I can make what we have work.

Yesterday morning I was on the way to the train station and my car's CHECK ENGINE light came on. Right in the middle of a turn. So since I don't believe in ignoring problems, I took it to the local shop that had worked on it less than 10 days ago to fix the same CHECK ENGINE light issue. At that time, there were hard errors from the computer saying misfiring of #6 cylinder, so they did a complete tune-up and everything went away. At the time, I was happy to get it cleared up.

Then yesterday morning happened. I was not happy.

Thankfully, they checked it out quickly and today they said they could not find anything from the computer and the CHECK ENGINE light had gone out. So it's crying wolf, and messing with my head.

I have no desire to drive my car if there's something really wrong with the engine as I depend on it far too much to have it say "Hey, buddy! I've got a problem here!" and ignore it. But at the same time, it's a major hassle to do without my car for a day each time it decides to throw up a CHECK ENGINE light with no real underlying cause.

Hopefully, they'll be able to track it down. Otherwise, it may be a continuing problem for me. I'd hate to have to go back to the dealership, but if necessary, that's what I'll do.

UPDATE: they looked and test drove it and nothing ever came up. This isn't good, but it's better than it blowing up on me on the way home. Alive is better than dead.

Today was the first day back from the long weekend where we went to Philadelphia for Liza's marathon. It was an amazingly crazy day that had more than it's share of issues and problems that I had to overcome. Not a fun day - not by a long shot.

First, I had more of the 'NFS not found' errors on my Sun box. This was supposed to have been cleared up with the new networking and cables, but it wasn't. This was made infinitely worse today as I had problems with the production MarketData server and without my Sun box to develop/debug, I was in the dark. They came and replaced the network card in the box and reconfigured it to use that new NIC as opposed to the one on the motherboard. This took about an hour from the time I started 'yelling' about the problem and it's impact to production, so that's not too bad, but that it was out after having told them to replace the NIC weeks ago is a little troubling. But I hope this is it for this problem.

Next, there was the problem that I needed that box to solve. That was, the database that I get symbol mappings from was not returning data as it should have. If I went into SQSH, the data looked to be there, but when I went in from the program and the SQLAPI++ libraries, it wasn't. After a bit I gave up and pointed my code to the primary database (as opposed to the read-only replicant) and it worked perfectly. I had seen a few database issues this morning early, but thought they were all fixed up. I was wrong.

Later this afternoon, I saw that the read-only database was working again, and I'll switch production back at the end of the day. I try to be a good corporate citizen as much as possible and sing the read-only database is something they want me to do. I'll comply as long as it's working, but when it fails, it's back to the primary as the users have to come first.

After I got that fixed up, I had to fix up problems with my server and the EQSVal libraries. It turns out that with the most recent version of the libraries (v7.3.24b) you have to set things that were once defaults in previous versions. This isn't horrible, and the fix was simple - but figuring out what needed to be done was the problem. Thankfully, the EQSVal guys were helpful and once I generated them a test datafile they were able to tell me the issue and what to do.

After that, I had to make several changes to the web-based editor for my server. Nothing major, but there were some edge conditions that were showing up as records in lists that should not have been there. The perl processing and filtering was about as good as I could do in a single pass, so I had to add a second pass on the data to further clean it up. This will take a little bit more on the execution, but it'll keep the records cleaner and therefore make it less confusing to the users. Good things to do.

Finally, I got the most recent version of a driver from the internal price source folks at the Bank. I needed this because their mapping rules from Reuters to what we needed weren't capable of being exchange-specific enough to really hit the points we needed hit. With this update, my contact Todd is able to get the rules as specific as necessary and that's great. I'm hoping that these work out in test over the next few days so we can roll them out to production as soon as possible. These are tickers that the users have been wanting more smarts in, but haven't been able to deliver due to the limited rules on Todd's side of things.

That's about it. It's been a monster of a day. I'm looking forward to calming down and taking it easy for a bit.

Well... Liza ran her second marathon this weekend (in Philadelphia) and did a great job. Things worked out from all the lessons we learned from Chicago - take the water, take the phone, be at specific spots - all worked well. The weather wasn't as nice as it could have been - low 40's and falling throughout the race, drizzle to rain, and a ton of wind, but she did it. Ran every step. Her half-marathon split was good, and she understandably slowed on the second half. Finished in about 4:53 - which is great for a second effort.

Don't know if she's going to be doing more of these, but it's certain that we have the procedure down. It's great to watch her run and great to cheer her on.

Today I finally figured out the problem I have been slugging out for a few days. Actually, there were a few - none of them of my creation, but legacy issues that I have to deal with in order to get the server making a little forward progress.

The first was the fact that the Bank's valuation library, EQSVal, deals in dates that are ints and I had assumed that the original author of the conversion code had checked to make sure that the conversion was done properly. After all, the data looked right. In C/C++ the standard for converting dates to ints is to compute the number of days since 1/1/1970 - the epoch for all Unix systems. And that's what the server's conversion code was doing - converting the dates based on that reference date. Interestingly enough, the EQSVal library authors have decided to reference the same date as Microsoft Excel. Why? I can't possibly tell you because it makes no sense to me, but that's not the point, I suppose. The point is that they reference the first day of the last century - 1/1/1900. This made all my dates off by 70 years - some 25,567 days.

I saw this problem and realized that the easiest thing to do was to make a new method on MMDate called daysSince1900() so as not to confuse the other uses of the method toInt() that got the date with respect to the Unix epoch. This fixed the first of the two problems. I'm convinced that when I was trying the latest production version of the EQSVal code I was getting what they classified as operating system errors most likely from doing the reverse of what I was doing - subtracting 25,567 from the '1900' date and then using the Unix tools to then trying to convert it to a Unix time struct. Getting negative numbers might really be a mess, and could easily have been the issue.

So... one down, one to go.

The second problem I found was the quote date for the stock 'products' going into the model. The original code had the driver date - which is the quote date offset by the settle offset for options. This was a typo in the code and was leading to problems in the model. EQSVal checks the volatility curves (volatility, skew and kurtosis) to see that the first point in each curve is not before the quote date for the evaluation. Because the stock product was using the driver date, and the first point in each curve was only a business day away, it was advancing over the start of the curve and it was flagging an error. Most of the time this would not be an issue, but near the expirations this is going to happen as the volatility curves are calculated on these expirations.

With both of these problems solved, the values were calculating properly and no flags were being raised. Good enough. Not easy to find, but I'm glad I stuck with it before I left for the weekend.

UPDATE: it turns out that the EQSVal libraries allow for me to specify the date of 1/1/1970 which they use for offset. In general, then, the difference should not have mattered, had we set the offset right. For now, we'll leave it, but it's nice to know that there's a place we can control the offset itself.

There are those days that make you love the Job. There are those days that make you hate it. But for the most part, it's just that - a Job, and there's nothing particularly exciting about it and nothing particularly aggravating about it either. Today is one of those days.

I'd rather be doing something other than what I'm doing - which is debugging other group's libraries in my code without their source or support. It has to be done, because there isn't a real alternative, and it's not horrible because I have decent ways around most of the limitations I've found in their code, but it's still just plain old grunt work. Nothing fancy or glamorous, but the bread-n-butter of the developing job.

It's not fast because a significant change requires a restart which takes about 20 mins, but I try to make those significant, and therefore informative. It's going to get solved, it has to get solved. But it's not really something I'm going to put on my resume. It's just the job.

Today, actually yesterday evening, a trader had a problem with some of the data my app was showing. Now I don't generate this data - I just display it, but as is often the case, if it's connected with your app, you get at least partial blame for it not being right. I understand this and it's something that, to a large degree, I agree with. The problem comes in where the guys feeding me the data won't (or can't) fix the data coming to me in a timely manner. Then I have to get into two modes: Defensive Systems Development and someone else's debugger.

Today was one of those days.

After coming up with a plan to avoid the problem in the short-term, I set out on the task of making my system capable of filtering out the bad data from the upstream source. It wasn't something I was planning on doing today, but it wasn't bad and allowed me to add another level of flexibility to my flagship middle-tier. Not bad really. But in the middle of trying to get al these changes in for stopping and starting the flow of updates from my main collector/distributor, there were a flurry of emails about the upstream system - a database, and trying to find the problem there so that we stop it at the source.

I'm all for fixing things at the source, but I soon realized that they weren't seeing the problem. So I had to just into the fray and try and figure it out myself. I'll admit it was a tricky little problem in a stored procedure. Basically, a table which originally presumed to have only one day's worth of data in it was having, for a few hours, two days worth of data. This wasn't bad because when they changed to making it have multiple days of data, they changed the stored procedures to look at just one day's data in that table. Or so we thought.

In actuality, it was not looking at just one day's data because the where clause on the defining cursor was not limiting the primary table's data to just one day. This meant that for a time, two day's data was being used in the cursor. This would have been fine, if it not for another bug in the storied procedure that didn't limit an update statement to a single day. Now the bug is evident - two days were being looked at in the cursor, both days were being updated at the same time in the update statement, and one of those days was not really complete.

That meant that the incomplete day was getting used and updating the complete day and that was the bug. Once I figured that out, I told the manager of the project, and when I had convinced him, told him what he had to do to fix it. He then had one of his guys fix the stored procedure and we should be fine.

I still finished my enhancements to the filtering on the data sources because I think it's a good thing to have. But it's probably no longer necessary. We have it fixed at the source and that's going to be all we need.

I was messing with Vim's ctags support today on my Mac and then somehow wondered if either SubEthaEdit or BBEdit had support for ctags like Vim had. I dug into it and in fact BBEdit can support ctags, but unfortunately not the ctags that exist with Mac OS X 10.4 - you need the exuberant ctags on sourceforge.net. They talk extensively about it in Chapter 14 of the BBEdit manual. You need to download, build and install the ctags app:

    cd ctags-5.7
    ./configure --prefix=/usr/local
    make
    sudo make install

In order to have it not conflict with the existing ctags I did the following:

    cd /usr/local/bin
    sudo mv ctags ectags

I also changed the name of the man page to get that matching the new command:

    cd /usr/local/man/man.1
    sudo ctags.1 ectags.1

and to make sure this man page gets into the existing MANPATH - if you don't want to add this location to your MANPATH, you can do:

    cd /usr/local/share/man/man1
    sudo ln -s /usr/local/man/man1/ectags.1 .

Then I created a simple alias that allows me to call it simply with the right arguments for the file BBEdit needs:

    alias ectags `ectags  --excmd=number --tag-relative=no  --fields=+a+m+n+S -R'

So that in the Makefiles for BKit and CKit I can add the target:

    CTAGS = ectags  --excmd=number --tag-relative=no  --fields=+a+m+n+S -R
    ...
    tags:
        @ $(CTAGS) `pwd`/src

And this way I can then automatically pick up the tags in both projects. It's nice that CVS doesn't try to update the file called 'tags', and placing it at the top of the source tree allows BBEdit to find it - as well as Vim, if I'm into that as well.

In the end, this is a really nice little addition. I'm a little surprised that the BBEdit folks didn't include it as a part of the BBEdit distribution so that you would not have to download it and build it. I mean really they know that Mac OS X 10.4 doesn't come with it, and they are going to need it, so I can't quite figure out why they didn't include it. But they didn't. Easy enough to download and build.

UPDATE: I also found that there are 'Jump' and 'Jump Back' menu commands that I've got hot keys set up for. This makes it very easy to jump to a functional definition based on the ctags and then back to where I was, and then back to the definition. Very nice. Gotta love BBEdit for this.