What's it all about, Alfie?

Well... this is a pain in the neck, but I'm glad it's solved. The problem is that I was trying to rebuild CKit on my Mac using make and Xcode 2.4.1. It compiled fine, the test programs linked fine against the generated .dylib dynamic shared libraries, but when I tried to run it I got:

    dyld: Library not loaded: /var/tmp//ccr3sui.out
    Referenced from: <path_to_test_app>/test
    Reason: image not found
    Trace/BPT trap

The problem was that the 'tmp' file wasn't there. When I did an otool -L on the application I got:

    test:
      /var/tmp//ccr3sui.out (compatibility version 0.0.0, current version 0.0.0)
      /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.3.3)
      /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.4.0)
      /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

When I looked at the shared library that I was testing, it had the same reference to this temporary file. I was stumped and blown away. I googled a lot of things and then found a reference that had a similar problem. The solution was amazingly simple and yet it should have been taken care of my Apple's compiler.

The link phase of the build of a dynamic shared library allows for the addition of a few options that shouldn't really matter unless you want to take advantage of the unique features of Mac OS X. They are:

    -install_name libCKit.dylib
    -current_version 1.0.0
    -compatibility_version 1.0.0

where you can make the versions anything you want, but the key to this riddle is the -install_name. By default, the linker is setting it to the temporary output of the link, and not the name in the -o parameter. The docs say this is the default for the install name, but it's not. When you use the same value for the -o option and the -install_name you'll see that the output of otool -L changes to be:

    test:
      libCKit.dylib (compatibility version 1.0.0, current version 1.0.0)
      /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.3.3)
      /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.4.0)
      /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

We can now clearly see the difference. As long as you have this library in the path specified by DYLD_LIBRARY_PATH, you're good to go.

I spent hours on this. I was trying all kinds of things to see where these temp files were. It's a stroke of luck that I found the web site with the reference that I needed. Yikes. Well... now it's here and maybe it'll help the next poor sap that's got the same type of problem.

For the last several days I've been heads down coding this 'infinite' precision floating point number in Java for BKit - BKFloat. I learned a lot of really interesting things in the process. Now that I've dug myself out of that all-encompassing task, I can take a little time to talk about it.

When I started working on it I thought that the best way to implement the class was to have a long as the whole number part and another as the fractional part. Then, when I needed to do any math, it was pretty easy as I could take advantage of the long's ability to do the math. And this got me quite a ways to the end. But I started running into a lot of problems when I got to the point of really building the add() and subtract() methods because I was getting into coding based on the long and not a general 'infinite' precision floating point number.

For instance, with the long, I still had to deal with the fact that I did have an upper limit on the number of digits I could represent. Sure, it was big, but it wasn't as big as it might need to be. Some of my test cases had numbers in scientific notation and for those guys I had 5.5511232344325E-17 and the like which made it very hard to make sure that I had enough digits to express the non-zero elements as well as the proper magnitude of the number.

The final problem that snapped this design was the use of the sign on the fractional part. Imagine that you had two longs - one for the whole number part and another for the fraction with an implied decimal point in between them. If I had a number like -0.5 the whole number part would be 0 and the fractional part would be 5 - but where did the sign go? If you had -1.5, the whole number part would be -1 - there's the sign. So I had to 'pack' the sign on the fractional part if the whole number part were zero. This lead to a lot of code to make sure I had the right sign of the number for the operation. It was looking ugly and I just knew there was a better way.

So after about a day of that I backed off and thought that the better way had to include the resizing of the digit storage in order to make sure that the only limitation to the size of the floating point number was the capacity of the machine. I also had a feeling that by separating the digits I'd be able to implement the arithmetic operations a lot easier because I could code it like third-grade math. So I started off on that tack.

The next day was spent gutting the code of the long-based code in factor of byte[] storage for the digits. I spent a little more than half a day getting to basically the same point that took me the previous day. I had added a lot of little things like the ability to shift the number right or left as if multiplying (or dividing) by 10. This made a lot of things easier and in general the code was drastically simplified.

The things we learn in third-grade are really powerful. Coding carry and borrow on the add() method was interesting and a bit of amazement at what we all do so easily. It's really quite amazing. I've done the classic computer numerical manipulation with the XOR for multiplication and the full adder, but this is interesting in that it wasn't base 2 and yet it was exactly a digit at a time. Very interesting. I don't think I've ever written code like it.

Multiplication was interesting and fun at the same. Division was almost fun once I got into it. Of course, the division was the first one that might have a loss of precision due to the mature of the operation so I had to do a few interesting things there, but in the end it was working remarkably well. I don't think this is going to set any speed records, but the point is not speed but precision and accuracy. The test cases I used in the development pointed out that there were plenty of times where a simple double in Java is just not very good at representing values. This class is for those times when you have a handful of numbers that have to be added, etc. without any loss of precision. For those cases, speed is typically not as important as the precision. Good enough.

Now I'm back down into the hole to convert it to C++ for CKit. Shouldn't take too long... all the logic and method calls are already worked out.

This morning I got a visit from Ammo and he's guarding my desk now when I take a break. It's nice to know that I've got an angry amoeba with an automatic weapon looking out for me.

Friday, a developer stopped by and talked about the problem of dealing with the precision problems in Java and C++. I thought about it a lot over the weekend and decided that I wanted to code something up for BKit that would be a general purpose object that could hold these floating point numbers without loss of precision due to representation or operation.

I think the best way to go about this is to have a long for the whole number part and a long for the fractional part. This way, it's very easy to work on the parts, but they will be able to hold a very large number.

It's off to coding...

UPDATE: after spending the day on this approach, I've decided that it's not the best. I'm going to start over tomorrow morning with byte[] where each byte is a digit in the two parts of the floating point number. The reason for this is simple - the longs were giving me limitations that I needed to deal with and it was getting very much code for the sake of the data storage choices. So I think I'll be slower, but much better with a more general storage like byte[]

Today another developer stopped by with a problem I've seen several times before - how to handle user validation of percentages. Specifically, let's say a user needs to put in how to subdivide something - 10% to this, 20% to that, and 70% to the other. Breakdowns like this are easy - until they get into factional percentages and numbers that are, by their very nature, impossible to represent exactly in a computer.

Take 0.12 - it seems easy enough to look at, but try to add it to 0.74 and 0.14 and you're going to find that a double is not exactly 1.0. That's because the numbers are not exactly what we typed in and there are representational errors in the doubles that make it something like 1.00000001. So how to fix this?

One way, which I've done for many cases, is to have an ε something like 10^-6 and check to see if the value you want is some ε from the target value by:

    if (Math.abs(sum - target) <= 1.0e-6) {
        // close enough to be considered equal
    } else {
        // not equal
    }

which can be put into a method or function that makes it easy to call. But the idea is that if you get close enough, you're equal. The problem with this is that the ε that you should use is greatly dependent on the values you're summing, in this example. Say you have values like 10, 20, and 70 - in this case, ε should be 0.1 because there's no need to have it any smaller as all the numbers are integers. But if you had numbers like 10, 20.22, and 69.78 then you may need to have ε set to 0.001. The optimal value of ε seems to be directly related to the maximum precision of the data. Also, there's always the possibility that some oddball case will be just large enough a difference to not be 'equal' and then you're flagging a false positive. But there is an interesting way that this won't happen.

Look at the problem of the user input as if you were a third-grader. Break the problem down into the whole number and fractional components. This makes everything an integer, and adding integers is very easy indeed - with no rounding problems. What I mean to say is that if you had the set of numbers: 10, 20.22, and 69.78 then break the summation down into 10 + 20 + 69 + (00 + 22 + 78)/100.0. This means that you only have to look at the string representation of the numbers - separate out the parts to the left and right of the decimal, find the maximum precision to the right of the decimal (for scaling), and then go to town.

After we talked about this approach for a little bit, the developer went back and coded it up in no time. It's really very simple. But given that users can type in arbitrary precision in the numbers, it's important to sum them as exactly as possible, and using integers for the whole and fractional parts is really pretty sweet. There's no rounding (until you get to the final double) and by then, you're comparing it to the test value, and if it's another integer (in the case of percentages), then it'll be easy.

I giggled a bit thinking about this. It's not often that the best way of doing something is the way a third-grader would do it. But that's the point - not everything needs to be so overly complex and over-designed.

Today I had to spend a lot of time on a application that got dumped on my lap after the last developer working on it couldn't get a new price feed hooked into it and was asked to leave. Since I had done a lot of work on this price feed, it seemed natural that I get the app hooked up to the new feed. It took me a few days and worked great. Then there came the need to add in permissioning for fee-liable exchange data, so again I was the logical choice because I had done similar work for other apps. Ahh... that slippery slope.

It didn't take long for me to be the maintainer of this app. That typically meant that features that haven't worked in years suddenly working because I found mistakes in the code and/or database and then they magically started working again. Some I was specifically asked to fix, others were resurrected completely by accident when I was fixing something else.

Now I've been working on this code for about 9 months. Not full-time, more on an as-needed basis, so I don't mind too much. But I need to say that this code is a real mess. I mean the worst code I've ever seen.

It's primarily C++ on Windows using Visual C++ - but the build systems is GNU make on Cygwin. It uses JNI, so the JDK is involved and that's a mess. There is virtually no documentation, there are dozens of sub-projects. The database access is through a separate process on another machine that has it's own configuration, it's asynchronous and not easily followed. It's over-designed and poorly coded. In short, it's a mess, and it'd be easier to re-write it but the logic is so buried in the code that it'd be easier to sit down and talk to the users to see what it's supposed to do as opposed to how it does it.

One of the features of this app is the ability to create new views (browsers) on the data. This controls the columns shown in the table, their order, how the view is updated, etc. Yesterday one of the big users of the app asked me if it would be possible to make it so that you could create new browsers. The app looks like it's supposed to work, but it doesn't. So I said I'd look into it and see what I could find out. Yikes! what a mess.

The code was a mess and had been that way since 1999 - I found comments where they had hard-coded the name of the new browser to 'x' - literally an 'x', and it was commented as happening in 1999. What were they thinking? I have no idea. I tore that out, figured out what was being passed into the method by way of these odd notification objects, and realized that it wasn't that far from working - on the client side. It took less than 50 lines of code changed, but finding those lines was the challenge. The real problems were on the database layer configuration, and stored procedures.

When moving to Sybase 12.5 the IDENTITY column had to change from an 'int' to a 'number', and in doing that, most all of their stored procedures were broken and never fixed. The access layer expected ints, and they weren't being returned that way - so basically, all the database work was getting dropped in the bit bucket. It was similar to what I'd seen before in this app, but what a striking example of poor updates and testing.

Then there's the bad UI, but that's just because it was created by so many developers over so may years there's no way it was going to have a consistent vision and style. Horrible. Which brings me to the point of this post - the code is ancient. It's been worked on by so many developers that had no idea what the point and architecture of the code was originally about. It's been slapped, hacked, poorly tested and given unworkable hard-coded hacks. It's amazing it still compiles. But this is what some people call developing. I can't imagine putting my name on this if I'd done it. I'm happy with the fixes I've made, but those are only in light of how bad it was to start with.

It's just amazing it works at all.

Yesterday afternoon one of the developers came by to ask me about finding historical prices for delisted symbols. "That's hard", I said - because it is. Bloomberg has historical data back some 20 years, but you have to know their symbology to get it out of Bloomberg, and one of the things my MarketData server does is to allow us to use our own symbology and it converts this to the symbology of the data provider (in this case, Bloomberg) and then requests the data. This is great so long as you can do the symbology mappings. The problem comes in with delisted symbols - they aren't in our mappings table so I can't convert from our symbology to Bloomberg's. This means that eventhough you might be able to get at the data, you don't know what to ask for.

I told him he'd have to investigate another data provider and get the data from them. This other provider needs to have the historical data and some sense of the re-use of tickers so that given a date and a ticker it knows that there was one - and only one instrument for that ticker on that date. This kind of historical data is not common, but it's available. You just have to know where to look.

So we started looking.

There's a project in the Bank that handles a ton of historical data from a lot of sources - one is Bloomberg. So we did a few tests to see if this source - which was in my MarketData server as a provider, had the data for a delisted company. Turns out it did. But at the same time, I found that there were problems with how I was calling and using this data source. So I dug into fixing them.

The first was pretty easy - when I requested data of this source, if I received nothing, then I assumed it was an error. In reality, if there's no data, they returned nothing. So if you asked for a delisted symbol after it was delisted, you'd receive nothing. I fixed those tests up to allow for nothing to be returned. Easy enough.

The second one was more interesting. The data fields you request should stay the way you request them - case-wise, but I wasn't preserving the case of some requests and mismatching the case of others. So I added several nice little methods to CKString to allow the user to copy a string and uppercase it (the only existing method uppercased the string in-place) and then I added equalsIgnoreCase() like Java's String to allow the code to not have to worry as much about case in the tests. When I put these in and realized that I needed an uppercase copy of the fields for getting data from the source, caching it, and then using the original field list to pull it out of the cache, things worked beautifully.

In the end, it was a lot of fun to add these things and see how they simplified the code I was trying to write to fix the problem. Lesson learned: add power to the underlying libraries and that will make any higher-level fixes much easier.

Every now and then I give a look-see to Sun's Java web site to see if they have stated when they'll start to deliver a Java plug-in for x86_64. There's a bug report on Sun's web site, with a ton of comments blasting Sun for not getting a x86_64 64-bit plug-in working, saying that this is addressed in JDK 1.7 (Dolphin). I took this for a good sign as I went to the Java web site to see if there was a beta of JDK 1.7 available - and there was.

So I downloaded it and looked inside the jre directory to see that no plugin exists. It's amazing. They say they're going to have it, but then they don't follow-up and actually deliver it. Now, it's possible, I suppose, that this is going to be added in the final release, but I'm guessing not, as that would mean that something is in the final release that was not in a beta release - not very likely.

So I decided to see if this Blackdown 64-bit java plug-in would work, knowing that it was stopped at JDK 1.4.2. Interestingly, their site, http://www.blackdown.org/ comes back with nothing in the web page. I've googled for this site, and it keeps pointing me back to this empty page. There are several sites like this that tell you how to install it, but all their links to download it are pointing to bogus sites. I finally found a page saying that Blockdown is no more. That explains it. I did find a mirror site that had it still, but when I tried it with my FC5 box and BonEcho 2.0.0.3, it crashed. Too bad.

Well... I guess I'll have to wait for Sun to decide that 64-bit desktops are important enough to support. Thankfully, my laptop will have it with the new version of Mac OS X (10.5) due out this month. Good thing I don't have to wait for Sun for that.

OK, I'm trying to be fair and even-handed about this, but this morning I got another gotcha from Java (1.6.0_02) and IE (6) and Firefox (2.0.0.7). It's this thin client development that is 90% of what's needed but when you hit that 10% it's a pain in the rear to try and get around it. This morning I spent several hours doing just that.

The problem came up because one of my servers uses a web client to maintain it's internal data. This is nice in that the previous 'editor' was a Java application and it was a lot heavier than the support staff needed to have. Plus, with the web delivery, it was easier to have these folks support the server from London and home. But there's always a cost.

One of the things I did a while back was to have the users able to edit start-of-day (SOD) positions for the instruments in the master file. In order to do this in a reasonably useful way, it meant that I had to have a Java applet. This was not the first Java applet on the editor, and I knew that there would be IE/Firefox issues and how to get around them to make sure that the applet launched, etc. Nothing new there. What was new was the fact that I needed to send back data when the user hit 'Save'. I'd probably have been happy enough with Java applets if I hadn't had to do that. But I did.

Moreover, I had to send these updates through CGI scripts where the old and new data were arguments to the GET request. Not terribly hard, but when you find that IE still allows JDK 1.6.0_02 to send URLs to a server and get back answers, and Firefox doesn't, then we get into the lovely realm of applet signing.

It's not that terribly hard, but I didn't want to get a certificate that would mean the users would have to manually accept, so I went on a search for the location of the certificate that the web developers have built to enable them to sign their applets. This way, I know the certificate is allowed on all the boxes, and I don't have to worry about that.

Easier said than done, and honestly, it wasn't Java's fault. But the idea that in these days of corporate intranets and apps deployed on them, it would seem reasonable to have the security system say "Hey, if it's on this domain, it's OK". I know that's not necessarily safe, but coupled with the fact that there's security in the assignment of IP addresses and the domain naming, it's very unlikely that someone is going to put a box on the net with the right IP and domain to publish fake jars. Could happen, but it's not likely.

So I had to sign the applet. Interestingly enough, Googling this revealed that it wasn't until a recent release of 1.6.0 that this difference in behavior between IE and Firefox came to light. Previously, the behavior of IE and Firefox was the same in that it'd allow the URL connections from the applet to a machine not hosting the applet. So this might change again in the future. Yum...

The next problem was NFS... and it's likely a linux problem, or a difference in the NFS system we have in the Shop, but it turns out that if you copy a file from one machine to a shared filesystem, there's about a 70% chance that the web server will think this is a corrupted file. So, after seeing that the ClassNotFound exception was on the Java console, I went to the web server and copied the file to the same shared location. This finally worked.

After all this, things started to click and I got all the issues worked out.

Now I know that Java isn't perfect, and it's a lot better than ActiveX, which came before it, and I'll even say that it's a alot easier to deal with than AJAX, but I keep thinking that there has to be a better way. I know that AJAX is making strides, but it's not really a revolution, more an evolution. I'm wondering where's the revolution in thin clients? Where's the easier system of making clients deployable easily and run cleanly. I've seen Java WebStart and its a lot like Marimba - you download the app and run it. Yeah, it makes deployment easier, but it's no different than fat clients.

Throwing together a Java RMI server would be possible, have the applet connect in and send the updates - that'd get rid of the CGI scripts, but it's not going to make the security issues with Java any more manageable. Make it easier to write powerful apps that are easily deployed remotely. Tons of OSs have done it - X, NeXTSTEP, even VNC does this. But these are for something large, like a desktop or a big app. I'm thinking small - like most applet work. That would be nice to find.

This past weekend Liza ran the Chicago Marathon, and it was an experience. First, it was a disaster as a good race as the temperature was >95°F. and there were problems with sufficient water on the route, and after a while they cancelled the race but told the runners they had to walk back in. The transportation that was to be provided to get the runners back to the starting line was not available, and so many runners, like my wife, had to walk it in. This added several hours to her time and I had no idea where she was. So... there are several things that we, as a Team, learned about this, her first marathon:

1. The Runner Needs a Cellphone - until the time that they have GPS on each runner, or the Chips mark each mile and an easy way for each support person to know where their runners are, a cell phone is a necessity. Had I been able to call her, or she to call me, we would have known that she was falling off her pace as early as mile 9, and I would have waited at mile 13 for her, and updated my expectations of her arrival time at mile 16.5. As it was, I was a bit frantic not being able to find her past mile 2.5.
2. It Doesn't Hurt to Carry Water - there were water stops that had no water at them when she arrived. This was a serious problem in the >95°F. weather. So, she and I talked about her wearing her water belt, and discounted it. Next time, if that heat is up there, she'll carry it just in case this happens again.
3. It Helps to Have Support Crew by the Phone - had they used the Internet to track her progress better - more Chip splits, then it would have been really nice to have someone at home with a computer that could monitor her and update me to her progress. As it was, the kids didn't know what to do, and the only splits the system sent out was her ending time. Hardly worth it.
4. Have a Go/No Go Point, with a Backup Marathon - we had the back-up marathon as many people had recommended, in case of injury, but there wasn't a point we had that she'd pull herself out of the race because it wasn't an enjoyable experience. For example, had we said "If you drop below a 5:30 pace by the 13.1 mark, it's time to pull out because you've trained at the 4:15 level and this would mean you're getting beat up by the weather or the course."

I'm sure there are a lot of things she's learned about running in a marathon, but these are the big things I've learned supporting her. Until I can take the time to train with her - or at least train for the same race like her, these are the things we're going to have to keep in mind if she hits the pavement again for another 26.2.