What's it all about, Alfie?

Well... it appears that the current Java 1.5.0 on Leopard is not all that great. Add to this that there's not even a pre-release version of 1.6.0 available for the early-adopters to use and it makes a somewhat compelling argument to hold off on Leopard for at least a little bit until Apple releases the 1.6.0 JDK. I'm sure it's coming, Macs have been for the last several years the Java development platform. But with the fact that 1.6.0 never fully materialized for Tiger (10.4) and that it's not in Leopard (10.5) - yet, it makes those hard-core Java-types say that Apple is out of touch. After all, Windows uses the Sun-provided JDK and Apple insists on writing it's own.

But let's recall a few things folks... with Mac OS X 10.3 and JDK 1.4.2 Apple made the apps load very fast and run even faster. So much so that Sun asked to have a look under the hood to see how they did it. With each successive release of Mac OS X the bar has been raised. Unfortunately, right now, 1.5.0 is almost two releases back and 1.6.0, while not widely adopted, is widely available for a long time.

It's a matter of timing. I'm sure that within a few weeks JDK 1.6.0 will be out on Leopard and it'll be once again setting the standard for Java development platforms. There are just those that are hearing all the hype and cheering from the new release that they need to put it all in perspective - which isn't a bad thing, as long as it's not use to just knock things down.

Leopard is fine, it's new, and it's got a lot of differences, and I'm still waiting for PHP and PostgreSQL for 10.5 to be available from Marc Liyanage as they are the best packaged, fullest-featured, versions of those tools. When he's got time to update them, I'll move to Leopard. I'm guessing that by then I'll be updating to JDK 1.6.0, or soon after. It's just a matter of a few weeks. Have a little patience.

UPDATE: interesting note about Java on Leopard... is seems like the guy got a lot of press for whining about a specific application instance. In general, it's better and faster, but there may be times that you have to actually code against the defaults to get optimal speed. In any case, one less thing to worry about with Leopard.

A very long time ago I put in a simple plist reader to the BKHashTree so that I could read in the plist files off my Mac OS X system into a map of maps, as the BKHashTree is, and display them in a nice graphical tool that I was building. The GUI widget I created has found a lot of use in the code I've written over the years, and when talking to a fellow developer late yesterday it was clear that what we might really need was a full-featured plist reader. I remembered something about this, looked it up, and sure enough, I had the basics of this already coded up. While he had found something that looked to have all we needed, it had somethings we didn't need, and I wanted to take a shot at cleaning up the reader I'd made, add a writer, and augment the data types allowed in the plist file.

The first thing that was missing in my code was the list/array element. Basically, we wanted to be able to provide the user the ability to specify a list (Java ArrayList) of elements as simply as possible:

    {
        names = (Bob, Fred, John);
    }

It was pretty easy to see that the simplest way was to process the array as a single argument (value, in this case) to the routine that I had that picked the correct object type of the element and made it available for placement in the Map. This worked for a while, but pretty soon I ran into serious limitations. What I needed to realize is that the array processing is very similar to the map processing - just with a series of values as opposed to a series of keys and values.

This was a major simplification in the code. I had a map processor and an array processor and they called each other. Should one of the elements of an array be a map, it passed control over to the map processor, which returned a Map to place in the array. Likewise, if the map processor hit upon an array it sent the code over to the array processor which returned an array that would be used as either the key or the value. Very nice. It was only at the very lowest level that we needed to look at the elements and determine the data type they were.

The next thing to do was to add in the URL and binary data types. The URL was pretty easy - it's a string with '://' in it and no spaces. The binary data was a string starting with '0x' contained within <> like this:

    {
        password = <0x347ab0ef21>
    }

There's still the limitation that there can't be a space after the '<' and before the '0x' because the BKData needs to look at the string it's given as starting with '0x', but that's something that I can fix without too much trouble - should it become a real issue. Right now, I can't imagine what we'd use the binary data for except passwords, and those we wouldn't be hand-edting because they'd probably be encrypted or at least Base64'ed.

The final thing I worked on today was the writing of the data and while the writing itself was easy, making it look decent when written wasn't. I tried to balance the ideas of putting individual items on a different line and making it look more readable by putting small arrays, etc. on the same line. I know it's not perfect, but it's very readable now and that's enough. It's not meant to be a beautiful output- only functional.

This all got started because we wanted to have a distributed Preferences system in Java that didn't have the current limitations that the current crop of Preferences Factories has in Java. More on that tomorrow.

Today a developer stopped by to say that one of the graphing classes - the combo graph (bars and lines), had problems when you degenerated it to a bar graph (no lines), and to a line graph (no bars). I checked and he was right on the money. Now one might ask why this is important, but there are times you'd like to just code it up and not worry about the number of lines or bars and have the GUI widget smart enough to just do the right thing. So I dug into the code to see how hard it might be. Interestingly enough, and I probably should have seen this coming, one was a lot easier than the other.

In the VantagePoint scheme, the combo graph is a subclass of the bar graph (hint, hint) and so the first thing I tackled was the case where there were no lines at all. The problem I had was that when I saw that the user didn't specify any lines in the combo graph I simply did nothing, and that's not the best thing to do in this case. What I needed to do was to see that there were supposed to be no lines, and then set that in the graph. Pretty easy.

Then I tried to do the same kind of thing for the bars, and surprise! that didn't work. NullPointerExceptions deep within the VantagePoint code that had clearly undergone some kind of code obfuscation so that the classes and method names meant nothing to me. Finally, through trial and error I came to the idea that VantagePoint was simply not going to allow a combo graph to have no bars. (well.. duh!)

I changed tack and said "Hey! A combo graph without bars is a line graph!" and what I did was to have the applet code detect the 'lack of bars' condition and switch the graph to a line graph and then plot the lines on that graph. Bingo!

I suppose I should have seen the lack of bars on a combo (bar) graph as a non-starter for VantagePoint, but I assumed that if it had either type of variable it would be OK. That was a mistake. The bars are the driving variables and the lines are the 'decorations' on the base graph. Makes sense now, but finding it in the VantagePoint NullPointerException was not easy. But I got it.

I've been working with SubEthaEdit a lot in the last few days, and also with BBEdit, trying to get the few features that I don't like fixed/cleared up, and while I really like both of these editors for their unique approach to the problem of a developer's editor, there is a lot to be said for Vim. I have done more coding in Vim than probably any other editor in my life. I've used it on so many systems it's amazing, and I've never had to compile it myself. There seems to be someone who's already figured out how to get it to work on every platform I've needed. The community support is really amazing.

I have been using the Vim package from macvim.org, and it's currently at 7.0.224 for Mac OS 10.4 (Tiger). I am able to use all the nice fonts, syntax highlighting, everything that I expect in Vim, just on my Mac. The only problem is that it's stuck in Vim's view of the world. That means you can't have multiple windows open on the same application instance. On Unix this isn't an issue because you typically start a Vim session from the command line, but on Mac OS X, it's a limitation that I find... well... annoying, and it makes me not use it as much as I might.

There are also several things that the other two do that are very nice that Vim misses out on. It's not horrible, but when given the choice, I really end up using SubEthaEdit and BBEdit more on my Mac than I do Vim. But it's close. Many of the requests I have in to the SubEthaEdit developers and BBEdit developers are things in Vim that I wish they'd do - or do better.

Still... I can't complain. Vim has been very very good to me and I hope they come out with a Leopard release (Mac OS 10.5) very soon so I can upgrade my laptop to it and keep Vim with me.

This week marks the time when we have to make final preparations for going off Daylight Savings Time (DST) which is important because Hong Kong does not observe DST, and so we're going to loose an hour at the end of our day and the start of theirs. This is important for trading and risk apps like mine. The original solution was to make my app run 24x5 without any interruptions. This was a significant technical challenge and after about a day or two of this, I decided that it'd be better to make use of extra hardware to have two servers - one that covered the US and Europe, and another that covered the Far East.

Problem was, this meant that there was going to be a lot more maintenance of the system to keep two systems up and in sync. After hearing me say this, most said "Yeah, yeah" nodding their heads, but thought I really didn't understand the situation. When I came back with a list of things that would have to be done each evening to make this work, all of a sudden the easier solution to the problem appeared.

The problem really comes down to the fact that we typically didn't start the nightly processing until about 5:30 pm, Chicago time. This was 6:30 am Hong Kong time, and after the DST change, it'd be 7:30 am Hong Kong time. Given that the start is then, it's likely that we are not up and going by 8:00 am local time in Hong Kong, and that's the problem. So I had suggested that we move up the end-of-day processing, but that got nowhere. Then they saw what they'd have to do to support the two servers capable of giving Hong Kong the data they needed when they needed it.

All of a sudden, the end-of-day processing was moved up to 4:15 pm Chicago time. Plenty of time before Hong Kong would need it.

While I'm a little frustrated by the fact that I wasn't listened to earlier, I have to take solace in the fact that eventually my ideas found ears, and things started to move in the right direction. Now the second hardware is for disaster/recovery and there's only one server covering the globe. It's simpler, and while there are bound to be bumps in the road, things are going to be better by moving up the end-of-day processing.

That first bump was last night when I got a call on the train. It was from someone asking if there was a way to edit the data that's sitting in the middle-tier of the application. I said "Nope, it flows from the server, and that's already thinking it's tomorrow." I knew what they wanted to hear, but it wasn't going to happen. The data in the middle-tier is too complex to simply edit with something like a text editor. The values are interrelated and computationally intensive calculations need to be done in order to derive some values from others. It's not easy. Which is why there's no easy editor of that data. But because of the early restart, the guys didn't check on the state of the system early enough in the afternoon to make sure that it would be good to go for the earlier restart.

Lessons learned. Like I said... a few bumps in the road, but it's a better place we're going to.

Whew!

It's been a week.

Pretty much all this week I've been heads-down coding this 'infinite' precision floating point number first in Java and then in C++. There were a lot more issues to deal with than I had expected to run into in this little class - face it, it holds a number, adds, subtracts, multiplies, and divides. That's it! But, boy-o-boy, there's a lot more to just that than it sounds.

I've talked about a few of the issues in the post I made after the Java version was done. But then the conversion was pretty easy for many things, but had quite a few gotchas in the C++ codebase. Little things like clearing memory on creation - thought that was covered, but then I realized No, of course not... and had to fix that. Then there's the locking scheme - mutexes are great, but it's nice that Java allows the same thread to "relock" a section of code without blocking. I'm guessing that I could probably do the same thing on the CKit's mutexes, but it might be a bit dodgey, and so I may just hold off on that idea.

The Java class was 2442 lines, and the C++ was over 3700. Lots of comments, or course, but still... that's a ton of coding for a week. The test apps were another 390+ lines each - so I was plenty tired of typing by the time I got the C++ version ported along with the test app.

I'm glad it's done and all checked in.

As an aside, this all started from a problem a developer came and talked to me about last Friday. Since then, his initial work-around failed for one condition and others put in their $0.02 to say that they didn't think it would be that hard to do. Silly people... If I spent a week on this then it's hard. Impossible? no. But it's a lot harder to get right than most people think. I know it was a lot harder than I thought it would be. I figured a few hours for the Java version and then a few more for the port. I should have thought "days" as opposed to "hours". Glad it's done.

Well... this is a pain in the neck, but I'm glad it's solved. The problem is that I was trying to rebuild CKit on my Mac using make and Xcode 2.4.1. It compiled fine, the test programs linked fine against the generated .dylib dynamic shared libraries, but when I tried to run it I got:

    dyld: Library not loaded: /var/tmp//ccr3sui.out
    Referenced from: <path_to_test_app>/test
    Reason: image not found
    Trace/BPT trap

The problem was that the 'tmp' file wasn't there. When I did an otool -L on the application I got:

    test:
      /var/tmp//ccr3sui.out (compatibility version 0.0.0, current version 0.0.0)
      /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.3.3)
      /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.4.0)
      /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

When I looked at the shared library that I was testing, it had the same reference to this temporary file. I was stumped and blown away. I googled a lot of things and then found a reference that had a similar problem. The solution was amazingly simple and yet it should have been taken care of my Apple's compiler.

The link phase of the build of a dynamic shared library allows for the addition of a few options that shouldn't really matter unless you want to take advantage of the unique features of Mac OS X. They are:

    -install_name libCKit.dylib
    -current_version 1.0.0
    -compatibility_version 1.0.0

where you can make the versions anything you want, but the key to this riddle is the -install_name. By default, the linker is setting it to the temporary output of the link, and not the name in the -o parameter. The docs say this is the default for the install name, but it's not. When you use the same value for the -o option and the -install_name you'll see that the output of otool -L changes to be:

    test:
      libCKit.dylib (compatibility version 1.0.0, current version 1.0.0)
      /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.3.3)
      /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.4.0)
      /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

We can now clearly see the difference. As long as you have this library in the path specified by DYLD_LIBRARY_PATH, you're good to go.

I spent hours on this. I was trying all kinds of things to see where these temp files were. It's a stroke of luck that I found the web site with the reference that I needed. Yikes. Well... now it's here and maybe it'll help the next poor sap that's got the same type of problem.

For the last several days I've been heads down coding this 'infinite' precision floating point number in Java for BKit - BKFloat. I learned a lot of really interesting things in the process. Now that I've dug myself out of that all-encompassing task, I can take a little time to talk about it.

When I started working on it I thought that the best way to implement the class was to have a long as the whole number part and another as the fractional part. Then, when I needed to do any math, it was pretty easy as I could take advantage of the long's ability to do the math. And this got me quite a ways to the end. But I started running into a lot of problems when I got to the point of really building the add() and subtract() methods because I was getting into coding based on the long and not a general 'infinite' precision floating point number.

For instance, with the long, I still had to deal with the fact that I did have an upper limit on the number of digits I could represent. Sure, it was big, but it wasn't as big as it might need to be. Some of my test cases had numbers in scientific notation and for those guys I had 5.5511232344325E-17 and the like which made it very hard to make sure that I had enough digits to express the non-zero elements as well as the proper magnitude of the number.

The final problem that snapped this design was the use of the sign on the fractional part. Imagine that you had two longs - one for the whole number part and another for the fraction with an implied decimal point in between them. If I had a number like -0.5 the whole number part would be 0 and the fractional part would be 5 - but where did the sign go? If you had -1.5, the whole number part would be -1 - there's the sign. So I had to 'pack' the sign on the fractional part if the whole number part were zero. This lead to a lot of code to make sure I had the right sign of the number for the operation. It was looking ugly and I just knew there was a better way.

So after about a day of that I backed off and thought that the better way had to include the resizing of the digit storage in order to make sure that the only limitation to the size of the floating point number was the capacity of the machine. I also had a feeling that by separating the digits I'd be able to implement the arithmetic operations a lot easier because I could code it like third-grade math. So I started off on that tack.

The next day was spent gutting the code of the long-based code in factor of byte[] storage for the digits. I spent a little more than half a day getting to basically the same point that took me the previous day. I had added a lot of little things like the ability to shift the number right or left as if multiplying (or dividing) by 10. This made a lot of things easier and in general the code was drastically simplified.

The things we learn in third-grade are really powerful. Coding carry and borrow on the add() method was interesting and a bit of amazement at what we all do so easily. It's really quite amazing. I've done the classic computer numerical manipulation with the XOR for multiplication and the full adder, but this is interesting in that it wasn't base 2 and yet it was exactly a digit at a time. Very interesting. I don't think I've ever written code like it.

Multiplication was interesting and fun at the same. Division was almost fun once I got into it. Of course, the division was the first one that might have a loss of precision due to the mature of the operation so I had to do a few interesting things there, but in the end it was working remarkably well. I don't think this is going to set any speed records, but the point is not speed but precision and accuracy. The test cases I used in the development pointed out that there were plenty of times where a simple double in Java is just not very good at representing values. This class is for those times when you have a handful of numbers that have to be added, etc. without any loss of precision. For those cases, speed is typically not as important as the precision. Good enough.

Now I'm back down into the hole to convert it to C++ for CKit. Shouldn't take too long... all the logic and method calls are already worked out.

Friday, a developer stopped by and talked about the problem of dealing with the precision problems in Java and C++. I thought about it a lot over the weekend and decided that I wanted to code something up for BKit that would be a general purpose object that could hold these floating point numbers without loss of precision due to representation or operation.

I think the best way to go about this is to have a long for the whole number part and a long for the fractional part. This way, it's very easy to work on the parts, but they will be able to hold a very large number.

It's off to coding...

UPDATE: after spending the day on this approach, I've decided that it's not the best. I'm going to start over tomorrow morning with byte[] where each byte is a digit in the two parts of the floating point number. The reason for this is simple - the longs were giving me limitations that I needed to deal with and it was getting very much code for the sake of the data storage choices. So I think I'll be slower, but much better with a more general storage like byte[]

Today another developer stopped by with a problem I've seen several times before - how to handle user validation of percentages. Specifically, let's say a user needs to put in how to subdivide something - 10% to this, 20% to that, and 70% to the other. Breakdowns like this are easy - until they get into factional percentages and numbers that are, by their very nature, impossible to represent exactly in a computer.

Take 0.12 - it seems easy enough to look at, but try to add it to 0.74 and 0.14 and you're going to find that a double is not exactly 1.0. That's because the numbers are not exactly what we typed in and there are representational errors in the doubles that make it something like 1.00000001. So how to fix this?

One way, which I've done for many cases, is to have an ε something like 10^-6 and check to see if the value you want is some ε from the target value by:

    if (Math.abs(sum - target) <= 1.0e-6) {
        // close enough to be considered equal
    } else {
        // not equal
    }

which can be put into a method or function that makes it easy to call. But the idea is that if you get close enough, you're equal. The problem with this is that the ε that you should use is greatly dependent on the values you're summing, in this example. Say you have values like 10, 20, and 70 - in this case, ε should be 0.1 because there's no need to have it any smaller as all the numbers are integers. But if you had numbers like 10, 20.22, and 69.78 then you may need to have ε set to 0.001. The optimal value of ε seems to be directly related to the maximum precision of the data. Also, there's always the possibility that some oddball case will be just large enough a difference to not be 'equal' and then you're flagging a false positive. But there is an interesting way that this won't happen.

Look at the problem of the user input as if you were a third-grader. Break the problem down into the whole number and fractional components. This makes everything an integer, and adding integers is very easy indeed - with no rounding problems. What I mean to say is that if you had the set of numbers: 10, 20.22, and 69.78 then break the summation down into 10 + 20 + 69 + (00 + 22 + 78)/100.0. This means that you only have to look at the string representation of the numbers - separate out the parts to the left and right of the decimal, find the maximum precision to the right of the decimal (for scaling), and then go to town.

After we talked about this approach for a little bit, the developer went back and coded it up in no time. It's really very simple. But given that users can type in arbitrary precision in the numbers, it's important to sum them as exactly as possible, and using integers for the whole and fractional parts is really pretty sweet. There's no rounding (until you get to the final double) and by then, you're comparing it to the test value, and if it's another integer (in the case of percentages), then it'll be easy.

I giggled a bit thinking about this. It's not often that the best way of doing something is the way a third-grader would do it. But that's the point - not everything needs to be so overly complex and over-designed.