What's it all about, Alfie?

In my work today on updating the valuation library to a current production version, I once again came across Boost smart pointers. Now there's a lot of good things in Boost, and I have barely scratched the surface of what Boost has to offer, but the implementations I've seen that use smart pointers are much more confusing than they are helpful. After all, the point is to make it easier to write code - not harder.

Well, I was chatting with a good friend about this today after the fact and he uses Boost's smart pointers a lot and doesn't see the confusion. I can agree that if I were using them all the time I'd probably be desensitized to them as well. But I'm not yet used to Boost's smart pointers, and honestly prefer to handle the memory management myself - that's one of the reasons I'm using C++ in the first place - careful resource management. But I see his points, and because I found the conversation very interesting, here's the gist of it.

Take the following little code snippet:

    typedef EQS::Shared<Operation>::Ptr OperationPtr;
    typedef EQS::Shared<Results>::Ptr ResultsPtr;
    ...
    OperationPtr    lOperation = lFactory.getOperation(mType);
    ResultsPtr      lResults = lOperation->execute();
    double          value = lResults->getValue();

While it's perfectly legal code, if you separate the typedefs into the header file and the remaining three lines into an implementation file you have something that's confusing to a traditional C/C++ developer. Are lOperation and lResults pointers or not? Well... in reality, they are and they aren't.

They are in the sense that you can use the "->" to access methods, but in another sense, they are removed when they go out of scope in the code - like traditional stack variables. While this might be seen as a benefit by some, to me it makes the code exceptionally confusing.

In my opinion the line is being drawn too finely. Make theme appear much different from pointers and the confusion goes away. For example, the following code looks odd, and different, but there's no possibility that someone will be confused by the pointer-nature of the "->":

    typedef EQS::Shared<Operation>::Ptr OperationPtr;
    typedef EQS::Shared<Results>::Ptr ResultsPtr;
    ...
    OperationPtr    lOperation = lFactory.getOperation(mType);
    ResultsPtr      lResults = lOperation..execute();
    double          value = lResults..getValue();

or, as if taking a page from Objective-C, this:

    OperationPtr    lOperation = lFactory.getOperation(mType);
    ResultsPtr      lResults = lOperation[execute()];
    double          value = lResults[getValue()];

In both cases, there is a distinct visual difference in how the objects are being used. In these examples, this is silly and trivial, but in large code sections where there is a lot of processing and the usage of the "->" implies traditional pointer, it's confusing.

I know they did this to make them seem as close to real pointers as possible and allow for the auto-CG, but I think they'd be better off - as C++ was with references, in making a new language element. And if they can't have that, then at least overload a different operator - or something to make it looks significantly different from traditional pointers.

I guess my point is that by using smart pointers you'd never think twice about doing this:

    SmartPtr    p = foo.bar();
    p->goof();

but you'd never do this:

    char        p = foo.bar();
    p->goof();

and while you could do this:

    typedef char *CharPtr;
    ...
    CharPtr     p = foo.bar();
    p->goof();

why would you?

Consistency... that's an important part of a language to me. Perl is nice, and yet it's strength is also it's weakness - you can do the same thing a million different ways. But almost all my Perl code does the same thing the same way - just for that consistency.

The conversation has continued and my friend has pointed out that there is a significant historical component to this that I wasn't aware of, and it sheds light on why they are used this way:

fair enough, call it something else, but, history helps shed light here. In the early days of c++, you didn't have any such thing. Then sometime after the first standard release, they said, hey, let's address a simple memory leak issue and added auto_ptr to the stl. Now auto_ptr seems like a reasonable name and they designed it so that you had almost no coding impact to replace your regular c-style pointers with auto_ptr. You could imagine the resistence they would get if you had to re-write huge chunks of code to migrate over to it. Instead, you had to change your type declarations and remove some deletes and that was basically it.

However, people started to realize that std::auto_ptr wasn't so good for various reasons and the boost guys came along and introduced a richer set of pointer objects that kept the same semantics so that it was easy to migrate.

I can see his point, but when he pressed me about what I thought C++ should do I said that they needed to add a fourth elemental data type:

• value
• pointer
• reference
• smart pointer

and that smart pointer is going to have a different decorator for accessing it's methods and ivars. By making it too close to a pointer, they have in effect made it more confusing.

Where Java Got it Right

After thinking about this overnight, I have to admit that this is where Java got it right. If they had wanted to add in a reference-counted dynamic object in C++, they should have created smart references. Had they done that, it would have allowed for the 'new' without the need for a 'delete'... it would have used the '.' as the method invocator, and it would have allowed for the complete absence of the '*' in definitions. In fact, I suppose that the only thing they are missing now is the use of '->' when '.' would be better. Then, simply call them smart references and the confusion is gone.

You can still have them hold a NULL, unlike traditional C++ references, but that would be the edge condition, as it is in Java, and not the typical use of the datatype. Yup, I have to say that they'd be so much better off in my mind if they went for 'references' as opposed to 'pointers'.

Today I took on a project that I was convinced would take me months - the upgrading of our calculation library framework from a very old version (more than 24 months) to a very recent version (in production systems now). I was convinced that going from a v6.x.x to a v7.x.x version was going to be a pain, and plenty of people didn't say I was wrong. But I was very wrong.

It took me about 2.5 hours from start to finish to update the calls and the use of Boost smart-pointers from regular pointers and clean up the code a bit. Amazing! I had no idea it'd be this easy or I'd have done it months ago. Easily. But that just goes to show you that you don't really know how hard something will be until you really roll up your sleeves and get into it.

With this, we'll be able to add in several new features that the traders have been wanting to see, and in fact have just been waiting to use because the older version of the valuation library constantly generated a ton of SegFaults when I gave it the skew and kurtosis curves for the instruments we had. I was able to turn that feature on in the code and it works beautifully. That's nice.

So today has been a very good day for me. Wonderful, in fact.

I know I've said it before, but one of the most important things I think a professional developer can do is to program defensively. There's no reason not to - it doesn't take that many CPU cycles to check inputs and return values, and the benefits are almost incalculable when you need them. Case in point was a problem I ran across yesterday afternoon and took until this morning to clear up.

Several years ago, a co-worker put together a nice object representation of the instrument data sitting in the MarketMash server. He created this because the current model we were using, and still use to this day, is very amorphous, and not well defined. Nothing like you'd expect to see if you were a traditional java developer. So he decided to make a new object model that would sit beside the existing model and provide a lot more structure to the data and therefore make it a lot easier for people to get at the data they are interested in.

The problem with a lot of structure is as things evolve, you have to maintain that structure a lot more actively than the amorphous model. This came to pass yesterday as I was looking at the output logs of one of the processes of a system of mine. I had put into CloudCity the ability to handle logging of new instrument-level tags once per instance, and then simply ignoring them after that. What I hadn't done was to do the same for the position-level tags, and that's what got me.

Well... what really got me was optimistic programming. Such might be the case as the example:

    CCToDNAMap  dnaMap = CCToDNAMap.getInstance();
    dnaPositionTag = dnaMap.pairFromTag(tag).getName();

In my problem yesterday, it was the call to pairFromTag() that was returning a null and not getting checked for. The additionally odd problem was that this was not throwing a NullPointerException as you'd probably expect. In fact, it was throwing an Exception, but that exception was null. When I broke out the calls and tested every return value, things started working just fine.

I then took the time to add in the new fields, but the real issue was the optimistic programming. For if I add another position-level attribute now, it'll log it once and then continue to work as if nothing bad happened. That's what I wanted to happen in the first place.

I guess this really isn't the fault of the tech salesforce, but it's the illusion that they are really providing something in the transaction. Oh, sure... there are some turn-key solutions where all we really need to do is to buy the hardware - speced out to the last detail, get the install disks from the vendor and that's basically it. But those solutions are really few and far between. Most big systems have to interface with systems in-place in your Shop so it's not a plug-n-play solution. Which is the supposed reason for the technical sales force (aka consultants) from the vendor.

Here's what happened... we have asked the vendor to be on the next version of their product - something that's a complete re-architecture from the currently shipping version. Totally different. But in a way, that's good. The existing system was a mess and could not possibly do what we need, and the new system may be new and green, but at least it's got a chance of being the 'right thing'. So we need some hardware to run this beast on, and I put together a list of a minimal set-up to get us going. I know it's probably not enough, but it's close and has a decent chance of being all we need for production. I send this to my manager, explaining why I've chosen this configuration, etc. and said "Approve this and we get moving". One of the guys who's not above me in the org chart, but has fewer boxes to the Big Boss said to run this by the tech sales person and see if it's OK.

Normally, I'd say "Great!", in fact, I would not have made the list myself but asked them for a list. But that's not really possible now. There's a totally new architecture that they've never delivered to any client, doing things that they never did before. How do they know what our situation is going to take? They don't. But even if they did... my guess was 'reasonable' - let's say the knew what it'd take. If it was less than this I'd be very surprised because the system we have now doing this task is using this same hardware and only doing a fraction of the job this system will be required to do. So doing more for less? Hmmm... then there's the fact that the existing system is all in C++ and this has significant parts in a scripting language. While I'm not saying it's slow, I am saying I'd be very surprised if it were faster than what we have now.

So let's say it's slower, and they know it. Are they going to say "Hey, this is only about half of what you need." Not a chance. We'd say "Half!? What's wrong with this thing?" So they'd keep it to themselves and realize that this will get us started, and as we get into the project we'll see that it's not enough and order more.

So all this asking them is a wasted exercise. We're not going to get any useful information from them. The product is like nothing they've ever delivered before - totally new. They're guessing just like we are. But by asking them we're delaying doing anything on the project for several days to get this useless bit of information. When you're putting something together for the first time you can't expect cookie-cutter answers. You have to use your experience, understanding of the problem, and goals and stick a finger in the air and guess.

If you're good, then your guess is pretty darn good, and you go from there. Asking someone else doesn't increase the confidence in your guess. Just do it.

Like a lot of places, it seems, we have gone through our fair share of people. Most don't fit, and detect this and get another job. Many are happy moving on, while some find that they really had it a lot better here than at the new place. Grass being greener syndrome. Every now and then there's a person that is asked to leave. While there are just a few basic reasons: not enough work, not enough money, or bad fit, there are a few in that last category that have either become people that would not have been hired, or were such consummate con artists that they were always this way and just hid it well enough in the interviewing process to get in the door. These last group of people really don't last very long, but the mutators are an interesting breed in their final days.

Such is the time I find myself in this week. One of the developers has decided to leave - right ahead of being asked to leave. I have no idea where he's going, nor do I really care, but it does amaze me to think how he could have gotten a job with the attitude he has displayed here for the last year or so. Understandably, people respond to their environments differently. And if there's a hostile environment there is always the fight or flight response. But there's more to it than that as we're supposed to be evolved beings.

If you find yourself in a job that you really don't like - change something. Change the way you do your job, change the way to approach your job, change the hours you do your job, change something to try and make it better - or change your job. Don't sit and complain all day about your situation... no one wants to listen after a bit. Don't complain that they aren't giving you enough advancement opportunities - go out and make some, and if you can't, then find some place that you can.

If you can't change anything about your job, and you can't find another, then you're in a pickle because it probably means that you shouldn't have been hired for the job you have, and need to think about another line of work. Write a book, change careers, do something.

Alas, the developer who is leaving here is most likely doing none of these things. He's probably interviewed just well enough to get a job based on the business knowledge he's gained while being here. His work ethic is bad - I can't remember him coming in regularly before the market open. His development skills are very marginal - I can't think of one compliment he's gained from a fellow developer in all the time he's been here. Most importantly, he's a complainer. It's just hard to work with guys like this for a long time. It's draining.

So rather than finish up his two weeks with a little bit of class, he's taken to coming in even later, doing even less, and when given the chance to get out of the last few days of his last week, chooses to put off doing things which would allow him to have a little time off between jobs. It's really almost amazing. He's not even actively leaving - one more thing he's not very good at.

People at his new workplace are probably going to see this and figure out what we have figured out. Sure, he may work harder because he likes the environment there better than here, but I have a feeling that in the end, it's more about him, than his environment. You either are a good worker, or your not. It doesn't matter what job you have, you either do it with some sense of quality and conviction, or you don't. And if you don't, no job is ever going to stop you from complaining.

I've been working with SubEthaEdit a lot in the last few days, and also with BBEdit, trying to get the few features that I don't like fixed/cleared up, and while I really like both of these editors for their unique approach to the problem of a developer's editor, there is a lot to be said for Vim. I have done more coding in Vim than probably any other editor in my life. I've used it on so many systems it's amazing, and I've never had to compile it myself. There seems to be someone who's already figured out how to get it to work on every platform I've needed. The community support is really amazing.

I have been using the Vim package from macvim.org, and it's currently at 7.0.224 for Mac OS 10.4 (Tiger). I am able to use all the nice fonts, syntax highlighting, everything that I expect in Vim, just on my Mac. The only problem is that it's stuck in Vim's view of the world. That means you can't have multiple windows open on the same application instance. On Unix this isn't an issue because you typically start a Vim session from the command line, but on Mac OS X, it's a limitation that I find... well... annoying, and it makes me not use it as much as I might.

There are also several things that the other two do that are very nice that Vim misses out on. It's not horrible, but when given the choice, I really end up using SubEthaEdit and BBEdit more on my Mac than I do Vim. But it's close. Many of the requests I have in to the SubEthaEdit developers and BBEdit developers are things in Vim that I wish they'd do - or do better.

Still... I can't complain. Vim has been very very good to me and I hope they come out with a Leopard release (Mac OS 10.5) very soon so I can upgrade my laptop to it and keep Vim with me.

This week marks the time when we have to make final preparations for going off Daylight Savings Time (DST) which is important because Hong Kong does not observe DST, and so we're going to loose an hour at the end of our day and the start of theirs. This is important for trading and risk apps like mine. The original solution was to make my app run 24x5 without any interruptions. This was a significant technical challenge and after about a day or two of this, I decided that it'd be better to make use of extra hardware to have two servers - one that covered the US and Europe, and another that covered the Far East.

Problem was, this meant that there was going to be a lot more maintenance of the system to keep two systems up and in sync. After hearing me say this, most said "Yeah, yeah" nodding their heads, but thought I really didn't understand the situation. When I came back with a list of things that would have to be done each evening to make this work, all of a sudden the easier solution to the problem appeared.

The problem really comes down to the fact that we typically didn't start the nightly processing until about 5:30 pm, Chicago time. This was 6:30 am Hong Kong time, and after the DST change, it'd be 7:30 am Hong Kong time. Given that the start is then, it's likely that we are not up and going by 8:00 am local time in Hong Kong, and that's the problem. So I had suggested that we move up the end-of-day processing, but that got nowhere. Then they saw what they'd have to do to support the two servers capable of giving Hong Kong the data they needed when they needed it.

All of a sudden, the end-of-day processing was moved up to 4:15 pm Chicago time. Plenty of time before Hong Kong would need it.

While I'm a little frustrated by the fact that I wasn't listened to earlier, I have to take solace in the fact that eventually my ideas found ears, and things started to move in the right direction. Now the second hardware is for disaster/recovery and there's only one server covering the globe. It's simpler, and while there are bound to be bumps in the road, things are going to be better by moving up the end-of-day processing.

That first bump was last night when I got a call on the train. It was from someone asking if there was a way to edit the data that's sitting in the middle-tier of the application. I said "Nope, it flows from the server, and that's already thinking it's tomorrow." I knew what they wanted to hear, but it wasn't going to happen. The data in the middle-tier is too complex to simply edit with something like a text editor. The values are interrelated and computationally intensive calculations need to be done in order to derive some values from others. It's not easy. Which is why there's no easy editor of that data. But because of the early restart, the guys didn't check on the state of the system early enough in the afternoon to make sure that it would be good to go for the earlier restart.

Lessons learned. Like I said... a few bumps in the road, but it's a better place we're going to.

Whew!

It's been a week.

Pretty much all this week I've been heads-down coding this 'infinite' precision floating point number first in Java and then in C++. There were a lot more issues to deal with than I had expected to run into in this little class - face it, it holds a number, adds, subtracts, multiplies, and divides. That's it! But, boy-o-boy, there's a lot more to just that than it sounds.

I've talked about a few of the issues in the post I made after the Java version was done. But then the conversion was pretty easy for many things, but had quite a few gotchas in the C++ codebase. Little things like clearing memory on creation - thought that was covered, but then I realized No, of course not... and had to fix that. Then there's the locking scheme - mutexes are great, but it's nice that Java allows the same thread to "relock" a section of code without blocking. I'm guessing that I could probably do the same thing on the CKit's mutexes, but it might be a bit dodgey, and so I may just hold off on that idea.

The Java class was 2442 lines, and the C++ was over 3700. Lots of comments, or course, but still... that's a ton of coding for a week. The test apps were another 390+ lines each - so I was plenty tired of typing by the time I got the C++ version ported along with the test app.

I'm glad it's done and all checked in.

As an aside, this all started from a problem a developer came and talked to me about last Friday. Since then, his initial work-around failed for one condition and others put in their $0.02 to say that they didn't think it would be that hard to do. Silly people... If I spent a week on this then it's hard. Impossible? no. But it's a lot harder to get right than most people think. I know it was a lot harder than I thought it would be. I figured a few hours for the Java version and then a few more for the port. I should have thought "days" as opposed to "hours". Glad it's done.

Well... this is a pain in the neck, but I'm glad it's solved. The problem is that I was trying to rebuild CKit on my Mac using make and Xcode 2.4.1. It compiled fine, the test programs linked fine against the generated .dylib dynamic shared libraries, but when I tried to run it I got:

    dyld: Library not loaded: /var/tmp//ccr3sui.out
    Referenced from: <path_to_test_app>/test
    Reason: image not found
    Trace/BPT trap

The problem was that the 'tmp' file wasn't there. When I did an otool -L on the application I got:

    test:
      /var/tmp//ccr3sui.out (compatibility version 0.0.0, current version 0.0.0)
      /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.3.3)
      /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.4.0)
      /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

When I looked at the shared library that I was testing, it had the same reference to this temporary file. I was stumped and blown away. I googled a lot of things and then found a reference that had a similar problem. The solution was amazingly simple and yet it should have been taken care of my Apple's compiler.

The link phase of the build of a dynamic shared library allows for the addition of a few options that shouldn't really matter unless you want to take advantage of the unique features of Mac OS X. They are:

    -install_name libCKit.dylib
    -current_version 1.0.0
    -compatibility_version 1.0.0

where you can make the versions anything you want, but the key to this riddle is the -install_name. By default, the linker is setting it to the temporary output of the link, and not the name in the -o parameter. The docs say this is the default for the install name, but it's not. When you use the same value for the -o option and the -install_name you'll see that the output of otool -L changes to be:

    test:
      libCKit.dylib (compatibility version 1.0.0, current version 1.0.0)
      /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.3.3)
      /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.4.0)
      /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

We can now clearly see the difference. As long as you have this library in the path specified by DYLD_LIBRARY_PATH, you're good to go.

I spent hours on this. I was trying all kinds of things to see where these temp files were. It's a stroke of luck that I found the web site with the reference that I needed. Yikes. Well... now it's here and maybe it'll help the next poor sap that's got the same type of problem.

For the last several days I've been heads down coding this 'infinite' precision floating point number in Java for BKit - BKFloat. I learned a lot of really interesting things in the process. Now that I've dug myself out of that all-encompassing task, I can take a little time to talk about it.

When I started working on it I thought that the best way to implement the class was to have a long as the whole number part and another as the fractional part. Then, when I needed to do any math, it was pretty easy as I could take advantage of the long's ability to do the math. And this got me quite a ways to the end. But I started running into a lot of problems when I got to the point of really building the add() and subtract() methods because I was getting into coding based on the long and not a general 'infinite' precision floating point number.

For instance, with the long, I still had to deal with the fact that I did have an upper limit on the number of digits I could represent. Sure, it was big, but it wasn't as big as it might need to be. Some of my test cases had numbers in scientific notation and for those guys I had 5.5511232344325E-17 and the like which made it very hard to make sure that I had enough digits to express the non-zero elements as well as the proper magnitude of the number.

The final problem that snapped this design was the use of the sign on the fractional part. Imagine that you had two longs - one for the whole number part and another for the fraction with an implied decimal point in between them. If I had a number like -0.5 the whole number part would be 0 and the fractional part would be 5 - but where did the sign go? If you had -1.5, the whole number part would be -1 - there's the sign. So I had to 'pack' the sign on the fractional part if the whole number part were zero. This lead to a lot of code to make sure I had the right sign of the number for the operation. It was looking ugly and I just knew there was a better way.

So after about a day of that I backed off and thought that the better way had to include the resizing of the digit storage in order to make sure that the only limitation to the size of the floating point number was the capacity of the machine. I also had a feeling that by separating the digits I'd be able to implement the arithmetic operations a lot easier because I could code it like third-grade math. So I started off on that tack.

The next day was spent gutting the code of the long-based code in factor of byte[] storage for the digits. I spent a little more than half a day getting to basically the same point that took me the previous day. I had added a lot of little things like the ability to shift the number right or left as if multiplying (or dividing) by 10. This made a lot of things easier and in general the code was drastically simplified.

The things we learn in third-grade are really powerful. Coding carry and borrow on the add() method was interesting and a bit of amazement at what we all do so easily. It's really quite amazing. I've done the classic computer numerical manipulation with the XOR for multiplication and the full adder, but this is interesting in that it wasn't base 2 and yet it was exactly a digit at a time. Very interesting. I don't think I've ever written code like it.

Multiplication was interesting and fun at the same. Division was almost fun once I got into it. Of course, the division was the first one that might have a loss of precision due to the mature of the operation so I had to do a few interesting things there, but in the end it was working remarkably well. I don't think this is going to set any speed records, but the point is not speed but precision and accuracy. The test cases I used in the development pointed out that there were plenty of times where a simple double in Java is just not very good at representing values. This class is for those times when you have a handful of numbers that have to be added, etc. without any loss of precision. For those cases, speed is typically not as important as the precision. Good enough.

Now I'm back down into the hole to convert it to C++ for CKit. Shouldn't take too long... all the logic and method calls are already worked out.