What's it all about, Alfie?

Today we had a meeting with a group of technical sales consultants from a Vendor for a very expensive package that we have purchased and I've been asked to integrate and get working. This meeting was about price feeds into this system, and how the current product's abilities are weak to say the least. They rely on Reuters exclusively, but with a Reuters infrastructure, you (the client of Reuters) can push data back into the RMDS feed, if you so choose. You just become another 'exchange' to Reuters. But that's not a realistic alternative for us.

What we need to do is to get at the Vendor's price feed service and inject prices from my CacheStation based on our rules. These prices are what's necessary - as opposed to what's possible from Reuters, and so I'm not really in favor of simply faking Reuters RMDS with our source - I want a new source that can co-exist with the Reuters feed and then we can configure the application to take certain prices from one feed and other prices from another. Sounds very reasonable.

What's amazing to me is that in this meeting we had close to a dozen people - of which two were really necessary - the lead Vendor representative and myself. Everyone else was at least redundant, and often (as it turned out), painfully in the way. I've worked with the feeds, I know what needs to be done, I understand the basics of the Vendor's price service, and all I need to do is to get them to do a few things to meet me half-way on this and we'll all be fine.

But as it turned out, people that had no idea what was needed or what was happening had to be in this meeting. Why? Politics. What a waste. This is something I hate - don't waste my time if you're going to have people unrelated to the issue blathering on about this or that... get to the point, or confess that you have no idea what is going on and get on with it.

In the end, you'll come back to me - or you won't. If you don't, then you got it figured out without me - Good for you! If you come back to me (as you have time and again), then it's because you have no clue about what to do, and you want me to Just do it.

And it seems this insanity - I can think of no better term for it, is going to continue before there is any hope that it will get better. Tomorrow we're having a meeting about the first hardware purchase to run a version that they've never deployed and don't even have finished. How can they possibly know? It's a guess, so let's guess and move on. Chances are, we're not going to underspend - this is a massive system and it's going to need more than we're throwing at the problem now. But they have to have these meetings with people that have no real clue about what they are doing. It's amazing.

Someday this will end... it'll end the day that they realize that this has to work, or it's a waste of millions of dollars. When that day comes, they have traditionally come to me and said 'Make it work!', and at that point, the meetings stop. Unfortunately, that's typically ridiculously close to the drop-dead date for implementing this thing, so I'm under a lot of pressure to get it working in a very short time.

But that's the day it'll end. I just wish I could sit all this out until that day. It's the kind of thing that gives me a headache - and reminds me why I don't sit in the manager/owner chair any more. It's just no fun.

I was sitting here today working on the bulk push() and pop() for the BKit queues and a fellow developer stopped by to inform me that I wasn't using the best performing locking schemes in my implementations of the queues. While I accept that there are possibly ways to squeeze a few percent out of mine in certain conditions, these aren't custom-designed queues, they are general purpose utility classes that can't make assumptions about their use-cases in their implementations.

But that wasn't at all clear to this developer that stopped by my cube.

And that wasn't the only thing he didn't get.

Good Java coders are first and foremost good developers. Period. That means they understand how the code is executed in the machine. Where things might seem to be unimportant, or for that matter, important, and where they really are.

Case in point about high performance queues. If you're in a process where you have many threads processing data off a single queue, and the processing time is small, then you are guaranteed to have locking contention on the queue - by definition. There's no way around it. The solution I've seen time and again is to remove things from the queue in bulk and then the processing time goes up and the locking contention goes down because the number of threads hitting the queue at the same time goes way down.

But to this developer, the answer was Java 5's optimistic locking. Yes, folks, assume you'll get the lock and then fail and retry if you don't. Well, that's a wonderful idea in a situation that's already got locking contention problems. What was he thinking? The answer was, he simply wasn't. He was thinking that Java's way of doing things was so easy that under the covers it really didn't have to do any low-level locking for the push() or the pop().

So I had to walk him through it. It finally dawned on him that his statements were crud, and when he did, I could see him deflate right in front of my eyes. But for those 15 minutes when he was sure I was missing the point I wanted to yell at him Are you a total idiot? Don't you know any language other than Java? And folks, unfortunately, the answer to that is No. If you don't have any more experience in developing than Java, it's hard to see that it's not the end-all-be-all.

Today I was helping out a developer refactor his code so that it would be more CPU intensive for a shorter time as opposed to "tailing off" as it ran. The problem is this: his app needed to read a large amount of data (30,000 to 50,000 records) per file for several files. But not all files had the same length. This meant that his original design where h file processor was a thread, had the limitation that as some threads finished, 'less' of the machine was dedicated to the task. He wanted to fix that.

The easiest way to fix that is to have all processing go through a single queue - as opposed to a queue per file. This is not without it's limitations, however, as now this queue will become the bottleneck as synchronization on it to push() and pop() items will be under contention by all threads. The refactoring had a few issues, but the real problem remained - the queue operations needed to be done in bulk.

So, I added the ability to push a Collection of objects onto each queue (LIFO and FIFO), as well as popping off a bunch of values (up to a provided limit). This is the only way to achieve balance between the processing threads and the locking on the queue. The end result is that he now has another parameter to tune - the batch size of the pops. If it's 50 to 100 he should be in fine shape and not have to worry about performance hits due to excessive locking contention, but I'll let him fiddle with that and find the optimal value for his process.

Today I ran into a problem that's somewhat sensitive to me. Because I'm sitting in this cube, and could be doing a lot of other things, I realize that there are certain trade-offs I've made to be here. I'm not in a managerial role, but had I wanted to stay in one, teaching or owning my own company would have afforded me all the opportunity I wanted for that. No... I didn't want that. I like doing, not deciding for a living.

But every now and then there comes along a situation that makes me rather upset with the people I work for. I'm capable of doing the management, and as such, require very little management from above. My management appreciates this, and when it suits them, take full advantage of it.

Today I wanted to get licenses for Tibco for several projects we're working on. This is a replacement of either free or home-grown messaging systems, so there's no obvious feature that this will bring that we don't have now. Yes, it'll be soft factors - industry standard, higher performance, scalability, etc. but today - right now, there's no problem that this addresses. Yet it's a $100,000 expense. Too much for casual spending.

So rather than trust me that it's needed, and will be useful, they want me to make the drawings and slides to show how it'll fit into the next phase of our architecture. If we had run into a problem, they'd never want to see this... they'd want to know what fixes it. But because there's no immediate problem to solve, it's not as important as other things.

Understood... then let's just drop it. I'll wait until it's critical again and then I'll remind you that I mentioned this and you'll say "Well... why didn't you... Oooo!" and we'll do it then. But now, I have to do this for a group of people that either really don't care, or really don't need to care. It's infrastructure stuff, folks. Why spell it out to the web developers how the trade processing will change with a new messaging system? They might care, but when they need to know, they'll ask.

I guess it's the fact that if they thought it was too expensive, just say so and we'll limp along until it's critical and then we'll say "Here's the solution", and get it then. Doing it this way is just part of the reason that I don't own my own company any more - I don't like doing this stuff. And for the vast majority of the time, they don't want me doing this kind of thing - they want me fixing problems or enhancing the products I'm responsible for.

But I have to go through the whole dog and pony show, and let them take it to the Governing Council and then have project charts made, etc. It's turned from a simple little replacement project that'd take a few days to something that will drag out for months. Yucch!

In my work today on updating the valuation library to a current production version, I once again came across Boost smart pointers. Now there's a lot of good things in Boost, and I have barely scratched the surface of what Boost has to offer, but the implementations I've seen that use smart pointers are much more confusing than they are helpful. After all, the point is to make it easier to write code - not harder.

Well, I was chatting with a good friend about this today after the fact and he uses Boost's smart pointers a lot and doesn't see the confusion. I can agree that if I were using them all the time I'd probably be desensitized to them as well. But I'm not yet used to Boost's smart pointers, and honestly prefer to handle the memory management myself - that's one of the reasons I'm using C++ in the first place - careful resource management. But I see his points, and because I found the conversation very interesting, here's the gist of it.

Take the following little code snippet:

    typedef EQS::Shared<Operation>::Ptr OperationPtr;
    typedef EQS::Shared<Results>::Ptr ResultsPtr;
    ...
    OperationPtr    lOperation = lFactory.getOperation(mType);
    ResultsPtr      lResults = lOperation->execute();
    double          value = lResults->getValue();

While it's perfectly legal code, if you separate the typedefs into the header file and the remaining three lines into an implementation file you have something that's confusing to a traditional C/C++ developer. Are lOperation and lResults pointers or not? Well... in reality, they are and they aren't.

They are in the sense that you can use the "->" to access methods, but in another sense, they are removed when they go out of scope in the code - like traditional stack variables. While this might be seen as a benefit by some, to me it makes the code exceptionally confusing.

In my opinion the line is being drawn too finely. Make theme appear much different from pointers and the confusion goes away. For example, the following code looks odd, and different, but there's no possibility that someone will be confused by the pointer-nature of the "->":

    typedef EQS::Shared<Operation>::Ptr OperationPtr;
    typedef EQS::Shared<Results>::Ptr ResultsPtr;
    ...
    OperationPtr    lOperation = lFactory.getOperation(mType);
    ResultsPtr      lResults = lOperation..execute();
    double          value = lResults..getValue();

or, as if taking a page from Objective-C, this:

    OperationPtr    lOperation = lFactory.getOperation(mType);
    ResultsPtr      lResults = lOperation[execute()];
    double          value = lResults[getValue()];

In both cases, there is a distinct visual difference in how the objects are being used. In these examples, this is silly and trivial, but in large code sections where there is a lot of processing and the usage of the "->" implies traditional pointer, it's confusing.

I know they did this to make them seem as close to real pointers as possible and allow for the auto-CG, but I think they'd be better off - as C++ was with references, in making a new language element. And if they can't have that, then at least overload a different operator - or something to make it looks significantly different from traditional pointers.

I guess my point is that by using smart pointers you'd never think twice about doing this:

    SmartPtr    p = foo.bar();
    p->goof();

but you'd never do this:

    char        p = foo.bar();
    p->goof();

and while you could do this:

    typedef char *CharPtr;
    ...
    CharPtr     p = foo.bar();
    p->goof();

why would you?

Consistency... that's an important part of a language to me. Perl is nice, and yet it's strength is also it's weakness - you can do the same thing a million different ways. But almost all my Perl code does the same thing the same way - just for that consistency.

The conversation has continued and my friend has pointed out that there is a significant historical component to this that I wasn't aware of, and it sheds light on why they are used this way:

fair enough, call it something else, but, history helps shed light here. In the early days of c++, you didn't have any such thing. Then sometime after the first standard release, they said, hey, let's address a simple memory leak issue and added auto_ptr to the stl. Now auto_ptr seems like a reasonable name and they designed it so that you had almost no coding impact to replace your regular c-style pointers with auto_ptr. You could imagine the resistence they would get if you had to re-write huge chunks of code to migrate over to it. Instead, you had to change your type declarations and remove some deletes and that was basically it.

However, people started to realize that std::auto_ptr wasn't so good for various reasons and the boost guys came along and introduced a richer set of pointer objects that kept the same semantics so that it was easy to migrate.

I can see his point, but when he pressed me about what I thought C++ should do I said that they needed to add a fourth elemental data type:

• value
• pointer
• reference
• smart pointer

and that smart pointer is going to have a different decorator for accessing it's methods and ivars. By making it too close to a pointer, they have in effect made it more confusing.

Where Java Got it Right

After thinking about this overnight, I have to admit that this is where Java got it right. If they had wanted to add in a reference-counted dynamic object in C++, they should have created smart references. Had they done that, it would have allowed for the 'new' without the need for a 'delete'... it would have used the '.' as the method invocator, and it would have allowed for the complete absence of the '*' in definitions. In fact, I suppose that the only thing they are missing now is the use of '->' when '.' would be better. Then, simply call them smart references and the confusion is gone.

You can still have them hold a NULL, unlike traditional C++ references, but that would be the edge condition, as it is in Java, and not the typical use of the datatype. Yup, I have to say that they'd be so much better off in my mind if they went for 'references' as opposed to 'pointers'.

Today I took on a project that I was convinced would take me months - the upgrading of our calculation library framework from a very old version (more than 24 months) to a very recent version (in production systems now). I was convinced that going from a v6.x.x to a v7.x.x version was going to be a pain, and plenty of people didn't say I was wrong. But I was very wrong.

It took me about 2.5 hours from start to finish to update the calls and the use of Boost smart-pointers from regular pointers and clean up the code a bit. Amazing! I had no idea it'd be this easy or I'd have done it months ago. Easily. But that just goes to show you that you don't really know how hard something will be until you really roll up your sleeves and get into it.

With this, we'll be able to add in several new features that the traders have been wanting to see, and in fact have just been waiting to use because the older version of the valuation library constantly generated a ton of SegFaults when I gave it the skew and kurtosis curves for the instruments we had. I was able to turn that feature on in the code and it works beautifully. That's nice.

So today has been a very good day for me. Wonderful, in fact.

I thought I had this all figured out, and to a point, I did. But today I noticed a bad conflict between FlySketch and Skitch when trying to do the Skitch 'selection grab' - Apple-Shift-5 hot key. There was a conflict between that and FlySketch's Ctrl-Apple-5 full screen hot key, but I had changed that, as I posted about a while back.

Surprise! It was back.

The problem was that FlySketch wasn't saving the change in it's Preferences. This meant that when I rebooted (like I have to do an update on the OS), I went back to the old mapping that had the conflict. I suppose that I could change the mapping in Skitch, but honestly, I'd like to have FlySketch set to what I wanted, and Skitch to stay where it is.

I asked the question on the FlySketch forum and we'll see what I get back. I'm hoping it's a simple fix - or maybe an update, but I'd hate to have to re-do this every time. I am somewhat forgetful in this regard.

UPDATE: I've decided that it's not worth the hassle, and if they fix it, that'd be great, but until they do, I'm moving Skitch's hot key to Apple-Ctrl-S to match the Apple-Ctrl-F for popping up FlySketch. There's a symmetry there that I can live with.

[11/8/07] UPDATE: I filed a bug with FlyingMeat on this. Turns out it's more likely to be acted upon in this manner as it's email-based and not the forums. We'll see if he responds today.

[11/8/07] UPDATE: I sent along the Console message in response to Gus' reply to my bug that it should, in fact, be stored in the Preferences. It was:

2007-11-08 13:34:14.322 FlySketch[2565] CFLog (0):
    CFPropertyListCreateFromXMLData(): plist parse failed; the data is not proper UTF-8.
    The file name for this data could be:
    /Users/drbob/Library/Cookies/Cookies.plist
    The parser will retry as in 10.2, but the problem should be corrected in the plist.

Later, when I had the time, I removed the entire preferences file for FlySketch from my ~/Library/Preferences/ directory and then restarted FlySketch and put in my serial number. Unfortunately, the same thing happened - the hot key for the full-screen capture was getting set, but it wasn't getting pushed to Preferences. Odd. We'll have to see what Gus makes of this.

[11/12/07] UPDATE: I got a new version of FlySketch from Gus and this fixed the problem right up. Not exactly sure what he changed, but whatever it was - it worked.

I know I shouldn't be surprised to see technology enter the medical profession, but with a sister that's a doctor, and her husband, and three roommates from college that are dentists, and after working with optometrists and doctors with their office automation, I have to say that I was honestly surprised to see that our local hospital, Edwards in Naperville, doesn't have xrays on film anymore. Well... maybe they do, but when you go to get a copy of them, you don't get films, you get a CD. "Hey!" I thought... "technology!".

Then I tried to read them on my Mac only to find out that they are a special file format: DICOM - Digital Imaging and Communications in Medicine. This format is not universally understood by apps like GraphicConverter or the like, but I was able to find that ImageJ could read the uncompressed DICOM files - which is nice because my wife's hand xrays came as uncompressed DICOM, but when I tried to load my son's rib xrays it said that it wasn't able to read compressed DICOM files. Crud.

I did a lot more googling and then came across OsiriX. What an amazing piece of open source software. This even has awards from Apple for it's excellent design - and tutorials on the Apple web site about how to use it. This is better than many large commercial outfits and yet it's totally free.

Let me give you a quick run-down of the application. First, it's Universal, and my guess is that they'll update it to Leopard to get the 64-bit processing on the Intel hardware as I believe it's capable of being 64-bit on G5 hardware. Second, it's got a built-in database that's smart enough to allow you to run it in a 'cache-only' mode (more on this later), but also capable of loading up xray series off CDs and keeping them on disk for later viewing. Thirdly, it's got automatic update checking, 'metal' theme (soon to change with Leopard), and responds very quickly to all user requests. It's a top notch Mac app.

One of the first things that made me say Wow! was the fact that on the toolbar there's an icon for importing a complete CD of xrays:

I hit that, and whoosh! everything from the CD is available in the app! I was stunned. I've been trying to get these xrays viewable for about 3 months, and all I had to do was to get this one program for my Mac, and click a button and everything is there - all the fields populated with when/where/why, etc. That's an amazing piece of coding!

But it didn't end there... oh no...

The default behavior is to have the application remove the entries from the database as soon as the CD is unmounted. This makes sense if a doctor is using it - there's no need to keep the images around if the disk is out of the drive. But for me, I want to keep the images around so I don't have to have the disk with me to look at the xrays. In order to do this you need to go into the Preferences for OsiriX and go to the 'Database' pane and select Copy files to OsiriX Data folder if: and then select If files are on a CD/DVD. This will mean that all the files will be copied from CDs. But we're not through yet... Next, go to the 'CD/DVD' pane and uncheck the option: Automatically remove images from the database when CD/DVD unmounted. This will then make sure that the images remain in the database once you remove the CD.

With this, I can carry around all the xrays of my family's accidents. I know it may sound a bit grizzly to some, but it's one of those things that makes me feel more connected to them.

The developer that has been doing a lot of work with the combo graph, and just graphs in general, came to me again today asking if it were possible to make the legend on a graph a fixed size. I didn't think so, but I didn't know for a fact. I knew that you could place the legend on the top of the graph, left, right (default), or bottom - as well as 'floating' it over the graph. I had put that into the base graph a while back, but failed to put it in the applet.

The problem is that if you are displaying graphs stacked on top of each other in a web page, you run the risk of having the legends different sizes (widths). If so, then the graphs themselves will be different sizes and the x-axes will not line up nicely one on top of the other. So the idea was to be able to set it to a fixed size large enough to hold all the legend entries on all the graphs, and then they would line up nicely.

What I found was that VantagePoint isn't as helpful here as it could be.

If you set the width of the legend you must also tell VantagePoint that you don't want it to sutomatically configure the size of the legend. This leads to the problem that if you don't set the height, then the default value is basically only one entry and it looks very bad. So when we set the width, we also have to set the height. Bummer there, folks. But thankfully, it looks like the scale is approximately 20 pixels for each entry in the legend.

Thankfully, I have the ability to see what's supposed to be on the graph, take that number, multiply it by 20 and use that as the height. It's not perfect, but it's pretty good. There might be problems if you change the font size, or even the font. Or if there is meant to be a scrolling list of legend entries and that's going to mess things up something fierce. But for a first cut, and because I've spent hours trying to figure this out with no luck, let's go with this for now.

If it turns out to be a bad guess, then I'll send an email to VantagePoint and ask them how to do a better job of estimation. I'm sure they have to have something that calculates the default height, all I need to do is to tap into that. May not be easy, but it's certainly an avenue if this doesn't work out.

I know I've said it before, but one of the most important things I think a professional developer can do is to program defensively. There's no reason not to - it doesn't take that many CPU cycles to check inputs and return values, and the benefits are almost incalculable when you need them. Case in point was a problem I ran across yesterday afternoon and took until this morning to clear up.

Several years ago, a co-worker put together a nice object representation of the instrument data sitting in the MarketMash server. He created this because the current model we were using, and still use to this day, is very amorphous, and not well defined. Nothing like you'd expect to see if you were a traditional java developer. So he decided to make a new object model that would sit beside the existing model and provide a lot more structure to the data and therefore make it a lot easier for people to get at the data they are interested in.

The problem with a lot of structure is as things evolve, you have to maintain that structure a lot more actively than the amorphous model. This came to pass yesterday as I was looking at the output logs of one of the processes of a system of mine. I had put into CloudCity the ability to handle logging of new instrument-level tags once per instance, and then simply ignoring them after that. What I hadn't done was to do the same for the position-level tags, and that's what got me.

Well... what really got me was optimistic programming. Such might be the case as the example:

    CCToDNAMap  dnaMap = CCToDNAMap.getInstance();
    dnaPositionTag = dnaMap.pairFromTag(tag).getName();

In my problem yesterday, it was the call to pairFromTag() that was returning a null and not getting checked for. The additionally odd problem was that this was not throwing a NullPointerException as you'd probably expect. In fact, it was throwing an Exception, but that exception was null. When I broke out the calls and tested every return value, things started working just fine.

I then took the time to add in the new fields, but the real issue was the optimistic programming. For if I add another position-level attribute now, it'll log it once and then continue to work as if nothing bad happened. That's what I wanted to happen in the first place.