Archive for the ‘Coding’ Category

Why Good Java Coders Need to Know C/C++

Friday, November 9th, 2007

cubeLifeView.gif

I was sitting here today working on the bulk push() and pop() for the BKit queues and a fellow developer stopped by to inform me that I wasn't using the best performing locking schemes in my implementations of the queues. While I accept that there are possibly ways to squeeze a few percent out of mine in certain conditions, these aren't custom-designed queues, they are general purpose utility classes that can't make assumptions about their use-cases in their implementations.

But that wasn't at all clear to this developer that stopped by my cube.

And that wasn't the only thing he didn't get.

Good Java coders are first and foremost good developers. Period. That means they understand how the code is executed in the machine. Where things might seem to be unimportant, or for that matter, important, and where they really are.

Case in point about high performance queues. If you're in a process where you have many threads processing data off a single queue, and the processing time is small, then you are guaranteed to have locking contention on the queue - by definition. There's no way around it. The solution I've seen time and again is to remove things from the queue in bulk and then the processing time goes up and the locking contention goes down because the number of threads hitting the queue at the same time goes way down.

But to this developer, the answer was Java 5's optimistic locking. Yes, folks, assume you'll get the lock and then fail and retry if you don't. Well, that's a wonderful idea in a situation that's already got locking contention problems. What was he thinking? The answer was, he simply wasn't. He was thinking that Java's way of doing things was so easy that under the covers it really didn't have to do any low-level locking for the push() or the pop().

So I had to walk him through it. It finally dawned on him that his statements were crud, and when he did, I could see him deflate right in front of my eyes. But for those 15 minutes when he was sure I was missing the point I wanted to yell at him Are you a total idiot? Don't you know any language other than Java? And folks, unfortunately, the answer to that is No. If you don't have any more experience in developing than Java, it's hard to see that it's not the end-all-be-all.

Adding the Bulk push()/pop() Methods to BKit Queues

Friday, November 9th, 2007

BKit.jpg

Today I was helping out a developer refactor his code so that it would be more CPU intensive for a shorter time as opposed to "tailing off" as it ran. The problem is this: his app needed to read a large amount of data (30,000 to 50,000 records) per file for several files. But not all files had the same length. This meant that his original design where h file processor was a thread, had the limitation that as some threads finished, 'less' of the machine was dedicated to the task. He wanted to fix that.

The easiest way to fix that is to have all processing go through a single queue - as opposed to a queue per file. This is not without it's limitations, however, as now this queue will become the bottleneck as synchronization on it to push() and pop() items will be under contention by all threads. The refactoring had a few issues, but the real problem remained - the queue operations needed to be done in bulk.

So, I added the ability to push a Collection of objects onto each queue (LIFO and FIFO), as well as popping off a bunch of values (up to a provided limit). This is the only way to achieve balance between the processing threads and the locking on the queue. The end result is that he now has another parameter to tune - the batch size of the pops. If it's 50 to 100 he should be in fine shape and not have to worry about performance hits due to excessive locking contention, but I'll let him fiddle with that and find the optimal value for his process.

Interesting Argument about Boost Smart Pointers

Thursday, November 8th, 2007

cplusplus.jpg

In my work today on updating the valuation library to a current production version, I once again came across Boost smart pointers. Now there's a lot of good things in Boost, and I have barely scratched the surface of what Boost has to offer, but the implementations I've seen that use smart pointers are much more confusing than they are helpful. After all, the point is to make it easier to write code - not harder.

Well, I was chatting with a good friend about this today after the fact and he uses Boost's smart pointers a lot and doesn't see the confusion. I can agree that if I were using them all the time I'd probably be desensitized to them as well. But I'm not yet used to Boost's smart pointers, and honestly prefer to handle the memory management myself - that's one of the reasons I'm using C++ in the first place - careful resource management. But I see his points, and because I found the conversation very interesting, here's the gist of it.

Take the following little code snippet:


    typedef EQS::Shared<Operation>::Ptr OperationPtr;
    typedef EQS::Shared<Results>::Ptr ResultsPtr;
    ...
    OperationPtr    lOperation = lFactory.getOperation(mType);
    ResultsPtr      lResults = lOperation->execute();
    double          value = lResults->getValue();

While it's perfectly legal code, if you separate the typedefs into the header file and the remaining three lines into an implementation file you have something that's confusing to a traditional C/C++ developer. Are lOperation and lResults pointers or not? Well... in reality, they are and they aren't.

They are in the sense that you can use the "->" to access methods, but in another sense, they are removed when they go out of scope in the code - like traditional stack variables. While this might be seen as a benefit by some, to me it makes the code exceptionally confusing.

In my opinion the line is being drawn too finely. Make theme appear much different from pointers and the confusion goes away. For example, the following code looks odd, and different, but there's no possibility that someone will be confused by the pointer-nature of the "->":


    typedef EQS::Shared<Operation>::Ptr OperationPtr;
    typedef EQS::Shared<Results>::Ptr ResultsPtr;
    ...
    OperationPtr    lOperation = lFactory.getOperation(mType);
    ResultsPtr      lResults = lOperation..execute();
    double          value = lResults..getValue();

or, as if taking a page from Objective-C, this:


    OperationPtr    lOperation = lFactory.getOperation(mType);
    ResultsPtr      lResults = lOperation[execute()];
    double          value = lResults[getValue()];

In both cases, there is a distinct visual difference in how the objects are being used. In these examples, this is silly and trivial, but in large code sections where there is a lot of processing and the usage of the "->" implies traditional pointer, it's confusing.

I know they did this to make them seem as close to real pointers as possible and allow for the auto-CG, but I think they'd be better off - as C++ was with references, in making a new language element. And if they can't have that, then at least overload a different operator - or something to make it looks significantly different from traditional pointers.

I guess my point is that by using smart pointers you'd never think twice about doing this:


    SmartPtr    p = foo.bar();
    p->goof();

but you'd never do this:


    char        p = foo.bar();
    p->goof();

and while you could do this:


    typedef char *CharPtr;
    ...
    CharPtr     p = foo.bar();
    p->goof();

why would you?

Consistency... that's an important part of a language to me. Perl is nice, and yet it's strength is also it's weakness - you can do the same thing a million different ways. But almost all my Perl code does the same thing the same way - just for that consistency.

The conversation has continued and my friend has pointed out that there is a significant historical component to this that I wasn't aware of, and it sheds light on why they are used this way:

fair enough, call it something else, but, history helps shed light here. In the early days of c++, you didn't have any such thing. Then sometime after the first standard release, they said, hey, let's address a simple memory leak issue and added auto_ptr to the stl. Now auto_ptr seems like a reasonable name and they designed it so that you had almost no coding impact to replace your regular c-style pointers with auto_ptr. You could imagine the resistence they would get if you had to re-write huge chunks of code to migrate over to it. Instead, you had to change your type declarations and remove some deletes and that was basically it.

However, people started to realize that std::auto_ptr wasn't so good for various reasons and the boost guys came along and introduced a richer set of pointer objects that kept the same semantics so that it was easy to migrate.

I can see his point, but when he pressed me about what I thought C++ should do I said that they needed to add a fourth elemental data type:

  • value
  • pointer
  • reference
  • smart pointer

and that smart pointer is going to have a different decorator for accessing it's methods and ivars. By making it too close to a pointer, they have in effect made it more confusing.

Where Java Got it Right

After thinking about this overnight, I have to admit that this is where Java got it right. If they had wanted to add in a reference-counted dynamic object in C++, they should have created smart references. Had they done that, it would have allowed for the 'new' without the need for a 'delete'... it would have used the '.' as the method invocator, and it would have allowed for the complete absence of the '*' in definitions. In fact, I suppose that the only thing they are missing now is the use of '->' when '.' would be better. Then, simply call them smart references and the confusion is gone.

You can still have them hold a NULL, unlike traditional C++ references, but that would be the edge condition, as it is in Java, and not the typical use of the datatype. Yup, I have to say that they'd be so much better off in my mind if they went for 'references' as opposed to 'pointers'.

Way Too Much Fun Coding

Thursday, November 8th, 2007

cubeLifeView.gif

Today I took on a project that I was convinced would take me months - the upgrading of our calculation library framework from a very old version (more than 24 months) to a very recent version (in production systems now). I was convinced that going from a v6.x.x to a v7.x.x version was going to be a pain, and plenty of people didn't say I was wrong. But I was very wrong.

It took me about 2.5 hours from start to finish to update the calls and the use of Boost smart-pointers from regular pointers and clean up the code a bit. Amazing! I had no idea it'd be this easy or I'd have done it months ago. Easily. But that just goes to show you that you don't really know how hard something will be until you really roll up your sleeves and get into it.

With this, we'll be able to add in several new features that the traders have been wanting to see, and in fact have just been waiting to use because the older version of the valuation library constantly generated a ton of SegFaults when I gave it the skew and kurtosis curves for the instruments we had. I was able to turn that feature on in the code and it works beautifully. That's nice.

So today has been a very good day for me. Wonderful, in fact.

Added More Legend Flexibility to the BKBaseGraphApplet

Tuesday, November 6th, 2007

comboGraph.png

The developer that has been doing a lot of work with the combo graph, and just graphs in general, came to me again today asking if it were possible to make the legend on a graph a fixed size. I didn't think so, but I didn't know for a fact. I knew that you could place the legend on the top of the graph, left, right (default), or bottom - as well as 'floating' it over the graph. I had put that into the base graph a while back, but failed to put it in the applet.

The problem is that if you are displaying graphs stacked on top of each other in a web page, you run the risk of having the legends different sizes (widths). If so, then the graphs themselves will be different sizes and the x-axes will not line up nicely one on top of the other. So the idea was to be able to set it to a fixed size large enough to hold all the legend entries on all the graphs, and then they would line up nicely.

What I found was that VantagePoint isn't as helpful here as it could be.

If you set the width of the legend you must also tell VantagePoint that you don't want it to sutomatically configure the size of the legend. This leads to the problem that if you don't set the height, then the default value is basically only one entry and it looks very bad. So when we set the width, we also have to set the height. Bummer there, folks. But thankfully, it looks like the scale is approximately 20 pixels for each entry in the legend.

Thankfully, I have the ability to see what's supposed to be on the graph, take that number, multiply it by 20 and use that as the height. It's not perfect, but it's pretty good. There might be problems if you change the font size, or even the font. Or if there is meant to be a scrolling list of legend entries and that's going to mess things up something fierce. But for a first cut, and because I've spent hours trying to figure this out with no luck, let's go with this for now.

If it turns out to be a bad guess, then I'll send an email to VantagePoint and ask them how to do a better job of estimation. I'm sure they have to have something that calculates the default height, all I need to do is to tap into that. May not be easy, but it's certainly an avenue if this doesn't work out.

They Don’t See the Value of Defensive Programming

Tuesday, November 6th, 2007

cubeLifeView.gif

I know I've said it before, but one of the most important things I think a professional developer can do is to program defensively. There's no reason not to - it doesn't take that many CPU cycles to check inputs and return values, and the benefits are almost incalculable when you need them. Case in point was a problem I ran across yesterday afternoon and took until this morning to clear up.

Several years ago, a co-worker put together a nice object representation of the instrument data sitting in the MarketMash server. He created this because the current model we were using, and still use to this day, is very amorphous, and not well defined. Nothing like you'd expect to see if you were a traditional java developer. So he decided to make a new object model that would sit beside the existing model and provide a lot more structure to the data and therefore make it a lot easier for people to get at the data they are interested in.

The problem with a lot of structure is as things evolve, you have to maintain that structure a lot more actively than the amorphous model. This came to pass yesterday as I was looking at the output logs of one of the processes of a system of mine. I had put into CloudCity the ability to handle logging of new instrument-level tags once per instance, and then simply ignoring them after that. What I hadn't done was to do the same for the position-level tags, and that's what got me.

Well... what really got me was optimistic programming. Such might be the case as the example:


    CCToDNAMap  dnaMap = CCToDNAMap.getInstance();
    dnaPositionTag = dnaMap.pairFromTag(tag).getName();

In my problem yesterday, it was the call to pairFromTag() that was returning a null and not getting checked for. The additionally odd problem was that this was not throwing a NullPointerException as you'd probably expect. In fact, it was throwing an Exception, but that exception was null. When I broke out the calls and tested every return value, things started working just fine.

I then took the time to add in the new fields, but the real issue was the optimistic programming. For if I add another position-level attribute now, it'll log it once and then continue to work as if nothing bad happened. That's what I wanted to happen in the first place.

Converting to Latest gfortran from Mac HPC

Monday, November 5th, 2007

xcode.jpg

In preparation for Leopard, I decided to see if the latest version of gfortran from the Mac HPC guys would build my thesis work. As little as about 6 months ago it wouldn't - there were issues with it that were supposedly fixed, but the build wasn't on the latest release. I stuck with g77 because it was Tiger (10.4), and it worked. But with Leopard using gfortran and no g77 port, it seemed like a good idea to get the latest gfortran and give it a try.

Interestingly, when I compiled it it compiled cleanly, but on linking I got the following:

    /usr/bin/ld: warning can't open dynamic library: /libgcc_s.1.dylib referenced
    from: /usr/local/lib/gcc/i386-apple-darwin8.10.1/4.3.0/../../../libgfortran.dylib
    (checking for undefined symbols may be affected) (No such file or directory,
    errno = 2)

I did a little searching and found that the solution was to change the 'install_name' (refer to a previous post about Xcode 2.4+ needing this) from /libgcc_s.1.dylib to something like where it's really located: /usr/local/lib/libgcc_s.1.dylib.

To do this simply go into /usr/local/lib and type:

    sudo install_name_tool -change /libgcc_s.1.dylib /usr/local/lib/libgcc_s.1.dylib
      libgfortran.3.dylib

(all one line). With this, the linking problem disappears and the code runs great. Now I know that I don't need g77 on Leopard. Nice. All we're doing is changing the 'install_name' in the gfortran library to have the complete path name as opposed to the simply leading '/'. Nice little tool to have around when issues like this come up.

Creating a Hybrid Java Preferences Framework

Friday, November 2nd, 2007

Creating a Better FileSystemPreferences Framework

Thursday, November 1st, 2007

BKit.jpg

I have spent the last few days - on and off, talking to a developer about the need for a global user Preferences system. The one that comes with most JDKs is decent but there's a problem using the one that ships with Windows - it uses the Registry. That's not the issue as much as that sets up the problem. The user preferences part of the Registry can be put in an unreliable state if you log into two machines and aren't very careful about what you're doing.

Imagine you have two Windows boxes: A and B. On these boxes you have an application - it resides on a server so it's up to you where you launch it. You decide to log into A and then B and launch the app on A. You move a few windows, make a few changes - basically change the Java Preferences values. When you exit the app, your changes have been written to the Registry on A. So far so good. If you logged out of B before logging out of A then the Registry will remain correct and the changes from A will be written to the PDC. But what if you log out of A first?

Well... your changes will be written to the PDC, and then when you log out of B you're going to have that Registry's state written to the PDC. You've lost your changes. No way to get them back.

This is a common problem for a lot of our users. They have multiple machines and we'd like to use Preferences as it's easy and clean, but there is this problem. On Linux/Unix this isn't an issue because the user Preferences are stored in your Home directory. Write it there and you're good to go. Login/logout order doesn't matter. But it does with Windows.

So we decided that it would be a good idea to use the FileSystemPreferences system that's used in Linux for Windows. The problem is that it's not available in the Windows JRE because it's not considered necessary. What a mistake. I've spent the day getting the source code for the FileSystemPreferences and FileSystemPreferencesFactory and creating BKit versions of them that are cross-platform.

That last part was the real key. The original source code for the FileSystemPreferences wasn't because it used three native methods that really didn't have to be used, but were used because they didn't take advantage of the NIO additions in JDK 1.4. I cleaned up all the code, got rid of the native methods, removed the silly XML dependency and used the PList work I had just done on the BKHashTree to serialize these maps to files.

In the end, we have something that's a drop-in replacement for Preferences that will work on all platforms. This means that we can point the user Preferences to a shared disk and so all flushes of the Preferences will write to the files and there won't be any problems like we're having now with the Registry. This is going to make things much nicer.

Adding the Ability to Customize the Legend in Graphs

Wednesday, October 31st, 2007

comboGraph.png

A developer came by today to ask me if it would be possible to add the ability to change the foreground and background colors on the Legend in the BKBaseGraphApplets. Also, he wanted to know if it would be possible to remove the border. Basically, he didn't like the grey background and the 3D border and wanted something that seemed to fit in the same plane as the graph itself. Seems reasonable, and after doing it, it actually looks reasonably nice.

Anyway, the work wasn't that hard - took me about an hour for the whole deal. But an interesting part was that you had to disable the 3D effect on the legend in addition to disabling the border. Seems the two are different visual objects. Nice of them, to add that level of customization. For now, however, turning the thing off or on is good enough.