What's it all about, Alfie?

I really love Sun hardware and software. My first exposure to Sun workstations was the 3/50 in grad school and it was the most impressive box I'd ever seen. Oh, I'd had an Amiga, and it was nice, but when I was at school and could work on a 3/50 - or later a 3/60, it was an entirely different experience. Night and day. Nothing beat Sun stuff. Nothing.

Then I went to Auburn, and again, Sun was king. Oh, we had PCs, and I had a nice Mac II, and later a IIci, but if you wanted to crunch numbers and display them nicely, Mac OS 6 or 7, or Windows 3.11 just didn't cut it. You had to get Sun.

Little did I know at the time that had I seen an SGI workstation, my opinion of Sun would have changed, and I would probably never have looked at Macs the same again. At heart, I'm a scientist and a numbers guy. Scientific apps and data visualization is what I love doing most.

Anyway... Sun ruled the roost. Then they stopped innovating. Solaris 2.5 was nice, 2.6 was good... heck, they all are nice, but they aren't anything in comparison to linux or Mac OS X these days. And here's the case that brought it to a head this week.

I've been working on writing a C++ wrapper for libcurl and I wrote it on the Mac where I knew I had a recent version of libcurl, with 32 and 64-bit support. Then I moved it to linux -in this case Fedora Core 7 on x86_64 and it again compiled without a hitch. Then I moved it to my Sun box to make sure it built there. I have to do this, and usually do it in this order as it's the fastest and easiest way I've found to get the cross-platform builds working.

When I went to Solaris I had to add an include for bzero() which on the Mac and Linux is in the more standard includes, but on Solaris 8 (9 and 10), it's in strings.h. No big deal, I added that. Then started the build again. It compiled the 32-bit version just fine and then hit the wall at the 64-bit version.

Oh, my code compiled just fine - it's generic C++ with a little STL, but it's the linking phase that messed up, and that's where Sun has been standing still.

Sun was one of the first to support 32-bit and 64-bit on the same deployment on the same hardware. It was an elegant solution - two sets of files. True, Apple had a trick up it's sleeve in it's historical use of the quad-FAT binary (HP-UX, Solaris, Intel, Motorola). This allowed them to jump ahead to a single file with multiple architectures and word-sizes. Linux copied Sun's ideas and delivered multiple libraries pretty early on, which brings everyone up to about the same level.

Almost.

The problem is that Sun has failed to recognize that libraries like libcurl, openssl, zlib, and the others that I've gotten from SunFreeware.com need to be available in the default installation of Solaris. I've got Solaris 8 and 10 and what's missing form them is significant. I know they have SunFreeware, but even there, they don't seem to have multiple architecture binaries of the packages. If they had, I'd be a happy camper. I have already gotten most of SunFreeware installed on my machines, but it's all the single 32-bit versions of the packages.

I can't be the only one that wants 64-bit versions of these libraries. Maybe Sun doesn't have to include them in the OS, but it'd be nice if I could get a CD of all the libraries on a linux box built into packages for installation. Disk space is cheap... install all of them both 32 and 64-bit. Then, should I want to make something in 64-bit mode, I can.

So I'm left looking at the Sun box and thinking how nice it could have been had Sun just kept moving while it was ahead. Keep moving on the UltraSPARC chips... innovate... be the best of the best. But in the end, clever people with commodity hardware have essentially put Sun in the position of an also-ran, and that's sad.

Like when I heard SGI was filing Chapter 11. Sad.

So I've emailed Steven C. at SunFreeware and asked him what it might take to build OpenSSL and zlib and curl for 64-bit on Solaris 8 and 10 SPARC. I expect that it'll be more than I want to do at this time. It's getting harder and harder to justify Sun hardware when I can get an 8-core x86_64 box with amazing speed for next to nothing. Cost will drive the market.

I'd love to keep working on Sun, and I'll never get rid of my SPARC 20, but I can see a time very soon that work will be driven off linux, and it's a solid platform that I've enjoyed using for years. It's just too bad that Sun was in the lead, and they stood still.

UPDATE: I've received email from Steve of SunFreeware that the most recent versions of Solaris do have things like OpenSSL, zlib, and curl in 64-bit as part of the install. If that's the case, then I'm happy for Sun, but a little disappointed that it wasn't made backward compatible in the sense of Solaris 8 and 10.

The other day a co-worker came up to me and asked me if I knew anything about this particular project in the Bank for getting index decompositions - basically, the instruments that make up the different indexes on the world markets. I hadn't heard about the project, but was very interested in the idea of adding it to my market data server as an additional data source.

So I looked at their web site and they had APIs in Java and C#, but nothing in C++ for several more months. I could wait, but I decided to fire a note to the developer as I'd worked with him in the past, and didn't know if he might have had a pre-release C++ API.

He mentioned that for the request/response work I'd be doing, the WebServices API was every bit as fast as the C++ was going to be, and since I could use the WebServices API right now, I decided to give it a go. Problem was, in CKit, I didn't have a nice way to get data back from URLs where you have any kind of complex request. So I decided to build one.

I looked around, and it was pretty clear that the best web services tool was going to be based on cURL. I know it's on every linux box, I can get it from SunFreeware.com for my Solaris boxes, and it's part of Mac OS X. It would handle a ton of different protocols and options, and it was pretty simple (looking) to work with. So I set off wrapping up cURL into CKit.

I have to say that I was more than a little surprised about the state of cURL. First, it does cover a ton of platforms. It's also got a ton of features. But what amazed me was the seemingly lack of attention to the details of really using the code. I mean it's not hard, but it is on ver. 7.18.1 and by that time I would have expected that they would have figured out how to get rid of these issues:

• Global Initializer - this blows me away. I read the docs and they say that it's because some of the libraries cURL uses are not themselves thread-safe, but to say that the cURL global initializer is not only not thread-safe, but needs to be done with only one thread active in the application is downright crazy... and sad. I've done what I can to try and make it as nice as possible, but it's certainly possible that I'm going to run into serious problems because of this. I just hope it's a good plan for the simple stuff.
• Keeping String Pointers - while I can certainly understand why they don't copy arguments passed in, there's no reason to have the 'easy' interface do that. Face it, 'easy' ought to mean 'fool-proof', and the way to do that is to control as much of the data as you can. Copy those arguments - don't require the caller to retain them for as long as you might need them. How's he to know when you're done with them?
• URL Structure Knowledge - this is something that I think they should have done - don't require the developer to know how to for a URL. Why make the developer encode the data when you know full well how to do it? Have the user give the API the data it needs and then have the API piece it together, encode it as necessary and then ship it off.
• Field Manipulation - when you add the POST variables to the handle, why not make it so that you can add them as key/value pairs? Why make the user encode them as a single string (which he has to keep around) and then pass them to you? Make it smarter than that. It's not hard - a list of key/value pairs - it's all strings, anyway. Make the data in the handle more manageable than it is now.

These are just the biggest problems I have with cURL. I mean it works, and it's found on almost all platforms, so I'll keep using it, but when I think that there has to have been a ton of revisions on this and these things aren't addressed, it make me think that the person writing it isn't really thinking about how it's being used.

That said, the CKURL does work, and does overcome each of these limitations. It doesn't require the user to do any global initializers, it copies all the data it needs from the arguments passed in, it creates the URL syntax from the general data you've given it, and it allows general field manipulation. All these things are making it a much more enjoyable piece of code to work with. But at the heart, it's still cURL. I'd just love to see them make a really 'easy' version.

For the last few days I've been trying to get an application to work with Java WebStart included with the JDK 1.6.0. It's been challenging, and what I've learned is that there's a lot you can do, and a lot you can't, and the trick is trying to turn the things you can't into things you can.

Packaging Shared Libraries

One of the things that isn't too hard to figure out is how to get shared libraries (so or dll) down to the client. Basically, you put them in a jar file and then sign that bad boy. There is one thing I learned: Sign all jars with the same key. The JWS is picky that way - they have to be the same key.

Delivering XML Files

Say you have an application (like I do) that uses XML files to configure itself properly. I didn't really pick this behavior, but it's something that I can see people dealing with. The Java code needs to be given the path to these XML files as arguments to the program, and then within the program it reads these XML files and deals with their contents. The problem is two-fold: getting the XML files there, and referencing their location.

Seems silly, but it's something I just lucked into. I had the XML files in a directory called cfg, and so I created a jar file with the cfg directory in it - just a bunch of XML files, and then signed that jar and placed it in the jnlp file as a nativelib resource. Something like this:

    <resources>
        <nativelib href="lib/cfg.jar" />
    </resources>

where I have ended up placing all my jar files in the lib subdirectory of the root of the JWS location. Then, if I use the arguments to the app indicating the location of the XML files are simply cfg/startUp.xml then the app will find them. Amazing, and interesting, and I wish someone had said the XML files worked like nativelib packages. I'm guessing just about anything can go into a nativelib and the code will find it.

Java VM Arguments

Again, this should be pretty obvious, but it wasn't in the examples I saw on the web. If you have a java command line like this:

    java -Xmx512m -DuseMyLog=true -DlogPath=logs/server.log -name Archie -type server

then the right way to split those up in the jnlp file is to put the -D arguments into the resources section and the regular args into the application-desc section. Something like this:

    <resources>
        <nativelib href="lib/cfg.jar" />
        <j2se version="1.5.0+" href="http://java.sun.com/products/autodl/j2se"
            max-heap-size="512m" />
        <property name="useMyLog" value="true" />
        <property name="logPath" value="logs/server.log" />
    </resources>
    <application-desc main-class="com.fun.MyServer">
        <argument>-name</argument>
        <argument>Archie</argument>
        <argument>-type</argument>
        <argument>server</argument>
    </application-desc>

I spent a ton of time trying to figure out what went where and for what reason. They certainly could have done a lot more in terms of the examples and explanations for this stuff. But, that's what I'm doing now, isn't it?

Finishing Touches

Once I got the thing working, I needed to get it updating and such. An addition in the 1.6.0 JWS you can add the update tag and that will let JWS know how to update the app based on changes on the server-side. I really liked this as I needed to make sure that I had this capability, and sure enough, it was there, just like I needed.

    <update check="always" policy="always" />

I have to admit that this has come a long way, and having it installed as a part of the JDK is really an important step in the right direction. I should be able to replace the third-party publish/subscribe application deployment system that's going away later this year with Java WebStart without too much trouble.

There are still things that I wish were there - variables. Make it so that you can put Java System property variables in the tags and then on invocation, have the JWS expand these. For example, say you wanted to put in the user name as an argument to the app. Since this is a static jnlp page served off an Apache web server, it's going to be very hard to get the username without placing a Java applet on the page that gets this data and sends it back to you in some form. Much too messy. Do this:

    <application-desc main-class="com.fun.MyServer">
        <argument>-name</argument>
        <argument>Archie</argument>
        <argument>-user</argument>
        <argument>@user.name</argument>
    </application-desc>

where the variable 'tag' @ is used to indicate that a Java System property variable name is following: user.name. Then, the JWS would replace the user's name in for this tag and pass the result to the application. This would solve a ton of issues as there are variables for most things you'd need, and the JWS is already running on the client machine, so it knows these things. It'd be really nice to have.

Thankfully, you can work that into your own code and do the same thing, but it's code we all don't need to write.

One of the nice things I like about Mac OS X 10.5 (Leopard) is the new way that X11 is launched. With launchd it's just a matter of a second or two when X11 tries to connect back and it's launched. Excellent. But it's not perfect.

When you close X11, it prompts you for a confirmation dialog, and there's no preference setting to say "don't confirm close". But some clever individual found a hidden defaults setting that does just that.

  defaults write org.x.x11 no_quit_alert -bool YES

And then the next time you launch X11 it'll not ask you 'are you sure?' when you want to quit. Nice.

David has released DataGraph 1.5 for Mac OS X 10.5 and the DataGraph Framework. This is a nice little upgrade as it adds several plot types for DataGraph and after the emails we have exchanged, I wanted to get the Framework (again) and look into the 'Real Time' example for ideas about plotting tick data or the simulation results.

He's also got a nice facelift to the icon. 🙂

After playing with the new release I started digging into the Framework and the example code for the real time data. Basically, he's sending a completely new data set to the graphing engine each time. This is a little different than I was looking at, but it's a worst case that shows that even in the case of redrawing 1000 points many times a second, DataGraph can keep up. Very smooth drawing. Impressive.

What I'm going to see if I can do is to take the results of one of my simulations and see if I can get this working on a simple 2D graph. I'm not sure how well it'll work out because the best fit right now would be the electrostatic solver and that's completely 3D, so I'll have to have it plot a slice through the workspace, I guess. Alternatively, hooking it into my FORTRAN code is going to be a little more difficult as that's not even generating any GUI and therefore I'll have a lot more work to do to get that going.

Anyway... fun times.

Several months ago, I looked into a graphing package called DataGraph. It's really good for what it does, and it's exceptionally inexpensive - unbeatable combination. I've been looking for something like VantagePoint for Mac OS X in Obj-C and this was looking like it was exactly what I wanted: it had a stand-alone package as well as a Framework for development and looked pretty nice. The only problem was that it was limited to simply 2D data.

The simulations I wanted to use this on are the electrostatic potentials and shark, my thesis work. The problem with these is that they are really 2D displays of 3D data. There was a more feature-rich product called DataTank, but that was $1300/seat. I wanted something in between - something that had the few 3D mapped to 2D graphs I needed without spending major dollars for this. I asked the developer at the time and (understandably) he wasn't as receptive as he might have been.

So I waited.

I decided to post another request on his message board asking if he'd be interested in this - given that I'm certainly willing to pay a few hundred dollars for this package. Lo and behold, he was interested, and emailed me back. I was thrilled!

I've been talking over a few ideas with him since then, and I'm really excited about the possibilities of getting DataImage (that's what he's tentatively calling it) as it will really enable me to do the work on the simulations I want as well as some interesting financial software that I've been thinking about writing. I can't wait to see what he's going to work up.

In the meantime, he's mentioned a few of the examples he delivers with his Framework and how they work with real-time data. I'm going to check those out and see what's up. Additionally, he mentioned that the communication with DataGraph, and therefore DataImage, is through a BSD sockets layer that he's written and Open Sourced. This means that I can have a look and see what it's going to take to integrate the FORTRAN code of my simulator to DataImage. That will be exciting as well.

Exciting times. Yeah!

I was reading this on Red Sweater Blog this morning and I can agree with the assessment, but I think he's being a little hard on Leo. I'm not sure Leo was advocating that everything for a computer be free, but there was a time that a lot of things were little "ditties" done by someone on their off-hours, and they simply posted it to the 'net and people liked it.

I did it. Lots did it.

It wasn't about making a buck, it was about having fun with the development environment, or the system, or the fun of seeing if it could be done. In the blog terms, it's Pure Charity.

Sure, it's not going to pay any bills, but that's what we did. For the Amiga, it was the Fish Disks - if you made it there, then you were living large. Seeing your code on an ftp site was a reason to be excited not upset. But those were the old days.

I don't disagree with Daniel's point. If you want something of quality, you should expect to pay for it. I do. I don't mind paying for good software, I like having good tools, and I let the people know that I like what they're doing, and when I'd like to see something a little different in the app. Totally agree.

But there was a time - and Leo and I remember it. That time may be gone now, and if it has, then OK. But don't slam the guy because he remembers the days when things were different. Hey, you may not have been around then, or even alive, but they were times, and often times we look at current events through the life experiences we've had. Nothing wrong with that.

Cut the man a break... he's not advocating the death of your industry, he might just be talking about days gone by.

I was reading Daring Fireball this morning and saw this post about using high-level languages to make development faster. I read this and had a few thoughts that didn't necessarily coincide with the original author's or John's. The gist of the original article was, I believe, this paragraph:

We couldn’t afford to keep spending time and energy writing software this way. Even if we could afford it, we didn’t want to spend our time this way. Web applications were being developed at an astounding pace in part because of centralized management and deployment (they never have to maintain different versions for Macs and PCs), but also because they were using modern interpreted languages.

and then he went on to say:

With high-level languages and good libraries, small teams can create great products at a rapid pace. We realized that we could write applications for the desktop in the exact same way.

True, very true. But the trade-offs seem to be completely overlooked. I mean completely overlooked. I don't disagree that if you want to reach a broad commercial audience you have to either go with the biggest audience or you have to try to go cross-platform in some kind of way. No disagreement there.

My disagreement with this argument is Why? Simply, Why? If you want commercial success then you need to be writing for the largest platform out there - Windows. Period. There's no need to write for Macs or Solaris or linux... just write for Windows and you're guaranteed to have the largest audience possible. No need to go any further.

When I started my company with my old friend we picked Windows for this same reason. I was a major Mac fan at the time, but the audience wasn't there. It just wasn't, and making a cross-platform anything was a waste of what little time there was in making it cross-platform. We needed clients, and they had Windows, it was as easy as that.

If you're trying to make something cross-platform, then pick a protocol and use that. I've done that numerous times and it works very well - be that a different OS or just a different language for the different APIs. Yes, it's more work, but in the end, it'll pay dividends and that's the point. If you're looking to make a fast buck, then by all means, write code in perl, python, Visual Basic, or Excel. There are people doing it and making a living at it - I did for 5 years. But that's something that we should get past, not aspire to.

Once you can pay the bills, then you should try to stop doing that kind of coding and get to the point that you take advantage of each operating systems' capabilities. There are things that are understandably cross-platform - the engine of the app, but there are a lot of things that need to take advantage of the differences in the different operating systems. Don't settle... write the best code possible. Don't say "Hey, we can, so we should."

I realize this is easy to say from my point of view, but I was there, and I did that work for years, and it paid the bills, and in the end, it was not emotionally fulfilling and so I left. I found something that was better, and while it's not self-employment, it's allowing me to set the terms of my work. I'm way past my 20s, and I'm looking for more than paying the bills at this point in my life.

Anyway... it's something to read and think about. Is it just about the speed and the buck, or is there more to the art of development that that? I think there is.

I was working on my latest price feeder today and got word that I wasn't sending enough information to the receiving system for it to make proper use of the data. Basically, I needed to get more data about the instruments from this system and then feed it back to the system with each price for each instrument. Problem was, that really made the way I was storing data for each instrument a royal pain and I needed to go from a simple STL data structure to a class that will hold all the data for each instrument and then put these instances into a data structure for efficient access.

What I did was something a lot like this test code:

/*
 * This trys to reproduce the bug I was seeing in the initialization
 * of the CKString while in a CKVector in a std::map.
 */
#include <iostream>
#include <map>
#include "CKVector.h"
#include "CKString.h"
 
typedef struct tuple_t {
	CKString	one;
	CKString	two;
	CKString	three;
 
	// constructors
	tuple_t() :
		one(),
		two(),
		three()
	{
	}
 
	tuple_t( const CKString & aOne,
			const CKString & aTwo,
			const CKString & aThree ) :
		one(aOne),
		two(aTwo),
		three(aThree)
	{
	}
 
	tuple_t( const tuple_t & anOther ) :
		one(),
		two(),
		three()
	{
		*this = anOther;
	}
 
	virtual ~tuple_t()
	{
	}
 
	tuple_t & operator=( const tuple_t & anOther )
	{
		one = anOther.one;
		two = anOther.two;
		three = anOther.three;
 
		return *this;
	}
 
	bool operator==( const tuple_t & anOther )
	{
		bool	equal = true;
		if ((one != anOther.one) ||
			(two != anOther.two) ||
			(three != anOther.three)) {
			equal = false;
		}
		return equal;
	}
 
	bool operator!=( const tuple_t & anOther )
	{
		return !operator==(anOther);
	}
 
	CKString toString() const
	{
		CKString	retval("[one=");
		retval.append(one).append(", two=").append(two).
			append(", three=").append(three).append("]");
		return retval;
	}
} tuple;
 
typedef CKVector<tuple> TList;
typedef std::map<CKString, TList> TListMap;
 
int main(int argc, char *argv[]) {
	// make the map - no entries
	TListMap	myMap;
	// this is the first key we'll be using
	CKString	myKey("key");
	// create the first map entry
	TList	& list = myMap[myKey];
	// create a tuple to place on the list
	tuple	t("a", "b", "c");
	std::cout << "tuple = " << t.toString() << std::endl;
	// put the tuple on the list
	std::cout << "list has " << list.size()
			<< " elems with capacity of " << list.capacity()
			<< std::endl;
	if (!list.contains(t)) {
		std::cout << "tuple not in list, adding..." << std::endl;
		list.addToEnd(t);
	}
	// print out the map
	std::cout << "here's the map now: ";
	TListMap::iterator	mi;
	for (mi = myMap.begin(); mi != myMap.end(); ++mi) {
		std::cout << mi->first << " = [";
		for (int i = 0; i < mi->second.size(); ++i) {
			if (i != 0) {
				std::cout << ", ";
			}
			std::cout << mi->second[i].toString();
		}
		std::cout << "]" << std::endl;
	}
} 

Lines 10-76 simply creates the object that is going to hold the instrument data - in this case, three pieces of data. Line 78 creates a type that is a simple vector of these tuples. Line 79 creates an STL map where the key is a string and the value is a vector of these tuples. Seems like a reasonable data structure, given that I'm basing this all on CKit.

In the main method, I'm basically creating a map of these string-list pairs, then creating a tuple and seeing if the new key isn't there - it's not, and then adding the tuple to the end of the list. This all works fine on Mac OS X 10.5.2 and GCC 4.0.1 (Xcode 3.0). But on RHEL 5 (CentOS 5) with GCC 4.1.2 we get an entirely different result.

At line 97 we get a very spectacular core dump. The gdp session is basically showing me that the initial value of one in the existing allocated storage in the vector for the first element is not being properly initialized. This initialization is done properly on OS X, but on RHEL5 it's not. When I change the typedefs to say something like this:

typedef CKVector<tuple> TList;
typedef std::map<CKString, TList*> TListMap; 

thus making it necessary for me to new a TList and place that into the TListMap, and then delete it when I'm done, then things work just fine and the initialization is properly done.

I'm not that big of a stickler for compiler bugs - they happen, but so rarely that it's not worth getting all bent out of shape over one that's so easily worked-around. But I'll admit, it's a little frustrating to think that it's your fault for several hours only to find out that it's not, that at least a part of me wants to send in a bug report, but I probably won't. Just glad I got it figured out.

Today I had to move my development of my latest price injector to a RedHat Enterprise Linux 5 machine (RHEL5) due to some third-party libraries that required version of libs that were present on RHEL5 but not RHEL3. No worries, I think, I have the machine, I just need to get a few things from the CentOS 5 site to complete the development tools installation, and then I'm ready to go.

I'm a Vim guy... love it. Don't even think about what I'm typing most of the time, so it's natural to put Vim on the RHEL5 box so I can edit the source. No surprise there. But what was a surprise is the fact that the menu redraw was exceptionally slow. I mean way way too slow for the machine and network I was using. Clearly, something was wrong.

So I started to investigate GTK themes a bit. Turns out, the one I was using (GTK-Step) uses bit-mapped graphics for the menus. This is a lot slower than we need. There's another called Smooth-Metal that was every bit as nice and after a few hours I was able to make it downright appealing to me, that uses a fraction of the bandwidth for the X11 GUI. The trick was having a tool to change the GTK Theme.

Enter gtk-chtheme. This is a wonderful little program that allows you to see the theme in action and then apply it to the window manager and voila! This is an important link to remember because I did a lot of searching to find something that worked as nicely as this little guy. Saved me a lot of grief.

UPDATE: gotta watch out though... the GTK Theme I like (Smooth-Metal) I've hacked up a touch to look even nicer uses the GTK Engine 'smooth' and that's not on RHEL3 - or FC3 systems. You have to get a reasonably recent build to get the 'smooth' engine. I've even tried compiling it from scratch for my RHEL3 boxes but it requires GTK 2.4+ and that is again more than is on the RHEL3 boxes and getting that on the boxes is too much because of the RPM dependencies. In short - it's too much grief for the benefit. So just be clear about what you're getting into with some of these themes.

[3/10/08] UPDATE: seems the theme engine is crucial. So I looked at the engines for FC5, RHEL3 and RHEL5 and came across the engine ThinIce. It works really well, and after about 20 mins of fiddling, I had something that looked as nice as the Smooth engine, but was working great on all the platforms I have. Now
I'm really set. Nice to get that out of the way.