What's it all about, Alfie?

The other day a co-worker came up to me and asked me if I knew anything about this particular project in the Bank for getting index decompositions - basically, the instruments that make up the different indexes on the world markets. I hadn't heard about the project, but was very interested in the idea of adding it to my market data server as an additional data source.

So I looked at their web site and they had APIs in Java and C#, but nothing in C++ for several more months. I could wait, but I decided to fire a note to the developer as I'd worked with him in the past, and didn't know if he might have had a pre-release C++ API.

He mentioned that for the request/response work I'd be doing, the WebServices API was every bit as fast as the C++ was going to be, and since I could use the WebServices API right now, I decided to give it a go. Problem was, in CKit, I didn't have a nice way to get data back from URLs where you have any kind of complex request. So I decided to build one.

I looked around, and it was pretty clear that the best web services tool was going to be based on cURL. I know it's on every linux box, I can get it from SunFreeware.com for my Solaris boxes, and it's part of Mac OS X. It would handle a ton of different protocols and options, and it was pretty simple (looking) to work with. So I set off wrapping up cURL into CKit.

I have to say that I was more than a little surprised about the state of cURL. First, it does cover a ton of platforms. It's also got a ton of features. But what amazed me was the seemingly lack of attention to the details of really using the code. I mean it's not hard, but it is on ver. 7.18.1 and by that time I would have expected that they would have figured out how to get rid of these issues:

• Global Initializer - this blows me away. I read the docs and they say that it's because some of the libraries cURL uses are not themselves thread-safe, but to say that the cURL global initializer is not only not thread-safe, but needs to be done with only one thread active in the application is downright crazy... and sad. I've done what I can to try and make it as nice as possible, but it's certainly possible that I'm going to run into serious problems because of this. I just hope it's a good plan for the simple stuff.
• Keeping String Pointers - while I can certainly understand why they don't copy arguments passed in, there's no reason to have the 'easy' interface do that. Face it, 'easy' ought to mean 'fool-proof', and the way to do that is to control as much of the data as you can. Copy those arguments - don't require the caller to retain them for as long as you might need them. How's he to know when you're done with them?
• URL Structure Knowledge - this is something that I think they should have done - don't require the developer to know how to for a URL. Why make the developer encode the data when you know full well how to do it? Have the user give the API the data it needs and then have the API piece it together, encode it as necessary and then ship it off.
• Field Manipulation - when you add the POST variables to the handle, why not make it so that you can add them as key/value pairs? Why make the user encode them as a single string (which he has to keep around) and then pass them to you? Make it smarter than that. It's not hard - a list of key/value pairs - it's all strings, anyway. Make the data in the handle more manageable than it is now.

These are just the biggest problems I have with cURL. I mean it works, and it's found on almost all platforms, so I'll keep using it, but when I think that there has to have been a ton of revisions on this and these things aren't addressed, it make me think that the person writing it isn't really thinking about how it's being used.

That said, the CKURL does work, and does overcome each of these limitations. It doesn't require the user to do any global initializers, it copies all the data it needs from the arguments passed in, it creates the URL syntax from the general data you've given it, and it allows general field manipulation. All these things are making it a much more enjoyable piece of code to work with. But at the heart, it's still cURL. I'd just love to see them make a really 'easy' version.

Over the course of the last few days, I've been having problems with getting a certain hostname resolved on my linux workstations at work. Basically, everything was fine until I tried to reference the host chatcentral and then apps like traceroute would hang for 20 sec and then work normally. Ping worked just fine, however. Very odd.

This first came up when I was trying to connect to this host with a simple C++ and Java socket connection. We did the migration to MindAlign recently, and I had written native connection clients to the MindAlign system and when switching over on my development workstation it appeared that the connection was hanging - in reality it was timing out on the hostname resolution. I didn't give it 20 sec, and thought "Hey, it's dead". I figured that when I was using different machines things worked fine, so I did the rest of the migration on those machines. But it was still bothering me, so I came back to it.

What I found with a little bit of Googling was that IPv6 was getting in the way of the normal host name resolution chain. If I put the machine into my /etc/hosts file then all was fast and well. But if I didn't want that, I needed to do something to get rid of the timeouts. With the help of a local Unix admin, we realized that the timeout was on the name lookup for IPv6. From there, it was a simple Google to see that the solution was to add the following line to /etc/modprobe.conf:

    alias ipv6 off

After a reboot, this fixed the problem.

Interestingly, this doesn't seemingly disable IPv6 as ifconfig -a still shows the IPv6 address for eth0, so I'm not sure exactly what's happening, but in this case, where the folks didn't put the hostname in the primary YP server, this really helps.

I've been using Firefox 3.0 betas on my linux boxes because it's the first normal distribution of Firefox that works as a 32-bit app properly on my x86_64 linux workstation. I can use the 32-bit JDK plugin and that's what I really needed. Very nice. But I've still been using the 2.0 version on my Mac and my PC because I wanted to make sure that I had stability over new features because I didn't need anything in 3.0 on these platforms.

Well, today I noticed that 3.0b5 on the Mac had a really nice look-n-feel to it. So I decided What the heck? and got it. Very, very nice. I still have 2.0 on my PC as it's production stuff and I have to have it working there, but the Mac can have an occasional crash and I won't be that upset.

I've read somewhere that they expect the final 3.0 release to be in June - which seems like a long time for this being the final beta, but I guess they have a release process that includes optimizations (or not), and so it'll be 'done' when they say so. Can't complain, this is pretty nice.

Then again, with the new fixes to WebKit, I'm excited to see the next Safari as well.

I was reading this on Red Sweater Blog this morning and I can agree with the assessment, but I think he's being a little hard on Leo. I'm not sure Leo was advocating that everything for a computer be free, but there was a time that a lot of things were little "ditties" done by someone on their off-hours, and they simply posted it to the 'net and people liked it.

I did it. Lots did it.

It wasn't about making a buck, it was about having fun with the development environment, or the system, or the fun of seeing if it could be done. In the blog terms, it's Pure Charity.

Sure, it's not going to pay any bills, but that's what we did. For the Amiga, it was the Fish Disks - if you made it there, then you were living large. Seeing your code on an ftp site was a reason to be excited not upset. But those were the old days.

I don't disagree with Daniel's point. If you want something of quality, you should expect to pay for it. I do. I don't mind paying for good software, I like having good tools, and I let the people know that I like what they're doing, and when I'd like to see something a little different in the app. Totally agree.

But there was a time - and Leo and I remember it. That time may be gone now, and if it has, then OK. But don't slam the guy because he remembers the days when things were different. Hey, you may not have been around then, or even alive, but they were times, and often times we look at current events through the life experiences we've had. Nothing wrong with that.

Cut the man a break... he's not advocating the death of your industry, he might just be talking about days gone by.

I was working on my latest price feeder today and got word that I wasn't sending enough information to the receiving system for it to make proper use of the data. Basically, I needed to get more data about the instruments from this system and then feed it back to the system with each price for each instrument. Problem was, that really made the way I was storing data for each instrument a royal pain and I needed to go from a simple STL data structure to a class that will hold all the data for each instrument and then put these instances into a data structure for efficient access.

What I did was something a lot like this test code:

/*
 * This trys to reproduce the bug I was seeing in the initialization
 * of the CKString while in a CKVector in a std::map.
 */
#include <iostream>
#include <map>
#include "CKVector.h"
#include "CKString.h"
 
typedef struct tuple_t {
	CKString	one;
	CKString	two;
	CKString	three;
 
	// constructors
	tuple_t() :
		one(),
		two(),
		three()
	{
	}
 
	tuple_t( const CKString & aOne,
			const CKString & aTwo,
			const CKString & aThree ) :
		one(aOne),
		two(aTwo),
		three(aThree)
	{
	}
 
	tuple_t( const tuple_t & anOther ) :
		one(),
		two(),
		three()
	{
		*this = anOther;
	}
 
	virtual ~tuple_t()
	{
	}
 
	tuple_t & operator=( const tuple_t & anOther )
	{
		one = anOther.one;
		two = anOther.two;
		three = anOther.three;
 
		return *this;
	}
 
	bool operator==( const tuple_t & anOther )
	{
		bool	equal = true;
		if ((one != anOther.one) ||
			(two != anOther.two) ||
			(three != anOther.three)) {
			equal = false;
		}
		return equal;
	}
 
	bool operator!=( const tuple_t & anOther )
	{
		return !operator==(anOther);
	}
 
	CKString toString() const
	{
		CKString	retval("[one=");
		retval.append(one).append(", two=").append(two).
			append(", three=").append(three).append("]");
		return retval;
	}
} tuple;
 
typedef CKVector<tuple> TList;
typedef std::map<CKString, TList> TListMap;
 
int main(int argc, char *argv[]) {
	// make the map - no entries
	TListMap	myMap;
	// this is the first key we'll be using
	CKString	myKey("key");
	// create the first map entry
	TList	& list = myMap[myKey];
	// create a tuple to place on the list
	tuple	t("a", "b", "c");
	std::cout << "tuple = " << t.toString() << std::endl;
	// put the tuple on the list
	std::cout << "list has " << list.size()
			<< " elems with capacity of " << list.capacity()
			<< std::endl;
	if (!list.contains(t)) {
		std::cout << "tuple not in list, adding..." << std::endl;
		list.addToEnd(t);
	}
	// print out the map
	std::cout << "here's the map now: ";
	TListMap::iterator	mi;
	for (mi = myMap.begin(); mi != myMap.end(); ++mi) {
		std::cout << mi->first << " = [";
		for (int i = 0; i < mi->second.size(); ++i) {
			if (i != 0) {
				std::cout << ", ";
			}
			std::cout << mi->second[i].toString();
		}
		std::cout << "]" << std::endl;
	}
} 

Lines 10-76 simply creates the object that is going to hold the instrument data - in this case, three pieces of data. Line 78 creates a type that is a simple vector of these tuples. Line 79 creates an STL map where the key is a string and the value is a vector of these tuples. Seems like a reasonable data structure, given that I'm basing this all on CKit.

In the main method, I'm basically creating a map of these string-list pairs, then creating a tuple and seeing if the new key isn't there - it's not, and then adding the tuple to the end of the list. This all works fine on Mac OS X 10.5.2 and GCC 4.0.1 (Xcode 3.0). But on RHEL 5 (CentOS 5) with GCC 4.1.2 we get an entirely different result.

At line 97 we get a very spectacular core dump. The gdp session is basically showing me that the initial value of one in the existing allocated storage in the vector for the first element is not being properly initialized. This initialization is done properly on OS X, but on RHEL5 it's not. When I change the typedefs to say something like this:

typedef CKVector<tuple> TList;
typedef std::map<CKString, TList*> TListMap; 

thus making it necessary for me to new a TList and place that into the TListMap, and then delete it when I'm done, then things work just fine and the initialization is properly done.

I'm not that big of a stickler for compiler bugs - they happen, but so rarely that it's not worth getting all bent out of shape over one that's so easily worked-around. But I'll admit, it's a little frustrating to think that it's your fault for several hours only to find out that it's not, that at least a part of me wants to send in a bug report, but I probably won't. Just glad I got it figured out.

Today I had to move my development of my latest price injector to a RedHat Enterprise Linux 5 machine (RHEL5) due to some third-party libraries that required version of libs that were present on RHEL5 but not RHEL3. No worries, I think, I have the machine, I just need to get a few things from the CentOS 5 site to complete the development tools installation, and then I'm ready to go.

I'm a Vim guy... love it. Don't even think about what I'm typing most of the time, so it's natural to put Vim on the RHEL5 box so I can edit the source. No surprise there. But what was a surprise is the fact that the menu redraw was exceptionally slow. I mean way way too slow for the machine and network I was using. Clearly, something was wrong.

So I started to investigate GTK themes a bit. Turns out, the one I was using (GTK-Step) uses bit-mapped graphics for the menus. This is a lot slower than we need. There's another called Smooth-Metal that was every bit as nice and after a few hours I was able to make it downright appealing to me, that uses a fraction of the bandwidth for the X11 GUI. The trick was having a tool to change the GTK Theme.

Enter gtk-chtheme. This is a wonderful little program that allows you to see the theme in action and then apply it to the window manager and voila! This is an important link to remember because I did a lot of searching to find something that worked as nicely as this little guy. Saved me a lot of grief.

UPDATE: gotta watch out though... the GTK Theme I like (Smooth-Metal) I've hacked up a touch to look even nicer uses the GTK Engine 'smooth' and that's not on RHEL3 - or FC3 systems. You have to get a reasonably recent build to get the 'smooth' engine. I've even tried compiling it from scratch for my RHEL3 boxes but it requires GTK 2.4+ and that is again more than is on the RHEL3 boxes and getting that on the boxes is too much because of the RPM dependencies. In short - it's too much grief for the benefit. So just be clear about what you're getting into with some of these themes.

[3/10/08] UPDATE: seems the theme engine is crucial. So I looked at the engines for FC5, RHEL3 and RHEL5 and came across the engine ThinIce. It works really well, and after about 20 mins of fiddling, I had something that looked as nice as the Smooth engine, but was working great on all the platforms I have. Now
I'm really set. Nice to get that out of the way.

It's not as often, that's for sure, but it's still there - the bug in Adium 1.2.1 where when I open my laptop and it wakes up from sleep the sound effects in Adium are missing, gone, silent, whatever you want to say. It doesn't seem to happen every time, and maybe it's in OS X, for all I know, but it would be nice not to have to restart Adium when this happens.

I know there's a new Quicktime 7.4.x update that I need to apply, but I'm doubting that that update is going to really effect me, but I'll give it a go and see if that helps anything. Meanwhile, I'll just have to restart it as I've already sent in the bug report on this already.

UPDATE: I did the Quicktime update. We'll see if this makes any difference on the sound from Adium.

Today two new releases on the FORTRAN front - both nice to hear. The first is that there's a new build of gfortran on the Mac HPC web site. This is the stand-alone compiler that works with the Xcode 3 tools from Apple and gives you basic command-line gfortran compiling. It's an easy install - simply un-tar it and it's ready to go.

The other is that the MacResearch guys have put together a .pkg file for installing the above mentioned gfortran and also an Xcode 3-compatible plugin for gfortran so that you can create FORTRAN projects in Xcode 3 and build them without having to use Makefiles.

I'm a huge fan of the compiler, and I'll probably get the package and install it on my iMac G5 at home to make sure it works OK, but on the whole, I don't do a lot of work with FORTRAN that isn't already covered with Makefiles. This is because there's usually also a few C files and then I'm old school when Makefiles were considered 'new'. So I have them in my projects already and there very fast to write if needed.

But if I were teaching someone about FORTRAN, I'd definitely make sure they had the Xcode 3 plugin installed. Thanks goes out to the two groups for taking the time to update the tools.

Today I needed to change our CVS pserver from an old Sun workstation to a linux box because some goobers at work decided to change the network taps on the Sun box and hook it to a non-responsive network port. Thanks, guys. Anyway, I took this opportunity to move the pserver from this old Sun workstation to a linux Blade in the server room. The steps are pretty simple.

First, make sure that /etc/services has the following line in it:

    cvspserver      2401/tcp

Then in the directory /etc/xinetd.d create a file called cvspserver and in it place:

    # default: off
    # description: A CVS pserver
    service cvspserver
    {
        disable = no
        socket_type = stream
        protocol = tcp
        user = root
        wait = no
        server = /usr/bin/cvs
        server_args = -f --allow-root=/path/to/your/cvsroot pserver
        log_on_success += USERID
        log_on_failure += USERID
    }

and then restart xinetd with:

    service xinetd restart

That's it. It works just as well as a pserver on Solaris because it's the same bloody thing. Now I don't have to worry about the goobers and network taps.

This is the second time this has happened to me - maybe more, I'm only starting to notice it today. I'm running Adium 1.2 on my MacBook Pro with 10.5.1 (fully updated) and when I put my laptop to sleep I think it turns off the sound on Adium until I restart the app. I'm going to check the website to see if they have any bugs reported on this - I know the last update included a new QuickTime - maybe that's to blame.

I just know that I don't like not being able to hear the chat announcements. I count on them to get my attention when I'm working at another machine. Crud.

UPDATE: I put in a trouble ticket for this bug. We'll see what they have to say when someone gets a chance to look at it.

[1/22/08] UPDATE: Adium 1.2.1 is out and it fixes this bug nicely. Don't know if it's something else, or my post, but I'm guessing that it was already in the works.