What's it all about, Alfie?

I got a new iMac this weekend and wanted to get sshd running on both the standard port 22 and port 2224 that I'll use in my firewall to redirect requests from outside. Common practice, really. I want to have the machine act normally within the network of my house, but I also want to be able to directly get to it from outside as well. The problem is that sshd is not run as a daemon on Mac OS X Tiger (10.4), it's run from launchd more like telnet.

There are configuration files - plists, that take the place if the xinetd conf files indicating what to run when a message comes in on a certain port. However, in this case, I needed to add an extra port to the configuration. If you look in the file /System/Library/LaunchDaemons directory you'll find a file called ssh.plist. In it's original form it looks something like this:

    <key>Sockets</key>
        <dict>
            <key>Listeners</key>
            <dict>
                 <key>Bonjour</key>
                 <array>
                     <string>ssh</string>
                     <string>sftp-ssh</string>
                 </array>
                 <key>SockServiceName</key>
                 <string>ssh</string>
            </dict>
        </dict>

It needs to be modified to look like this:

    <key>Sockets</key>
        <dict>
            <key>Listeners</key>
            <array>
                <dict>
                    <key>Bonjour</key>
                    <array>
                        <string>ssh</string>
                        <string>sftp-ssh</string>
                    </array>
                    <key>SockServiceName</key>
                        <string>ssh</string>
                </dict>
                <dict>
                    <key>SockServiceName</key>
                        <string>sshalt</string>
                </dict>
            </array>
        </dict>

The key is that for launchd the value for Listeners can be a dictionary or an array of dictionaries. It was in the man page for launchd and that's what lead me to the breakthrough. You also need to add a line to /etc/services for the new service 'sshalt' (ssh-alternative) which references the port number that you want to monitor as well.

Now I can control SSH with the standard tools, it won't run until a user comes to the box and it's all in-line with the Mac OS X way of doing things.

OK, found an interesting bug in the Sybase jConnect JDBC drivers for 5.5 and 6.0. Seems that the execute() command does not respect the transactional nature of the system. For example, if the SQL to execute includes a begin transaction and end transaction with some interesting SQL in the middle, then the execute() will not really commit the changes to the database when the command returns. In order to do that you need to use the executeQuery() - even if you don't care about the return values.

This nailed me for about three days because I was worried that data wasn't getting written to the database as it should. I implemented a read-after-write scheme and even that told me that things were indeed getting written only to see that they really weren't in the database. Why? Because for the transaction, the data was there and that meant that the reads were within the transaction and therefore "saw" the data. Yet, then the JDBC connection was dropped the data was all rolled back.

One more reason to hate JDBC.

A few days ago, I noticed that sparky was dead. I don't like this kind of surprise, but there's not a lot I can do about it other than to try and find out the problem and fix it.

The last time I had a hardware fault it turned out to be a dead hard drive. This time, I took one of the internal drives out and placed is in a simple 911 case, so as to reduce the load on the power supply and remove one of the heat generating devices in the case. This didn't help. But I was happy to see that the drives booted. That's something.

The next thing was a Google search for Watchdog resets. Interestingly, the first hit was from the Ross page talking about the possibility that there might be a fault in one of the CPUs. Thankfully, I had the SuperSPARC processors that I had originally used in the box, and so I swapped out the CPU modules - going from four HyperSPARCs to two SuperSPARCs. Bummer of a downgrade. But it seems to be stable.

Since I really only use sparky as a server, and don't really need the speed as much as I need the stability, it's a decent trade. I really didn't want to put a lot of money into this box, but I need to keep it running for a while longer.

OK, I'm the first person to admit that I enjoy the fact that I understand many (but certainly not all) of the systems in use at work. It makes it easier to understand where the ones I build fit in and what jobs are best done by other systems. But there's a dark, ugly side of this... the 1:30 am phone call.

In the past week I've been called three times at 1:30 am to help the operators figure out problems they are having with systems. Every time, it was an upstream system that had the problem, it was just that when it got to my system it was clearly apparent. So they called me. Unfortunately, all I was able to do was to point them to the systems that had problems - which they already knew from other sources. So I'm not really helping a lot, but I'm certainly wide awake and that's no fun.

I have put together web pages to assist in figuring out what system is at fault, and what might be done to clear up the problem. Of course, these aren't foolproof documents as different kinds of problems keep popping up and we have to adapt to the new data sources, etc. But it's a darn good start, and far more than most developers give their support staff. But if they don't read it, it's not going to help them, and it's not going to stop the 1:30 am phone call.

So now I can tell them to read the web page and call me back if it isn't solved. But that's still not stopping the calls. Once they are used to me having the answers for them they are going to skim the docs and call me. It's a little disappointing and very tiring.

It's taken months but the final trick to getting stability into the BBGServer has been to take all references to the Bloomberg API out of the main server and place it into a simple, small, single-threaded, C app that can be loaded and run from the server. The idea is that Bloomberg's API is itself not thread-safe. It can't handle several requests coming and going, and everything I tried to do with locking didn't change that basic fact.

So I created a simple C app that would open up a socket connection back to the server and wait for requests to process. When it received a request it sent it to Bloomberg via it's own connection and then waited for the response. When it got the response, it sent it back to the server and waited for another request on the socket. This was the ticket... get Bloomberg in it's own application with only one request pending at a time.

What's happened is that as soon as I put that into the code things stabilized. No more Bus Errors. No more crashes. It was a wonderful sight.

OK, I'm the first to admit I'm a workaholic... but I like that as it gives me something to do each day. But I've just come off a vacation and am so ready to go back to work it's wonderful. I'm feeling refreshed, happy and excited about the same old problems that I've been dealing with for the last several months at work - but at least now, they don't seem bad. It's all about the recharge.

Whew! I've gotten a few very difficult memory issues taken care of in an app I'm working on. Interestingly enough, the points boil down to a few things:

• don't use mutexs in destructors if possible - this one got me on a few occasions
• delete pointers in maps obviously - and don't try to set the value part of the map to NULL after the delete. Use a while loop on the map's empty() method to get all the front() elements in the map.

Before I made sure of these few things I had lots of unusual and hard to pin down memory problems. Now that I've implemented these in all the classes of my app, things are running much smoother now.

You'd think that the following is a valid way to build a destructor:

    std::map<int, char*>	mMap;

    Egg::~Egg()
    {
        std::map<int, char*>::iterator     i;
        for (i = mMap.begin(); i != mMap.end(); ++i) {
            if (i->second != NULL) {
                delete i->second;
                i->second = NULL;
            }
        }
        mMap.clear();
    }

Oh, but you'd be wrong. The problem seems to be in the setting of the NULL into the value part of the map after the (char*) has been deleted. The way to get this to properly run in GCC 3.3.2 on Solaris 8 is to frame the destructor a little differently:

    std::map<int, char*>	mMap;

    Egg::~Egg()
    {
        std::map<int, char*>::iterator     i;
        while (!mMap.empty()) {
            i = mMap.front();
            if (i->second != NULL) {
                delete i->second;
            }
            mMap.erase(i);
        }
    }

Guess there's a good way and a not so good way to do STL things in GCC.

I was doing some malloc checking on Solaris today and noticed that I was getting a malloc trap on the auto-detect Sybase library version method of SQLAPI++. You can have the library determine the version of the library in use or you can specify it manually. Up to this point, I've been doing the auto-detection with the code:

    mDBConnection.setOption("CS_VERSION") = "Detect";
    mDBConnection.setOption("CS_APPNAME") = "myApp";
    ...
    mDBConnection.Connect(server, user, password, SA_Sybase_Client);
    ...

but in the Connect() method the malloc checker saw a problem. So, now I'm trying to set the version I know I'm using (12.5) with the code:

    mDBConnection.setOption("CS_VERSION") = "CS_VERSION_125";
    mDBConnection.setOption("CS_APPNAME") = "myApp";
    ...
    mDBConnection.Connect(server, user, password, SA_Sybase_Client);
    ...

Tests are looking much better so far, and that's really encouraging.

OK, memory problems in C and C++ programs are a pain in the rear to find. When you have a massively multi-threaded app they aren't easy to find and that's a real pain. But I've read a nice little bit on a Google search about the watchmalloc library on Solaris.

On Solaris 8 (at least) a man watchmalloc gets you all the information you'll need to figure out how to use it. Basically, it's capable of looking at reads and writes from memory blocks that have already been freed and that's the core of the problem. Any access throws a trap that generates a core dump that can be dealt with in gdb.

To activate it, simply define a few environment variables before running the application:

    setenv LD_PRELOAD watchmalloc.so.1
    setenv MALLOC_DEBUG WATCH,RW

and then you run your app as normal. The WATCH checks for writes to freed memory blocks and the RW does the same for reads. Together, the man pages say this can slow down your app by 1000x - but it's worth it to find the bugger.

I'm running my code with this now and I'm happy to say I think it's found a problem in a third-party library already. That's great news as I can use a work-around to escape this problem.

I hope my luck holds out...

I had need to convert hex numbers from a data file I was loading into a database into decimal numbers for the database. I figured that there had to be some simple way to do this - after all, we've got decades of Unix here. And sure enough - it came through for me like a champ!

The program is dc and it's easy to get it to convert a hex number to decimal:

  echo "16 i FF p q" | dc

returns:

  255

I simply put the variable I need into the command:

  dec=`echo "16 i ${var} p q" | dc`

and that's it! Nice.