What's it all about, Alfie?

There are times when I wish pthreads had a two-pass read/write mutex. I guess I should be happy that it's got the read/write mutex and I can make something like a two-pass lock with retries. Basically, if I want to lock n objects in C++ using pthread mutexes, I have to try_lock the first one, if it fails, optionally wait a bit and try again. If it succeeds, then I can try the next one, and if it fails, back out the ones I've received, otherwise try the next one, etc. This scheme is nice in that it allows me to make the functionality I need, but it'd be great if pthreads had the ability to prepare_to_lock and then lock so you could make the two-pass system that would make it possible to make sure you could get everything you needed before locking anything up. My solution locks and unlocks, which can be a waste if one of the objects is write-locked, for instance.

But I suppose the fact that I can get most of what I want with a very little bit of code is nice. But it'd be great if they added that. Or maybe BOOST could make that out of other primitives.

The reason for this is that you get deadlocks when you try to lock multiple objects in a single, atomic unit, and that's what's been happening in my code. I have one section where a family of objects are locked for read, and another where a similar set are locked for writing. The problem I've hit is that the timing of the multiple threads in the system is very hard to pin down, and I don't want to use exclusive locks (mutexes) which would make things a lot easier, but would lock more than I want, and slow down performance in the server. So, I am looking to use the 'tryLock...' in the read lock part of the code and by putting in the 'try' get rid of the deadlocks. The only problem is that if the 'try' fails, the clients will have to retry, but that's already in the client code, so that might not be too bad. If I can get rid of these deadlocks, I'm making a big step forward for stability. We'll see how the tests go.

OK, I think I have the Tricky Bug on the ropes now. What was unique about yesterday was that I had an instrument with a single-point volatility curve with a value of 0.01 - which is exceptionally small. This lead to outlandish greeks and TVs - we're talking 30 digits. So, I got to thinking... it makes a lot of sense if the calculation libraries were able to come up with something for the results, but in so doing caused memory corruption problems that messed up the file descriptors, and therefore, the sockets.

These calculation libraries have outright crashed with bad data in the past, so I do a lot of pre-emptive checks to make sure that what I'm passing in has a good chance of making it all the way through. But if this data is just right on the edge, it's possible that the values are good enough to make it back, but internally, the calculation libraries are so badly damaged that they throw off the box.

To check this, I've added the checks for the small single-point volatility curves into the code. If this stops the problem then I'll know. I really think this is the problem.

Today I was hit by a very large number of calculation process stalls - nearly 200 in all. This points out that I hadn't really solved the problem with the changes I've made, and I need to get a little more creative on the problem and it's solution. So that's what the majority of today was about - getting creative.

From today's work I could see that the complete pass of a calculation was being done. First, the Are you ready? was being sent to the calculation process, and it was answering with Yup, I'm ready. Then the calculation set was sent, operated on, and the results returned to the server process, and then the added step of the Thank You being sent and the Welcome returned. All this worked every time.

The problem seems to be in the starting of the process the next time. Again, this doesn't happen all the time, and in fact, most times it's fine. But it's in the sending of the Are you ready? message that never seems to get to the calculation process that things hit a snag. So I created a new method on the server-side communication object: handshake() which does the sending and receiving of an int to (and from) the calculation process. This new method is now used in a lot of places in the server-side object, and in addition to the things it always did, it's got a retry based on a timeout of the response from the calculation process.

See... the calculation process should do this handshaking very fast, and so a simple 30 sec. timeout is about 30 times bigger than it needs to be. But after 30 sec. we can be sure that there's no way that the calculation process is going to answer. So we'll try it again. The question will become: what happens then?

If the retries are done and they all time out, then we'll know that it's not a timing issue, but a socket state issue. There are really only two things that can be at fault in this case: the timing of the data was such that the buffers were corrupted, or the socket is really disconnected when it thinks it's connected, and so we need to kill the process and start another.

Personally, I'm thinking it's the socket. I think it's gotten itself into a state where it thinks it's OK, but it's not. The problem is then that I need to kill the connection from the server-side, and then re-add the calculation bundle back to the queue so it's not lost to the world. I think this will be something I can do, but I need to know for sure that this is the problem and not a simple timing issue.

I've been dealing with this nasty communications bug in the code I'm working with (MarketMash server), and I thought I'd solved the problem by eliminating a possible problem in the serialization of a list of pointers, but I was wrong. It wasn't fixed.

The basic protocol for the communication of one unit of work between the server and the calculation engine goes something like this:

• the server sends the engine Are you ready?
• the engine responds to the server Yup, I'm ready
• the server sends a complete description of the calculation(s) to perform - serializing them out over the socket in a byte stream
• the engine gets the request, processes it, and streams back the response

then the process repeats itself over and over again. The problem manifests itself as the engine is waiting at the top of the loop for an Are you ready? message, and the server is waiting for something from the engine. So, to try and nail down that the response is getting sent to the server and received properly, I've modified the protocol to look like this:

• the server sends the engine Are you ready?
• the engine responds to the server Yup, I'm ready
• the server sends a complete description of the calculation(s) to perform - serializing them out over the socket in a byte stream
• the engine gets the request, processes it, and streams back the response
• the server receives the complete response and sends the engine Thank You
• the engine logs the Thank You and responds to the server Welcome
• the server receives the Welcome and logs if it doesn't get it

The goal of this is to make sure that I can see that the response is getting sent back to the server and received properly. If not, then the Thank You will not be received and I'll be able to tell that in the engine logs.

I sure hope this helps me track down what the problem really is.

I've been working on a very tricky little bug in a C++ server that has been pestering me for literally months. For the longest time I was convinced that the bug was not in my code, but was, in fact, in the linux kernel and it's handling of socket I/O. It was a compelling argument, and I'm not convinced yet that the kernel isn't making matters worse, but that's for later.

The problem manifested itself as this: one server process on a machine and five machines each with eight calculation processes all talking to the server process on the on 'main' machine. Things would be fine for a long time... then for no apparent reason, one of the calculation machines would have all it's calculation processes (all eight of them) stop communicating with the server process. Since each calculation process (32 in total) each connected to the server process, it seemed very unlikely that one of the calculation processes was effecting the others on the box. The Red Hat engineer agreed with me, as the processes were independent processes, and the only thing shared would be some part of the kernel on that box.

So I did a lot of debugging in the different processes, and it appeared that the problem was finally in the poll() method on the main machine. Everything pointed to this - but I had to back up a bit and then take a hard look at what I was doing and the assumptions I'd made to come to this point. Because I had the feeling that there was no way it was in poll().

What I started looking at was the possibility that it was not the discrete method calls, but that it was in the implied asynchronous functioning of the socket I/O. For example, the data was getting sent from the calculation process to the server process, but it was being done buffered. While it appeared that the write and read operations were completing, the write was really writing to a buffer, and that buffer would be sent when the kernel got around to it. Likewise, the read would be when sufficient data was received to let the kernel pass it to the process. So it might be possible for there to be a disconnect on the writing and reading.

I started looking at the serialization code and ran into the following code for serializing a vector of pointers:

    template< class T >
    void writePointers( Writer & aWriter, tList<T *> & aList )
    {
        aWriter << aList.length();
        tIterator<T *> lIterator = aList.begin();
        while (lIterator.hasNext()) {
            aWriter << *lIterator.getNext();
        }
    }

with a similar method for reading them in on the other end:

    template< class T >
    void readPointers( Reader & aReader, tList<T *> & aList )
    {
        aList.clear();
        int  lLength = 0;
        aReader >> lLength;
        for (int i = 0; i < lLength; i++) {
            T  *lNew = new T;
            aReader >> *lNew;
            aList.addBack(lNew);
        }
    }

so, in theory, we write the size and then each element, and the reader gets that size and then reads in that number. Pretty simple. Problem was, when I looked at it in light of the buffered socket I/O I realized that if the size changed after the writing of the size, then we were in trouble. Also, what about NULLs?

So, the change I made was to tag each element before transferring it. Basically, a handshaking was done within the list process - a code said "Hey, I'm sending a NULL", and that could be delt with by the receiver. Another code would be "Hey, here comes a good one", and a final code said "Hey, no more to send", and with this, I didn't have to send the size first, I could let the size be determined by the contents and not the size before the iterating.

So far, this has gotten rid of these stalls in the calculation processes. It's all about defensive programming. Assuming things really get us all into trouble.

UPDATE: unfortunately, it only took a few days and this bug popped up again. While I am happy with the change I made, it wasn't the core of the issue. Crud. Now I'm back to trying to find out why the communication is getting messed up.

OK, this is a pain I've run into several times so I wanted to jot it down here. Basically, if you have a process that is holding on to a socket the easiest way to find out what that is (on linux) is the fuser command:

  fuser -u -v -n tcp 6024

This will get all the processes that have the port 6024 open. No rocket science, but it's nice to have.

I have been having to move many of my applications at work to 64-bit address space because of the memory limitations on the default 32-bit. As a consequence, I've had to port CKit to 64-bit and I have to say, I like the way I've handled the build. Rather than have a multi-pass build from scratch like SQLAPI++, I've decided to have .o and .o64 files and have them built side-by-side. This means that we still use make and the dependency tool and that means rebuilds are faster. Not earth shattering, but nice and clean.

OK, found an interesting bug in the Sybase jConnect JDBC drivers for 5.5 and 6.0. Seems that the execute() command does not respect the transactional nature of the system. For example, if the SQL to execute includes a begin transaction and end transaction with some interesting SQL in the middle, then the execute() will not really commit the changes to the database when the command returns. In order to do that you need to use the executeQuery() - even if you don't care about the return values.

This nailed me for about three days because I was worried that data wasn't getting written to the database as it should. I implemented a read-after-write scheme and even that told me that things were indeed getting written only to see that they really weren't in the database. Why? Because for the transaction, the data was there and that meant that the reads were within the transaction and therefore "saw" the data. Yet, then the JDBC connection was dropped the data was all rolled back.

One more reason to hate JDBC.

It's taken months but the final trick to getting stability into the BBGServer has been to take all references to the Bloomberg API out of the main server and place it into a simple, small, single-threaded, C app that can be loaded and run from the server. The idea is that Bloomberg's API is itself not thread-safe. It can't handle several requests coming and going, and everything I tried to do with locking didn't change that basic fact.

So I created a simple C app that would open up a socket connection back to the server and wait for requests to process. When it received a request it sent it to Bloomberg via it's own connection and then waited for the response. When it got the response, it sent it back to the server and waited for another request on the socket. This was the ticket... get Bloomberg in it's own application with only one request pending at a time.

What's happened is that as soon as I put that into the code things stabilized. No more Bus Errors. No more crashes. It was a wonderful sight.

Whew! I've gotten a few very difficult memory issues taken care of in an app I'm working on. Interestingly enough, the points boil down to a few things:

• don't use mutexs in destructors if possible - this one got me on a few occasions
• delete pointers in maps obviously - and don't try to set the value part of the map to NULL after the delete. Use a while loop on the map's empty() method to get all the front() elements in the map.

Before I made sure of these few things I had lots of unusual and hard to pin down memory problems. Now that I've implemented these in all the classes of my app, things are running much smoother now.

You'd think that the following is a valid way to build a destructor:

    std::map<int, char*>	mMap;

    Egg::~Egg()
    {
        std::map<int, char*>::iterator     i;
        for (i = mMap.begin(); i != mMap.end(); ++i) {
            if (i->second != NULL) {
                delete i->second;
                i->second = NULL;
            }
        }
        mMap.clear();
    }

Oh, but you'd be wrong. The problem seems to be in the setting of the NULL into the value part of the map after the (char*) has been deleted. The way to get this to properly run in GCC 3.3.2 on Solaris 8 is to frame the destructor a little differently:

    std::map<int, char*>	mMap;

    Egg::~Egg()
    {
        std::map<int, char*>::iterator     i;
        while (!mMap.empty()) {
            i = mMap.front();
            if (i->second != NULL) {
                delete i->second;
            }
            mMap.erase(i);
        }
    }

Guess there's a good way and a not so good way to do STL things in GCC.