Memory Profiling on Solaris
OK, memory problems in C and C++ programs are a pain in the rear to find. When you have a massively multi-threaded app they aren't easy to find and that's a real pain. But I've read a nice little bit on a Google search about the watchmalloc library on Solaris.
On Solaris 8 (at least) a man watchmalloc gets you all the information you'll need to figure out how to use it. Basically, it's capable of looking at reads and writes from memory blocks that have already been freed and that's the core of the problem. Any access throws a trap that generates a core dump that can be dealt with in gdb.
To activate it, simply define a few environment variables before running the application:
setenv LD_PRELOAD watchmalloc.so.1
setenv MALLOC_DEBUG WATCH,RW
and then you run your app as normal. The WATCH checks for writes to freed memory blocks and the RW does the same for reads. Together, the man pages say this can slow down your app by 1000x - but it's worth it to find the bugger.
I'm running my code with this now and I'm happy to say I think it's found a problem in a third-party library already. That's great news as I can use a work-around to escape this problem.
I hope my luck holds out...