Archive for the ‘Coding’ Category

Leak on SQLAPI++

Thursday, June 17th, 2004

I was doing some malloc checking on Solaris today and noticed that I was getting a malloc trap on the auto-detect Sybase library version method of SQLAPI++. You can have the library determine the version of the library in use or you can specify it manually. Up to this point, I've been doing the auto-detection with the code:

    mDBConnection.setOption("CS_VERSION") = "Detect";
    mDBConnection.setOption("CS_APPNAME") = "myApp";
    ...
    mDBConnection.Connect(server, user, password, SA_Sybase_Client);
    ...

but in the Connect() method the malloc checker saw a problem. So, now I'm trying to set the version I know I'm using (12.5) with the code:

    mDBConnection.setOption("CS_VERSION") = "CS_VERSION_125";
    mDBConnection.setOption("CS_APPNAME") = "myApp";
    ...
    mDBConnection.Connect(server, user, password, SA_Sybase_Client);
    ...

Tests are looking much better so far, and that's really encouraging.

Memory Profiling on Solaris

Thursday, June 17th, 2004

OK, memory problems in C and C++ programs are a pain in the rear to find. When you have a massively multi-threaded app they aren't easy to find and that's a real pain. But I've read a nice little bit on a Google search about the watchmalloc library on Solaris.

On Solaris 8 (at least) a man watchmalloc gets you all the information you'll need to figure out how to use it. Basically, it's capable of looking at reads and writes from memory blocks that have already been freed and that's the core of the problem. Any access throws a trap that generates a core dump that can be dealt with in gdb.

To activate it, simply define a few environment variables before running the application:

    setenv LD_PRELOAD watchmalloc.so.1
    setenv MALLOC_DEBUG WATCH,RW

and then you run your app as normal. The WATCH checks for writes to freed memory blocks and the RW does the same for reads. Together, the man pages say this can slow down your app by 1000x - but it's worth it to find the bugger.

I'm running my code with this now and I'm happy to say I think it's found a problem in a third-party library already. That's great news as I can use a work-around to escape this problem.

I hope my luck holds out...

Unix Hex-to-Decimal

Tuesday, May 4th, 2004

I had need to convert hex numbers from a data file I was loading into a database into decimal numbers for the database. I figured that there had to be some simple way to do this - after all, we've got decades of Unix here. And sure enough - it came through for me like a champ!

The program is dc and it's easy to get it to convert a hex number to decimal:

  echo "16 i FF p q" | dc

returns:

  255

I simply put the variable I need into the command:

  dec=`echo "16 i ${var} p q" | dc`

and that's it! Nice.

Java Threads

Wednesday, March 31st, 2004

OK... I've decided that I like pthreads in C++ a lot more than the threading model in Java - at least if you're not totally familiar with the Java model. One of the classes that I've been dealing with this morning is a FIFO Queue. It's thread-safe because those methods that need to be synchronized to ensure single instance access to the data are, and those that might require multiple thread access are left unsynchronized. This seemed reasonable because in the methods, where necessary, I had a synchronized block on the variable of import. I was so very wrong.

The problem is that I had created two mutexes and therein lay my problem. The confusion I had was that a synchronized method with a synchronized block in it containig a wait(); would block all further access to that instance by other threads. The effect of having a synchronized method is really no more than encasing the entire method contents in a synchronized block on this. So... if the wait() was on this then I could simply synchronize the method and the wait() would release the lock and let other threads message this instance.

This made the code a lot easier as the push() method then simply did a notifyAll() when placing something of interest on the queue. Again, if the push() method was synchronized, then the control would pass to the waiting threads and then back to the push() thread and things would just work out.

So really, it's when I didn't understand the actual mutexes in place in the Java code and tried to work up something to do what was necessary that I got into trouble. Now I just lock on the queue instance itself and the synchronized methods do all the hard work for me.

Passing Data to Threads

Tuesday, March 30th, 2004

One of the things that I seem to be doing on a somewhat frequent basis is the re-inventing the wheel. Case in point: I had a section of code in a project that was single-threaded, could easily have been multi-threaded and all I really had to do was to handle getting data into the threads and the processed data out.

I thought about it and built something that should have worked, but had totally forgotten that I'd built a thread-safe FIFO queue that would be a wonderful tool in this situation. In the original version, I tried to handle the passing of each bit of data from the controller thread to each worker thread and that turned out to be a major hassle. When I changed focus to the FIFO queue, it was exceptionally easy and fast.

Next time, I'm really going to try to look at the components that work well and build around them. It was amazing the difference.

Great CLASSPATH Trick

Thursday, March 18th, 2004

A lot of Java projects have several JARs you need to put into the CLASSPATH in order to get things to compile properly. Whether you're using jikes or javac, you need to build up the CLASSPATH so that the user's environment isn't expected to provide the correct CLASSPATH for building.

Using gnu make, and assuming that all the JARs are in a single directory - as they usually are, you can do the following:

empty:=
space:= $(empty) $(empty)

JARS:= $(shell find ../libs -name *.jar)
CLASSPATH:= $(subst $(space),:,$(JARS))
list:= find src -name *.java | grep -v /testers/
SRC:= $(shell $(list))
CLASSES:= $(SRC:%.java=%.class)

my.jar: $(SRC)
        javac -classpath $(CLASSPATH) $?
        rm -f my.jar
        jar cf my.jar `find src -name *.class | grep -v /testers/`

This allows you to simply place JARs in the lib directory and they will qutomatically be picked up into the CLASSPATH. Also, this compiles all the java source files that have changed with respect to the jar and then creates a new version of that jar.

UPDATE

I have completely updated the makefile for BKit and here it is:

#
#  This is a very simple makefile for BKit.
#

#	If we're on a platform with jikes, use it as it's faster
ifeq ($(shell uname),Darwin)
JAVAC:=jikes -bootclasspath /System/Library/Frameworks/JavaVM.framework/Classes/
classes.jar:/System/Library/Frameworks/JavaVM.framework/Classes/ui.jar -extdirs
/Library/Java/Home/lib/ext:/Library/Java/Extensions:/System/Library/Java/
Extensions -nowarn
RMIC:=rmic
else
JAVAC:=javac -J-ms32m -J-mx32m
RMIC:=rmic
endif

#	get the CLASSPATH we'll need
CLASSPATH:=/usr/local/MQClientV2:/usr/local/VantagePoint/Jars/vpJava2JFC.jar:
/usr/local/jConnect/4.2/classes:/usr/local/jep/jep.jar

#	get all the Java files that need building
list:= find one -name *.java | grep -v /ado/
SRC:= $(shell $(list))
CLASSES:= $(SRC:%.java=%.class)
#	get all the classes that need rmic-ed
rmics:= find one -name *.java | xargs -J % grep -l UnicastRemoteObject % | 
	sed -e 's/.java//' -e 's:/:.:g'
REMOTE:= $(shell $(rmics))

all: .compile

jar: .compile
	@ rm -f bkit.jar
	@ jar cf bkit.jar `find one -name *.class | grep -v /testers/ | 
	grep -v /ado/`

.compile: $(SRC)
	@ $(JAVAC) -classpath .':'$(CLASSPATH) $?
	@ $(RMIC) -classpath .':'$(CLASSPATH) $(REMOTE)
	@ touch .compile

install-applet: jar
	@ cp bkit.jar $(HOME)/Sites/applets/classes/

clean:
	@ rm -f bkit.jar
	@ rm -f `find one -name *.class`

We get all the rmic classes as well as getting done just what we need. Not bad.

Java Serialization Loops

Friday, March 12th, 2004

In general, I appreciate the work the Java folks have done to make remote procedure calls easy. And to a large part, they have done a pretty good job. The single weakness that I can find is their misunderstanding of their own data structures.

For example, say I have two HashMaps in JDK 1.3.1. Each has a single key/value pair: a String "other" as the key and a HashMap (the other) as the value. It seems pretty obvious that if you're looking at the structure you'd see one, move to the other, and see the first again and stop. But Java isn't that smart. It creates an infinite loop trying to evalutate the HashMaps as opposed to seeing that they are in fact one in the same.

This means that to serialize the example structure I have to create four objects - two of each with the "other" link disabled in two of the four. Now, no matter where I am, I can "see" the relationship, but it's not circular.

This is a pain and there's no reason for it as the serialization would be able to identify the objects and provide placeholders for de-serialization. But that's life with Java...

Code Rot

Thursday, March 11th, 2004

I'd like to think that I do good design. But even so, things change. The needs of the system changes, and the system responds. So it's really unavoidable that you get into situations where code is old and while it's working, it's not working well. You've got Code Rot

Yesterday it became clear that the large system I'm working on had a fundamental limitation due to the size of the dataset it's working with. I couldn't let it remain like it was. So I knew it was time to dive into the code that was at least a year old and fix it. But it wasn't going to be a simple fix - it was going to be a massive re-write.

Amazingly, I re-wrote the entire system in less than 24 hrs. The reason it was so easy was that I knew there were a lot of simplifications that could be made to the way in which events were handled in the code. These simplifications meant about a 30% memory savings as well as lots of speed gains as things weren't done over and over again for similar data. In the end it's amazing how nice and clean it is.

It'll probably last another year! 🙂

Cross-Platform #defines

Friday, March 5th, 2004

I've been doing a lot of cross-platform work - linux, solaris, and Mac OS X and I figured out the important machine-dependent #defines that are used on each. Thankfully, all three platforms are using gcc, with Linux being the only 2.95.x-based compiler.

Anyway, here they are:

Machine Vendor CPU
Mac __APPLE__ __MACH__
Solaris __sun__ __sparc__
Linux __linux__ __i386__

So now that they are in the journal, I won't have to go hunting for them on the scrap of paper I used to use. That was just silly.

GKrellM and perfmeter

Thursday, March 4th, 2004

For the production, and development for that matter, machines that I use, I like to run gkrellm - a GTK+ based monitoring app. It's nice and informative and it's got a very small footprint compared to, say, top. I was looking for a version for Solaris 8, but realized that there was already something on Solaris that's almost exactly what I wanted - perfmeter.

OK, perfmeter doesn't do all the skins, and it's not got the extensive list of modules for monitoring email, temperature, etc. But that's not what I use it for anyway. I'm primarily interested in CPU and memory usage, and while perfmeter isn't really great at the last of these, it is decent with swap and context monitoring. And it's on every Solaris box. Nice.

Just goes to show - there's no reason for new tools if you've got what you need. Go with it and get on with solving the real problems of the day.