What's it all about, Alfie?

Here's something that I guess I really finally realized today - you really have to have CSS for even the simplest web pages if you want it to look similar on multiple platforms and browsers. Case in point: I love Coda from the Panic crew for it's clean interface and it's ability to really nicely maintain small to medium-sized web sites. I know it's got limits - it's not a Tomcat build system, for instance, but for what it can do, it does it very well. Well... I was initially looking at the features and thought "Yeah, I can use that... and that... but what's this CSS editor? Well... no harm to have it in there."

Silly me.

I was doing a little, and I mean little, page for work - basically re-working it for a new Perl interface from a very complicated PHP/Java page and wanted to add a little color to the tables of data to make them a little easier to read. So I spent a bit of time just dealing with the table tags and the attributes set there. Looked OK to me in Firefox on Linux. I checked it all in, thought myself done, and went on with my day.

Coming back into my cube later in the day I thought - Hmmm... what's it look like in IE? Ah... now we see that I had remembered from the last style changes I had made - it looks crummy in IE when using the table tags. I needed to simply give in and make the style additions to the CSS for the page and get the two platforms, two browsers looking reasonably similar. Not horribly hard, but it means making a change, then hitting the refresh on three browsers on two machines. Not rocket science, but annoying to get it all done right.

Then I flashed back to Coda... Ah... now I see the reason for it's inclusion. Most folks are doing the same thing I was doing for work and they needed to have that - even on the small sites. On most of the small sites I have I haven't cared a lot about the style on the different browsers and platforms, but I can see that this is changing and it's going to become more important as time goes on. Even the simplest pages will need to have CSS written for them to make them look the same on the two browsers. Looks like I need to spend more time looking at CSS. Not exactly what I think of as 'fun', but it's something I can see now is going to be more important for even the simplest pages.

Today I wanted to clean up a problem that has been sitting around for a long time. One of my price servers has had C++ and Java clients, but no Perl client. I have been holding off on the Perl client for a while because the API uses binary data for the price structures and I was worried about the complexity of dealing with that in Perl. Wow, was I ever wrong.

I took a similar API that had been written for me, and changed all the aspects of how the requests were sent, and how the data was returned. But once I had the data, I was still at a loss as to how to deal with it effectively. Then I came across pack() and unpack() in my Googling. What an incredible simplification this made for me.

I could simply point the unpack() to the start of the individual components in the fixed-length binary record and it handled all the details for me. Amazing. I was thinking that it was going to take me several iterations to get the byte order right and make sure all the numbers got passed back properly, but in actuality it took me a few minutes to get the first data type going and then a few seconds to do the others. Really quite an incredible feature.

I was then able to re-write a web page which was a PHP page shelling out to a Java program to hit the price server with a simple cgi perl script and it'll run much faster with much less load on the web server. Really outstanding. I am beginning to like Perl. It's got a lot of power that isn't obvious until you start to really exercise it.

Here's what I'd love to see: a JVM plugin for Apache, like PHP, for combined Java/HTML/PHP/etc. Think about it - this is what I don't like about using Java on a web server - you have to use Tomcat (or something like it) and then you have to set up the project with the directory structure, build it, restart it or tell it to reload itself. It's a mess. What would be better? I'll tell you, what.

Look at PHP. Not as a language but it's integration with Apache. You can place PHP files in any directory in the web server. Any directory. You can put the static HTML next to PHP - or even both in the same file. Imagine the same with Java. Wouldn't that be nice? There's the Bean Shell for the same kind of thing for CLI environments, but nothing for web pages.

There are J2EE/JBoss fans that will say how easy it is to make a simple web site with JBoss, etc. But I have to laugh at them. Make something like the JSP pages but with the CLASSPATH specified in some config file or as an option in the tag that starts the Java code section of the page. Then you can simply Page.out.println() the HTML you want. Sure, it's not the same as Tapestry, but a lot of the time I don't the complete MVC for a web page. Many times what I want is a simple (and I mean simple) web page that accesses a service/server from within Java. Simply make a connection to a server, get the data, and make a page of it and then quit. Nothing complex. But it's Java.

It's interesting that PHP has the classes, the add-ins, the connections to databases, etc. It's got all that you'd need for simple (and maybe not-so-simple) pages. But you can drop the pages anywhere you want and you can edit and view immediately. No need to rebuild, restart, etc. I wish there was something for Java like PHP.

I don't want to re-write a different version of the Java classes into PHP just for a few simple pages. I can make that happen a lot of different ways - calling out to a Java app is one simple way to do it. Likewise, I don't want to set up a Tomcat/Tapestry instance just to vend a few pages. That's the key - just a few simple pages. No major web app. Just a few pages.

Today I finished up a lot of little things at work... got some hardware re-tasked so that we can decommission some, got a few little things updated, etc. Then this afternoon I spent some time adding in some simple mathematical operations to the BKTable - add, subtract, multiply, divide, transpose, inverse - enough so that I could then add these methods to the BKJEP parser and then tables could model matrices and do some simple linear algebra.

Nothing amazing, I'll agree, but it's nice and it adds a level to the JEP parser that has been missing up to now.

There is a lot to like about the threading model in Java. It's easy to make threads, it's decently fast, and it's stable. One thing that I wished they'd put into the JVM - or at least into javac like all the other 1.5/1.6 compiler 'hacks', is the idea that synchronization on a setter can be controlled by the instance variable itself. What I'm asking for is an automatic way to have javac create thread-safe setters and getters by simply naming the ivar.

The most obvious way of making a thread-safe setter is this:

    /**
     * This is the standard setter for the temperature
     * of this instance.
     */
    public synchronized void setTemp(double aTemp)
    {
        _temp = aTemp;
    }

but the downside is that this means that if this instance is in a highly threaded environment, or there are other methods that take a significant amount of time to process, the ability to 'set' the temperature is dependent on the other synchronized methods on this class.

Sure, you can make an object to lock on and it might look like this:

    private     double      _temp = 0;
    private     String      _tempLock = "TempLock";
    
    /**
     * This is the setter for the temperature uses an
     * additional ivar - a String, to protect multiple
     * threads changing the value at once.
     */
    public void setTemp(double aTemp)
    {
        synchronized (_tempLock) {
            _temp = aTemp;
        }
    }

but then you have all these extra ivars that you have no real need for and no need to really put into the setter other than as a thread-safety mechanism.

Some will argue that if I had used a Double and not a double I could synchronize on the ivar itself. But it's not really safe doing that. You'd have to be careful of the null value and then what if you changed the reference within the synchronized block - you'd no longer be protected. No, there needs to be the ivar itself - allowing null, and then some mechanism to ensure thread safety on a more finer-grained level than locking the entire instance.

They need to have something that does the obvious things for an ivar of a given type. You want to save typing? Make something like this:

    /**
     * These create the setter and getter for the ivar
     * '_temp' because the need to lock is implied
     * on the setter and typically unnecessary on the
     * getter (unless it's a long).
     */
    public threadsafe setTemp(double aTemp) : private double _temp;
    public threadsafe getTemp() : private double _temp;
    
    /**
     * These versions would create 'protected' versions
     * of the getter and setter methods.
     */
    protected threadsafe setTemp(double aTemp) : private _temp;
    protected threadsafe getTemp() : private _temp;

    /**
     * This creates both setter and getter for the ivar
     * '_temp' because the structure to create the two
     * is clear from this information alone.
     */
    public threadsafe accessors Temp : private double _temp;

The information to build the code for the setter and getter is clearly specified in the above examples. In the first set, the ivar _temp is clearly private, a double and the names of the methods to build are clear. There's even the argument type - which is really redundant, if you think about it.

A possibly even simpler solution would be to combine the definition of the ivar and the scope of the setter and getter in one statement - like the last example. That way, you would know the ivar's name, type, scope, and then the scope of the setter and getter. You could even allow one to use a PropertyChange system with listener registration, etc. This is not that hard, and I've used it in tons of classes. The clutter it'd save would be far far more than the collapsed for-loop.

Anyway, today I did a lot of the 'mutex' variables because I needed to remove as many synchronized methods as possible on a class I was working with. Virtually every one of them was a setter, and it was annoying to think that they made a goofy collapsed for-loop in 1.5, but they left this little nugget out.

Amazing. I think it shows to me that defensive programming is not heavily exercised by the Java Developers. Otherwise, they too, would be sick of making thread-safe but not lock greedy accessor methods.

Back in late 1999 I was doing a little work at First Chicago NBD (BankOne) on linux as the development platform of the future. I had made a position as the Head of the Technical Architecture Group in Capital Markets that Java/CORBA/Sybase with a bean server like Jaguar CTS would be the way to create apps in the future. In order to prove my point, I got a nicely powerful box (at the time) and started doing some work for a project in the commercial card services division. What I found was that there were a few things that needed to be done in order for the linux desktop to fit into the Bank's infrastructure easily.

Most things worked pretty well right away. Email, at the time, was SMTP/POP3 so that wasn't a problem... the only biggies were source code repository and database access. We were using Microsoft Visual SourceSafe, and there turned out to be SourceOffSite that runs on linux and accesses the Visual SourceSafe repository just fine. The final step was access to Sybase at the C library level.

FreeTDS was a great open-source project that did most of the TDS spec for Sybase and SQL Server. It wasn't 100%, but it was all that was necessary to get the job done, and that meant that I could get access to the Sybase databases from my linux box. I used it and didn't have to look back. With it, I was able to build SQSH, and the access from apps was pretty simple.

Fast-forward to this morning and I was talking to a friend about something completly unrelated and another guy stopped by asking about the JDBC URL for jTDS.

"jTDS?" I asked.

"Yeah, it's a JDBC driver based on the TDS protocol for Sybase - by the guys that wrote FreeTDS"

Wow! I talked to him about my previous experience and we laughed about how things come around again (and again). Turns out, in his tests, and he's not alone, jTDS outperformed Sybase jConnect by a factor of better than 2! That's impressive. So, after I was done with the updates we were talking about, I went back to my cube and got jTDS and tried it. There were a few little changes to my code - mostly because the URL is different and I wanted to make it optional to use jConnect or jTDS. But in about 15 mins. I had something working and the tests looked great. Amazing.

Highly recommended.

Today I noticed that DataGraph had gone to version 1.4 and there was a new forums site. So I decided to register a version, get the Framework, and request the features that I've wanted to have for a long time: contour and heat graphs. 2D graphs, but based on 3D data. These would be the types of graphs that I really want to put into my Potentials code that I started building a long time ago. The simplicity and design of DataGraph is really quite amazing. While it's not exactly what I would have done, it's certainly a new and interesting take on the data plotting application and it's got everything that you could ask for in 2D plotting - save these two graph types.

Certainly, something like VantagePoint would be great, but that's Java, and it's $25,000 for a license. Additionally, there's no application with VantagePoint - you have to code up everything yourself. Now, I've gotten pretty good at using VantagePoint, but still... it's not the easiest thing for creating new plot types as you have to build an entire testing framework to see if you have the parameters set right for the graph you want. It's got great documentation, but still... to have an application would be a plus.

That's what I was hoping for with DataGraph. We'll see if he's interested in making DataGraph Plus and charging $200 or so - making it fall between the $30 DataGraph and the $1200 DataTank. While DataTank would be nice, I'm just not doing that much visualization to justify the cost. It'd be nice to be able to make a few big improvements in the output visualization of the fields. That would be a lot of fun.

Yesterday I was putting in some new hardware for an app that I have that's been running nicely for a while. Unfortunately, the machine is old, getting underpowered for the task at hand, and it was time to update the hardware. At the same time, I wanted to update the message queue system that it was using. About once a month (or so) the existing message broker (Sun ONE Message Queue 3.0.1) will get into a confused and I'll have to restart it. This is bad as it's in a place in the infrastructure where it means that I'll have to restart several things because of the loss of the connection to the message broker.

Now that Sun has open-sourced the message broker, calling it OpenMQ, I decided to update to it and see if it was any better. I got the latest OpenMQ binaries (ver. 4.1) and then unpacked them using jar. There were precious little in the way of instructions, and in the previous version I had used (3.0.1) no configuration was needed. But that would prove to be a mistake for me.

The proper was to configure OpenMQ 4.1 given that it's installed into a directory called $INST is to first update the $INST/mq/etc/imqenv.conf file to include the proper value for IMQ_DEFAULT_JAVAHOME to the location of JDK 1.5.0. For me this was simply:

    IMQ_DEFAULT_JAVAHOME=/usr/local/jdk1.5.0

then I wanted to have the default memory size set to at least 4GB, so in $INST/mq/bin/imqbroker look to the line around line 82 that looks like:

    _def_jvm_args="-Xms32m -Xmx128m -Xss128k"

and change it to what you want. In my case, I changed it to:

    _def_jvm_args="-Xms32m -Xmx4096m -Xss128k"

since I was running a 64-bit JDK 1.5.0 this was going to work out nicely.

The next thing I needed to do (but didn't do at the time) was to set the maximum individual message size to unlimited. The default is 70MB, but there are some messages in my system that are very large. The place to do this is in $INST/mq/lib/props/broker/default.properties on the line that starts with:

    imq.message.max_size=70m

and change it to what you want it to be. A value of -1 means unlimited, so I set:

    imq.message.max_size=-1

next, for those topics (queues) that are auto-created, there are a few parameters in the section on 'destination based topics' that you might want to use:

    imq.autocreate.destination.maxTotalMsgBytes=-1
    imq.autocreate.destination.maxNumProducers=500
    imq.autocreate.destination.maxBytesPerMsg=-1

the first and last are basically saying "make no limits on the size of an individual message, or the total size of all messages". The middle one is saying that you might want to have a queue that has a lot of producers (injectors) and one consumer (reader), and the default is 100 and it's OK for most folks, but I wanted to make sure that we didn't run into problems if we had clients directly hitting the broker. Now we should be all done with the configuration.

There is an /etc/init.d/imq script in the $INST/mq/etc/init.d directory, but I didn't use that one as I already had one that was working from the 3.0.1 version. All I needed to do was to change the installation location of OpenMQ (the $INST directory) and then it was ready to go. It was already using JDK 1.5.0 and on this machine that means 64-bit.

Add it to the chkconfig start-up on levels 3, 4, and 5 and then start it with /etc/init.d/imq start and all should be just fine.

But that's not how it really happened for me. Here's the problems I ran into to come up with what should have been done in the first place.

First, if you miss the definition of the IMQ_DEFAULT_JAVAHOME variable it means that you have to edit the imqbrokerd script. This was not unusual as I was already in there for the maximum size, so I added that in there and didn't think a thing of it.

Next, if you don't remember to set the maximum message size, then changing it in the default.properties file is not going to be good enough. You have to remove all the data for the instance as I'm convinced that the configuration of the queues is stored in that persistence and the default.properties file is only read when there's nothing in the instance to base the queues off of. So, if you start it without the maximum message length what you need it to be, then you're going to have to shut down IMQ, then remove the entire directory $INST/mq/var/instances/imqbroker. If you don't remove the imqbroker directory I don't think the changes are going to properly work. Thankfully, all my code makes the queues automatically so wiping out the existing configuration is no big deal.

Also for auto-creating queues, you have to remember the two critical size configuration parameters or else you're going to have those queues in trouble when they try to send through large messages. Sure, the 100 producer limit on auto-created topics is reasonable for most installations, but I had talked to another developer here using OpenMQ 4.0 and he had to set it to 500, so I figured that while I was in the config file, I'd up that limit too.

Also, it's important that your app uses the $INST/mq/lib/imq.jar and $INST/mq/lib/jms.jar or else you can have connection problems. Specifically, if going from a 3.x imq.jar to a 4.x IMQ, you're going to get connection errors if you're not using the 4.x imq.jar. So just be safe, get the one with the OpenMQ distribution you're using.

Today there have been a lot of problems with a system that we have in the shop that takes a price feed. It seems that this vendor's custom code to interface their system to the Bank's price feed was having stability problems. Specifically, it's a Java process using JNI that was blowing out of 1.5GB of RAM allocated to the 32-bit process. Having worked with the Bank's price feed for a few projects I know the symptoms of this kind of problem and how to fix it. I'm not going to say this is the only way to fix the problems, but I've tried a lot of things before finding this solution, so I know what's not going to work to a large extent.

So... the vendor throws up it's hands and asks us for help. Earlier, I had sent one email message as I saw so many flying around about these stability issues. I said "Hey, I've got it working, I know it's hard, but there is a way to make this work." The response I received was "Thanks, but we're going to try to save this design and impact the code as little as possible." Normally, I'd agree with them on the minimal change issue, but this time I knew that a minimal change was not going to work. It wasn't a hard change to make - less than a few hours, but it was a fundamental change in the way they were processing the data.

You see, the data is coming on on a (virtually) single-thread calling an onMessage() method to pass in the message containing the data. Because of the way the Bank's price feed is written, you have to make sure that you take as little time as possible in dealing with this message and return control to the calling thread as soon as possible. This means you can't do anything other than throw it on a queue and then have some other thread(s) taking it off the queue and doing the real processing.

So we get into this phone meeting and they start to say what they've done and tried. They quote some timing figures for how long they take on the processing of an event. This doesn't matter a bit. It's how fast you return control to the onMessage() caller that's going to make or break this system. So on and on they go... I finally say "Here's what you need to do..." and outline what they need to do to make it work.

They say "That may work for you, but it can't work for us."

Remember now, they emailed us throwing up their hands for help on the solution to the problem. So this attitude was more than a little shocking coming from the people asking for help. I was only suggesting a way to queue/dequeue the messages - nothing that couldn't be retro-fitted into their code (I had it on a print-out in front of me) in an afternoon at most.

But still they wouldn't take the advice. So I have to say you can give a developer the answer, but you can't make him use it. I know that in the end, they are going to have to use it to get any kind of scale for long-term stability and growth. Right now, they are, as they have been for the years I've been dealing with them - completely inflexible. Great attitudes when it comes to asking for help, eh?

This morning I have finally finished the coding of the SOD Position editor applet and web page. Today it was the applet activation in IE. Amazingly painful. But in the end, it's working and that's all the really matters.

Also, AdiumX 1.1 was released over the weekend and I updated it. The bug fixes look nice, and I have to say, this is an amazing step up from Fire, and I thought Fire was it. Use it every day, all day long.

On Saturday, I received the package from Apple for iWork '08 and put that license code in to stop the 30-day trial and get things set up right. I am still amazed at the level of thought put into Numbers. The increment GUI tools are clever, and completely obvious to use. The graphs are nice - much nicer than the default graphs in Excel. I'm still very glad I got this.

Today I'm going to try and get caught up on all the things I've let slide a few days as I've been working on the SOD Position Editor. Shouldn't take too long - the first thing (catching up on my journal) is now done.