Archive for August, 2007

Java’s Threading Tools

Tuesday, August 21st, 2007

There is a lot to like about the threading model in Java. It's easy to make threads, it's decently fast, and it's stable. One thing that I wished they'd put into the JVM - or at least into javac like all the other 1.5/1.6 compiler 'hacks', is the idea that synchronization on a setter can be controlled by the instance variable itself. What I'm asking for is an automatic way to have javac create thread-safe setters and getters by simply naming the ivar.

The most obvious way of making a thread-safe setter is this:


    /**
     * This is the standard setter for the temperature
     * of this instance.
     */
    public synchronized void setTemp(double aTemp)
    {
        _temp = aTemp;
    }

but the downside is that this means that if this instance is in a highly threaded environment, or there are other methods that take a significant amount of time to process, the ability to 'set' the temperature is dependent on the other synchronized methods on this class.

Sure, you can make an object to lock on and it might look like this:


    private     double      _temp = 0;
    private     String      _tempLock = "TempLock";
    
    /**
     * This is the setter for the temperature uses an
     * additional ivar - a String, to protect multiple
     * threads changing the value at once.
     */
    public void setTemp(double aTemp)
    {
        synchronized (_tempLock) {
            _temp = aTemp;
        }
    }

but then you have all these extra ivars that you have no real need for and no need to really put into the setter other than as a thread-safety mechanism.

Some will argue that if I had used a Double and not a double I could synchronize on the ivar itself. But it's not really safe doing that. You'd have to be careful of the null value and then what if you changed the reference within the synchronized block - you'd no longer be protected. No, there needs to be the ivar itself - allowing null, and then some mechanism to ensure thread safety on a more finer-grained level than locking the entire instance.

They need to have something that does the obvious things for an ivar of a given type. You want to save typing? Make something like this:


    /**
     * These create the setter and getter for the ivar
     * '_temp' because the need to lock is implied
     * on the setter and typically unnecessary on the
     * getter (unless it's a long).
     */
    public threadsafe setTemp(double aTemp) : private double _temp;
    public threadsafe getTemp() : private double _temp;
    
    /**
     * These versions would create 'protected' versions
     * of the getter and setter methods.
     */
    protected threadsafe setTemp(double aTemp) : private _temp;
    protected threadsafe getTemp() : private _temp;

    /**
     * This creates both setter and getter for the ivar
     * '_temp' because the structure to create the two
     * is clear from this information alone.
     */
    public threadsafe accessors Temp : private double _temp;

The information to build the code for the setter and getter is clearly specified in the above examples. In the first set, the ivar _temp is clearly private, a double and the names of the methods to build are clear. There's even the argument type - which is really redundant, if you think about it.

A possibly even simpler solution would be to combine the definition of the ivar and the scope of the setter and getter in one statement - like the last example. That way, you would know the ivar's name, type, scope, and then the scope of the setter and getter. You could even allow one to use a PropertyChange system with listener registration, etc. This is not that hard, and I've used it in tons of classes. The clutter it'd save would be far far more than the collapsed for-loop.

Anyway, today I did a lot of the 'mutex' variables because I needed to remove as many synchronized methods as possible on a class I was working with. Virtually every one of them was a setter, and it was annoying to think that they made a goofy collapsed for-loop in 1.5, but they left this little nugget out.

Amazing. I think it shows to me that defensive programming is not heavily exercised by the Java Developers. Otherwise, they too, would be sick of making thread-safe but not lock greedy accessor methods.

Finally a Web Page for frosty

Tuesday, August 21st, 2007

I finally got around to putting up a very simple web face to frosty, my iMac G4 that used to be my Mom's until the hard drive died and I got her a new 15" MacBook Pro as a replacement. It's nothing fancy, in fact, it's pretty much the same web site I put up for all my machines. It's clean, simple, and just enough to look decent.

Ah... that feels better.

Working at the Speed of Mistakes

Monday, August 20th, 2007

Normally, I like the fast-paced environment that I work in, but every now and then it becomes clear to me that there's someone asleep at the wheel. Meaning, they are so focused on the next six weeks (for instance) that they forget to look past that and realize that as soon as this short-term thing is done, it's not going to stop the problem, it'll only make it worse. Take 24hr processing.

If we have one system that's 24hr, then they're going to say "Hey, nice, but we can't blank, and we have to be able to do that!" So solving one problem by making it available 24hrs is not really solving anything - it's causing the next thing to become the 'problem'. Then, when they have that second thing, the users will say "Yeah, that's better, but what we want is blank." and it'll continue until everything runs 24hrs a day and had they just stopped and realized that in the beginning, then we could have saved a lot of time and trouble by designing the solution that will make everything work that way.

Trying to do this piecemeal is not the answer. Everyone involved would agree if they stepped back enough to look at the issues really driving these decisions. But they won't. It's enough to say "They asked for x, we'll give them x." and then move on.

So it's going to be a messy year as more band-aids are applied to systems that don't really need them if the systems driving them were working 24hrs, and they'll eventually have to because the users will pick them off one by one until they have them all running that way. But they won't fess up that's what they want, so it's band-aids.

Yeah... it's going to be ugly for a while.

SSHKeychain Goes to 0.8.1

Monday, August 20th, 2007

One of the things that I've wished I had was a good version of SSHKeychain. The problem with the 0.7.1 version (aside from being PPC-only) was that it didn't allow for X11Forwarding and I use that a lot on the connections to my unix boxes. Interestingly enough, it's not just the graphical apps - Vim checks to see if you have X11Forwarding on and if not, then it complains a bit. A hassle, yes, but so easy to include when you can configure your ~/.ssh/config file to suite your needs and hosts.

So I got the 0.8.1 version and I sent off the question about the X11Forwarding to the author to see if it's there now, or if not, can it be? I'm guessing that it's likely that since he doesn't use it, it's not in there. But maybe he'll take my suggestion and put it in there. If that were the case, then I'd gladly drop my scripts to start ssh-agent and stop messing with that. It's a hassle and I'm worried that Leopard will break it again. Hard to tell. But it's great that he's back at the coding on SSHKeychain... maybe I'll get what I need yet.

UPDATE: I read this off Daring Fireball this morning about SSHKeychain. Seems there's a new developer on the project and he's a little more security conscious and the setuid on the tunnel app and the way the SSH pass phrase is stored in the KeyChain is not very secure at this time. He's hoping to change that, and the work-arounds are reasonable for the tunneling - which I don't do, but the suggestion for the pass phrase is to not put it in the KeyChain - which defeats the purpose of the app, in my opinion. There's still the outstanding issue of the missing X11Forwarding which I need to have. He sounds like he's going to get to these things, but it'll be a few releases before I can stop using ssh-agent.

FreeTDS and jTDS

Friday, August 17th, 2007

Back in late 1999 I was doing a little work at First Chicago NBD (BankOne) on linux as the development platform of the future. I had made a position as the Head of the Technical Architecture Group in Capital Markets that Java/CORBA/Sybase with a bean server like Jaguar CTS would be the way to create apps in the future. In order to prove my point, I got a nicely powerful box (at the time) and started doing some work for a project in the commercial card services division. What I found was that there were a few things that needed to be done in order for the linux desktop to fit into the Bank's infrastructure easily.

Most things worked pretty well right away. Email, at the time, was SMTP/POP3 so that wasn't a problem... the only biggies were source code repository and database access. We were using Microsoft Visual SourceSafe, and there turned out to be SourceOffSite that runs on linux and accesses the Visual SourceSafe repository just fine. The final step was access to Sybase at the C library level.

FreeTDS was a great open-source project that did most of the TDS spec for Sybase and SQL Server. It wasn't 100%, but it was all that was necessary to get the job done, and that meant that I could get access to the Sybase databases from my linux box. I used it and didn't have to look back. With it, I was able to build SQSH, and the access from apps was pretty simple.

Fast-forward to this morning and I was talking to a friend about something completly unrelated and another guy stopped by asking about the JDBC URL for jTDS.

"jTDS?" I asked.

"Yeah, it's a JDBC driver based on the TDS protocol for Sybase - by the guys that wrote FreeTDS"

Wow! I talked to him about my previous experience and we laughed about how things come around again (and again). Turns out, in his tests, and he's not alone, jTDS outperformed Sybase jConnect by a factor of better than 2! That's impressive. So, after I was done with the updates we were talking about, I went back to my cube and got jTDS and tried it. There were a few little changes to my code - mostly because the URL is different and I wanted to make it optional to use jConnect or jTDS. But in about 15 mins. I had something working and the tests looked great. Amazing.

Highly recommended.

DataGraph Update, Registration, and Request

Thursday, August 16th, 2007

Today I noticed that DataGraph had gone to version 1.4 and there was a new forums site. So I decided to register a version, get the Framework, and request the features that I've wanted to have for a long time: contour and heat graphs. 2D graphs, but based on 3D data. These would be the types of graphs that I really want to put into my Potentials code that I started building a long time ago. The simplicity and design of DataGraph is really quite amazing. While it's not exactly what I would have done, it's certainly a new and interesting take on the data plotting application and it's got everything that you could ask for in 2D plotting - save these two graph types.

Certainly, something like VantagePoint would be great, but that's Java, and it's $25,000 for a license. Additionally, there's no application with VantagePoint - you have to code up everything yourself. Now, I've gotten pretty good at using VantagePoint, but still... it's not the easiest thing for creating new plot types as you have to build an entire testing framework to see if you have the parameters set right for the graph you want. It's got great documentation, but still... to have an application would be a plus.

That's what I was hoping for with DataGraph. We'll see if he's interested in making DataGraph Plus and charging $200 or so - making it fall between the $30 DataGraph and the $1200 DataTank. While DataTank would be nice, I'm just not doing that much visualization to justify the cost. It'd be nice to be able to make a few big improvements in the output visualization of the fields. That would be a lot of fun.

What Kind of Weird Universe is This?

Thursday, August 16th, 2007

I had an almost surreal experience this morning. To set the stage a bit, I know that good developers are seemingly harder and harder to find. It's been tough given the list of folks we've been interviewing. Now layer on the Rock Star I've talked about in the past and you have to wonder what can really be done about this?

So here's what happened...

I got a chat from Rock Star about the possible values for a field from a data provider we use for Market Data that is pretty much known for it's lack of documentation in the industry. I've worked with this provider for several years, and the people you deal with are nice, and want to help, but the documentation you get is just very bad. No two ways about it. But given that, you learn that the best way to find out what a field is, is to call it with known (expected) output values and see over the course of a few dozen tests if it's what you think it is. If it looks like you're on the money, then you set up bigger tests, and if it holds, you can assume that it's what you thought it was. It's not efficient, it's not even fun, but when you have little documentation and a phone call to the provider yields "I dunno... have you tried it?", you learn that it's better to figure some things out on your own.

But not for the Rock Star, oh no...

"Where's the documentation for these fields? What are the valid return values?" he asks.

"Here's what we have, and that's all there is." I reply.

"That's unacceptable."

OK, that may not be what you want, but you'll accept it because that's all there is. It is, by definition, acceptable, because that's all they are going to give up. I had to actually laugh at Rock Star and ask him Who do you think you are?

I ask him if he's tried the field on a few instruments to see what he gets back - explaining that this is exactly what I've had to do in the past for anyone that needed to know what was what through this API.

"That's unacceptable."

I can't believe this guy. I'm giving him the way to answer his own question and he's refusing to do it because he thinks there has to be a better solution. I get a little upset and simply tell him I'll talk to the provider and see what they say, but that's it. He starts to say more and I simply answer with "Stop talking and walk away." I was getting upset, and didn't want to get to the point that I'd be unprofessional with this guy even though he was clearly being unprofessional in his refusal to even try the approach I suggested.

So I have to wonder - Is this what my beloved industry has come to? Are we to the point that prima donnas, Rock Stars, are the best we can get if you want someone with any real experience? What's happened to all the people that are good at their job, not interested in using new technologies simply for the sake of padding their resume, and are willing to hunker down and solve the problems as they present themselves - not giving in and working with what's available? Are they all gone? Or are they just so well established in their current positions that the only real turn-over is the remainder?

I don't know, but I certainly need to be smarter about cutting off Rock Star when the conversation degenerates to what it was this morning. It's not helpful for anyone. If he's upset with the answers I give him, I'm going to have to tell him to take it to Management. If they back Rock Star, then so be it. But I think I can make a convincing case that me experience in these providers trumps his silly little assertions of unacceptability.

OpenMQ Message Size

Wednesday, August 15th, 2007

dukeplug.gif

Yesterday I was putting in some new hardware for an app that I have that's been running nicely for a while. Unfortunately, the machine is old, getting underpowered for the task at hand, and it was time to update the hardware. At the same time, I wanted to update the message queue system that it was using. About once a month (or so) the existing message broker (Sun ONE Message Queue 3.0.1) will get into a confused and I'll have to restart it. This is bad as it's in a place in the infrastructure where it means that I'll have to restart several things because of the loss of the connection to the message broker.

Now that Sun has open-sourced the message broker, calling it OpenMQ, I decided to update to it and see if it was any better. I got the latest OpenMQ binaries (ver. 4.1) and then unpacked them using jar. There were precious little in the way of instructions, and in the previous version I had used (3.0.1) no configuration was needed. But that would prove to be a mistake for me.

The proper was to configure OpenMQ 4.1 given that it's installed into a directory called $INST is to first update the $INST/mq/etc/imqenv.conf file to include the proper value for IMQ_DEFAULT_JAVAHOME to the location of JDK 1.5.0. For me this was simply:

    IMQ_DEFAULT_JAVAHOME=/usr/local/jdk1.5.0

then I wanted to have the default memory size set to at least 4GB, so in $INST/mq/bin/imqbroker look to the line around line 82 that looks like:

    _def_jvm_args="-Xms32m -Xmx128m -Xss128k"

and change it to what you want. In my case, I changed it to:

    _def_jvm_args="-Xms32m -Xmx4096m -Xss128k"

since I was running a 64-bit JDK 1.5.0 this was going to work out nicely.

The next thing I needed to do (but didn't do at the time) was to set the maximum individual message size to unlimited. The default is 70MB, but there are some messages in my system that are very large. The place to do this is in $INST/mq/lib/props/broker/default.properties on the line that starts with:

    imq.message.max_size=70m

and change it to what you want it to be. A value of -1 means unlimited, so I set:

    imq.message.max_size=-1

next, for those topics (queues) that are auto-created, there are a few parameters in the section on 'destination based topics' that you might want to use:

    imq.autocreate.destination.maxTotalMsgBytes=-1
    imq.autocreate.destination.maxNumProducers=500
    imq.autocreate.destination.maxBytesPerMsg=-1

the first and last are basically saying "make no limits on the size of an individual message, or the total size of all messages". The middle one is saying that you might want to have a queue that has a lot of producers (injectors) and one consumer (reader), and the default is 100 and it's OK for most folks, but I wanted to make sure that we didn't run into problems if we had clients directly hitting the broker. Now we should be all done with the configuration.

There is an /etc/init.d/imq script in the $INST/mq/etc/init.d directory, but I didn't use that one as I already had one that was working from the 3.0.1 version. All I needed to do was to change the installation location of OpenMQ (the $INST directory) and then it was ready to go. It was already using JDK 1.5.0 and on this machine that means 64-bit.

Add it to the chkconfig start-up on levels 3, 4, and 5 and then start it with /etc/init.d/imq start and all should be just fine.

But that's not how it really happened for me. Here's the problems I ran into to come up with what should have been done in the first place.

First, if you miss the definition of the IMQ_DEFAULT_JAVAHOME variable it means that you have to edit the imqbrokerd script. This was not unusual as I was already in there for the maximum size, so I added that in there and didn't think a thing of it.

Next, if you don't remember to set the maximum message size, then changing it in the default.properties file is not going to be good enough. You have to remove all the data for the instance as I'm convinced that the configuration of the queues is stored in that persistence and the default.properties file is only read when there's nothing in the instance to base the queues off of. So, if you start it without the maximum message length what you need it to be, then you're going to have to shut down IMQ, then remove the entire directory $INST/mq/var/instances/imqbroker. If you don't remove the imqbroker directory I don't think the changes are going to properly work. Thankfully, all my code makes the queues automatically so wiping out the existing configuration is no big deal.

Also for auto-creating queues, you have to remember the two critical size configuration parameters or else you're going to have those queues in trouble when they try to send through large messages. Sure, the 100 producer limit on auto-created topics is reasonable for most installations, but I had talked to another developer here using OpenMQ 4.0 and he had to set it to 500, so I figured that while I was in the config file, I'd up that limit too.

Also, it's important that your app uses the $INST/mq/lib/imq.jar and $INST/mq/lib/jms.jar or else you can have connection problems. Specifically, if going from a 3.x imq.jar to a 4.x IMQ, you're going to get connection errors if you're not using the 4.x imq.jar. So just be safe, get the one with the OpenMQ distribution you're using.

Asking for Help the Wrong Way

Tuesday, August 14th, 2007

Today there have been a lot of problems with a system that we have in the shop that takes a price feed. It seems that this vendor's custom code to interface their system to the Bank's price feed was having stability problems. Specifically, it's a Java process using JNI that was blowing out of 1.5GB of RAM allocated to the 32-bit process. Having worked with the Bank's price feed for a few projects I know the symptoms of this kind of problem and how to fix it. I'm not going to say this is the only way to fix the problems, but I've tried a lot of things before finding this solution, so I know what's not going to work to a large extent.

So... the vendor throws up it's hands and asks us for help. Earlier, I had sent one email message as I saw so many flying around about these stability issues. I said "Hey, I've got it working, I know it's hard, but there is a way to make this work." The response I received was "Thanks, but we're going to try to save this design and impact the code as little as possible." Normally, I'd agree with them on the minimal change issue, but this time I knew that a minimal change was not going to work. It wasn't a hard change to make - less than a few hours, but it was a fundamental change in the way they were processing the data.

You see, the data is coming on on a (virtually) single-thread calling an onMessage() method to pass in the message containing the data. Because of the way the Bank's price feed is written, you have to make sure that you take as little time as possible in dealing with this message and return control to the calling thread as soon as possible. This means you can't do anything other than throw it on a queue and then have some other thread(s) taking it off the queue and doing the real processing.

So we get into this phone meeting and they start to say what they've done and tried. They quote some timing figures for how long they take on the processing of an event. This doesn't matter a bit. It's how fast you return control to the onMessage() caller that's going to make or break this system. So on and on they go... I finally say "Here's what you need to do..." and outline what they need to do to make it work.

They say "That may work for you, but it can't work for us."

Remember now, they emailed us throwing up their hands for help on the solution to the problem. So this attitude was more than a little shocking coming from the people asking for help. I was only suggesting a way to queue/dequeue the messages - nothing that couldn't be retro-fitted into their code (I had it on a print-out in front of me) in an afternoon at most.

But still they wouldn't take the advice. So I have to say you can give a developer the answer, but you can't make him use it. I know that in the end, they are going to have to use it to get any kind of scale for long-term stability and growth. Right now, they are, as they have been for the years I've been dealing with them - completely inflexible. Great attitudes when it comes to asking for help, eh?

Facebook Insanity

Monday, August 13th, 2007

Why is it so hard to find a person in Facebook? I got an invite chat from an old co-worker this morning and I've been trying to 'find' him on Facebook quite a few times today. Why don't they have a "Last, First" name locator? I can tell if it's Dan, so what's the big deal? It's like you have to look through one of the email/chat networks or one of the other social links. You can't just try to find him by name. Seems like a very bad idea to me.

I'll keep trying, but this is insanity not to have a better way to look people up.

UPDATE: Seems there's a simple name search, but you have to get past the recommended account initialization steps which seems to be looking in your existing networks for friends. Seems that would be one of the options in that initial set-up. Oh well... it's done now.