What's it all about, Alfie?

Yesterday I was putting in some new hardware for an app that I have that's been running nicely for a while. Unfortunately, the machine is old, getting underpowered for the task at hand, and it was time to update the hardware. At the same time, I wanted to update the message queue system that it was using. About once a month (or so) the existing message broker (Sun ONE Message Queue 3.0.1) will get into a confused and I'll have to restart it. This is bad as it's in a place in the infrastructure where it means that I'll have to restart several things because of the loss of the connection to the message broker.

Now that Sun has open-sourced the message broker, calling it OpenMQ, I decided to update to it and see if it was any better. I got the latest OpenMQ binaries (ver. 4.1) and then unpacked them using jar. There were precious little in the way of instructions, and in the previous version I had used (3.0.1) no configuration was needed. But that would prove to be a mistake for me.

The proper was to configure OpenMQ 4.1 given that it's installed into a directory called $INST is to first update the $INST/mq/etc/imqenv.conf file to include the proper value for IMQ_DEFAULT_JAVAHOME to the location of JDK 1.5.0. For me this was simply:

    IMQ_DEFAULT_JAVAHOME=/usr/local/jdk1.5.0

then I wanted to have the default memory size set to at least 4GB, so in $INST/mq/bin/imqbroker look to the line around line 82 that looks like:

    _def_jvm_args="-Xms32m -Xmx128m -Xss128k"

and change it to what you want. In my case, I changed it to:

    _def_jvm_args="-Xms32m -Xmx4096m -Xss128k"

since I was running a 64-bit JDK 1.5.0 this was going to work out nicely.

The next thing I needed to do (but didn't do at the time) was to set the maximum individual message size to unlimited. The default is 70MB, but there are some messages in my system that are very large. The place to do this is in $INST/mq/lib/props/broker/default.properties on the line that starts with:

    imq.message.max_size=70m

and change it to what you want it to be. A value of -1 means unlimited, so I set:

    imq.message.max_size=-1

next, for those topics (queues) that are auto-created, there are a few parameters in the section on 'destination based topics' that you might want to use:

    imq.autocreate.destination.maxTotalMsgBytes=-1
    imq.autocreate.destination.maxNumProducers=500
    imq.autocreate.destination.maxBytesPerMsg=-1

the first and last are basically saying "make no limits on the size of an individual message, or the total size of all messages". The middle one is saying that you might want to have a queue that has a lot of producers (injectors) and one consumer (reader), and the default is 100 and it's OK for most folks, but I wanted to make sure that we didn't run into problems if we had clients directly hitting the broker. Now we should be all done with the configuration.

There is an /etc/init.d/imq script in the $INST/mq/etc/init.d directory, but I didn't use that one as I already had one that was working from the 3.0.1 version. All I needed to do was to change the installation location of OpenMQ (the $INST directory) and then it was ready to go. It was already using JDK 1.5.0 and on this machine that means 64-bit.

Add it to the chkconfig start-up on levels 3, 4, and 5 and then start it with /etc/init.d/imq start and all should be just fine.

But that's not how it really happened for me. Here's the problems I ran into to come up with what should have been done in the first place.

First, if you miss the definition of the IMQ_DEFAULT_JAVAHOME variable it means that you have to edit the imqbrokerd script. This was not unusual as I was already in there for the maximum size, so I added that in there and didn't think a thing of it.

Next, if you don't remember to set the maximum message size, then changing it in the default.properties file is not going to be good enough. You have to remove all the data for the instance as I'm convinced that the configuration of the queues is stored in that persistence and the default.properties file is only read when there's nothing in the instance to base the queues off of. So, if you start it without the maximum message length what you need it to be, then you're going to have to shut down IMQ, then remove the entire directory $INST/mq/var/instances/imqbroker. If you don't remove the imqbroker directory I don't think the changes are going to properly work. Thankfully, all my code makes the queues automatically so wiping out the existing configuration is no big deal.

Also for auto-creating queues, you have to remember the two critical size configuration parameters or else you're going to have those queues in trouble when they try to send through large messages. Sure, the 100 producer limit on auto-created topics is reasonable for most installations, but I had talked to another developer here using OpenMQ 4.0 and he had to set it to 500, so I figured that while I was in the config file, I'd up that limit too.

Also, it's important that your app uses the $INST/mq/lib/imq.jar and $INST/mq/lib/jms.jar or else you can have connection problems. Specifically, if going from a 3.x imq.jar to a 4.x IMQ, you're going to get connection errors if you're not using the 4.x imq.jar. So just be safe, get the one with the OpenMQ distribution you're using.

Today there have been a lot of problems with a system that we have in the shop that takes a price feed. It seems that this vendor's custom code to interface their system to the Bank's price feed was having stability problems. Specifically, it's a Java process using JNI that was blowing out of 1.5GB of RAM allocated to the 32-bit process. Having worked with the Bank's price feed for a few projects I know the symptoms of this kind of problem and how to fix it. I'm not going to say this is the only way to fix the problems, but I've tried a lot of things before finding this solution, so I know what's not going to work to a large extent.

So... the vendor throws up it's hands and asks us for help. Earlier, I had sent one email message as I saw so many flying around about these stability issues. I said "Hey, I've got it working, I know it's hard, but there is a way to make this work." The response I received was "Thanks, but we're going to try to save this design and impact the code as little as possible." Normally, I'd agree with them on the minimal change issue, but this time I knew that a minimal change was not going to work. It wasn't a hard change to make - less than a few hours, but it was a fundamental change in the way they were processing the data.

You see, the data is coming on on a (virtually) single-thread calling an onMessage() method to pass in the message containing the data. Because of the way the Bank's price feed is written, you have to make sure that you take as little time as possible in dealing with this message and return control to the calling thread as soon as possible. This means you can't do anything other than throw it on a queue and then have some other thread(s) taking it off the queue and doing the real processing.

So we get into this phone meeting and they start to say what they've done and tried. They quote some timing figures for how long they take on the processing of an event. This doesn't matter a bit. It's how fast you return control to the onMessage() caller that's going to make or break this system. So on and on they go... I finally say "Here's what you need to do..." and outline what they need to do to make it work.

They say "That may work for you, but it can't work for us."

Remember now, they emailed us throwing up their hands for help on the solution to the problem. So this attitude was more than a little shocking coming from the people asking for help. I was only suggesting a way to queue/dequeue the messages - nothing that couldn't be retro-fitted into their code (I had it on a print-out in front of me) in an afternoon at most.

But still they wouldn't take the advice. So I have to say you can give a developer the answer, but you can't make him use it. I know that in the end, they are going to have to use it to get any kind of scale for long-term stability and growth. Right now, they are, as they have been for the years I've been dealing with them - completely inflexible. Great attitudes when it comes to asking for help, eh?

Why is it so hard to find a person in Facebook? I got an invite chat from an old co-worker this morning and I've been trying to 'find' him on Facebook quite a few times today. Why don't they have a "Last, First" name locator? I can tell if it's Dan, so what's the big deal? It's like you have to look through one of the email/chat networks or one of the other social links. You can't just try to find him by name. Seems like a very bad idea to me.

I'll keep trying, but this is insanity not to have a better way to look people up.

UPDATE: Seems there's a simple name search, but you have to get past the recommended account initialization steps which seems to be looking in your existing networks for friends. Seems that would be one of the options in that initial set-up. Oh well... it's done now.

This morning I have finally finished the coding of the SOD Position editor applet and web page. Today it was the applet activation in IE. Amazingly painful. But in the end, it's working and that's all the really matters.

Also, AdiumX 1.1 was released over the weekend and I updated it. The bug fixes look nice, and I have to say, this is an amazing step up from Fire, and I thought Fire was it. Use it every day, all day long.

On Saturday, I received the package from Apple for iWork '08 and put that license code in to stop the 30-day trial and get things set up right. I am still amazed at the level of thought put into Numbers. The increment GUI tools are clever, and completely obvious to use. The graphs are nice - much nicer than the default graphs in Excel. I'm still very glad I got this.

Today I'm going to try and get caught up on all the things I've let slide a few days as I've been working on the SOD Position Editor. Shouldn't take too long - the first thing (catching up on my journal) is now done.

Today I've been doing more coding on the SOD position file editor. I talked about it a lot in the post yesterday, but it's nice to be moving forward - even if it's slowly. The Java GUI tools are powerful, I'll grant you that, but they aren't something that you can use once every year and expect to be really proficient at much of anything. You have to dedicate a lot of time to getting into each class and how it fits into the collective whole to be able to code a good looking GUI without having to hit Google for every other feature.

Also, today is my anniversary - 22 years. Amazing. Going to try and have a little fun this weekend.

I've been in the Groove on adding the start-of-day (SOD) position file editing to the editor web for my server. Java GUIs aren't my favorite thing to code up because of the mastery of the object library to really get it done right. Thankfully, Google is there to help.

UPDATE: One of the problems was getting the columns sized correctly. And when I say 'correctly' I mean sized to fit the data in the column headers and column data. So that if you give it enough horizontal space, you'll be able to see everything without having to resize the columns each time you do something like sort the table by a column. The code turned out to turn off the auto-resizing and then go through each column, getting the renderer and asking it what it's preferred size was. Add those up, tack on a bit of margin and you have it. The code is in BKSimpleTable, if you want to have a look.

The next thing was getting the changes back to the server from the applet. I decided to make a cgi script that simply took the old and the new, and replaced one with the other in the SOD position file. It turns out that it's not that bad - but not really efficient either, as you have to scan the file for the matching line. But that's not horrible as we're not doing hundreds of these a day.

The next hurdle was getting the applet to auto-activate in IE. Turns out, IE requires that you call Javascript in another file in order to get it to work. Basically, make a Javascript function myPrintln(s) and in that simple call document.writeln(s); and as long as this is in another file, it'll work. Then, write out each line with this method. Seems like a waste, and it is, but that's IE compatibility. I do with they'd get on the ball.

In the end, things are looking good for a 1.0 release of the page. I've shown it to the Data Team and they will play with it and see if they can find any problems or additions that they need. I'm hoping that this makes the corporate action processing so easy that it's not a major issue any more.

Yesterday Apple updated iWork '08 and with the addition of Numbers to iWork, it was something I really had to try. I've been putting off getting MS Office for a long time. I just didn't like the idea of spending money on another Microsoft product, and TextEdit and Mesa did almost all the tasks I needed. But they weren't 100%, and in the end, I still had to do a lot of fiddling with Word, Excel, etc. So I was happy to see that iWork added a spreadsheet.

I got the trial and tried to open up some of the critical Excel and Word docs that I needed to have. For the most part, I need to be able to read and write these formats, so it was very nice that right there on the web sites, it says Pages and Numbers can read and write Office formatted files. Combine that with the fact that the style of these apps is incredible, and you have a really compelling argument.

Then again, it's $79.

Sold.

I will use the trial until I get the serial number in the email, or get the package from Apple in a few days. It's nice to be able to support the Good Guys and at the same time fit in with the rest of the business world on the file formats. Yup... I'm a big Apple fan.

We have been conducting phone screenings for developer positions at work and I have to say, it seems that the really good developers have gone into hiding... maybe the witness protection program, or something. It's amazing. I've tried to think of my old co-workers and realized that most of them are happy (enough) in their jobs - that or they have moved out of the city/state and don't want to move back. It's getting surprisingly hard to find really top-notch candidates to hire.

I'm sure the I'm happy with my current position plays into it a lot, as does the desire not to get management upset after learning that you're in the market for a new position. But really... I'd have expected more than we're seeing.

Well... the search goes on.

Well... I spent a lot of time Wednesday and Thursday working on these improvements to the Server, but the more I got into it, the more I was convinced that it was a horrible idea - and it was my idea. The basic idea of the enhancement was to have the instruments in the server contain a business date, and then according to a configurable schedule, we'd roll the business date by region (Asia, etc.) and that way the instruments would pick up new SOD positions and 'look' like they were ready for the next business day.

But the problem was really that all the changes required for this to work would have made the server far less stable in the short-term, and that might be a public relations problem that would take months to recover from. Additionally, I needed to know the exact requirements from the traders in Hong Kong, as they are the ones pushing this. Do they need to see the positions for Europe and the Americas when they get in? Do they need to see the marks? These are things that are going to make a huge difference to the code.

In the end, I decided that it would be far, far better to make the server capable of reloading it's complete position set - as well as marks (which it already does), so that we can make a trial SOD position file and start a new server off that - ideally, London. Then, when the final SOD position file is ready, we copy that over to London, and reload all the positions into the London server. This means that the Asia positions and marks will be rock-solid, and the Europe/Americas positions will be close and the refinement will make them just perfect.

This solution has many upsides, but the one downside is the human cost of data maintenance. If we have two servers, then any editing done to one has to be done to the other as well. This just means that the effort I would have placed in the server now needs to go into the server and editor. Basically, all the editing operations that are now being done need to be made as simple and easy as possible. That way, the load for the data maintenance of one server goes down, and hopefully, the addition of the other is not a horrible problem.

There are several things to work out, but the number of systems that have to change in this plan is much fewer - only the Server/Editor. This makes it much more likely that the stability will remain high during the changes, which is a major benefit. We have to deal with the trade processing, but that shouldn't be hard - again, it's by region. Also, we need to have a trial SOD position file, and that should not be hard either as it's already capable of being generated by several systems at this time.

Not all my plans are jewels... this one is far better than my original. Far, far, better.

Today was spent working through a few issues on the conversion of the server from a nightly restart to continuous operation with multiple regions rolling over as the day progresses. It's really that latter part that's the pain in the neck as a single timezone implementation wouldn't be nearly as hard, but the problem is that we can't get people here to stop work early enough to allow the system to be ready for the start of the far east day.

Today was really just trying to get all the references to the global constants for time and date to be region-specific so that we can have n regions and not have to worry that there are just two - like there are at the present time. While I don't think there will ever be more than two regions, it's not a good idea to build in such a concept as it represents a serious limitation going forward.

Tomorrow will be more heads-down coding. Not a lot of time to sit up and look around, I need to get this code in place and working on a single timezone as soon as possible to check and see what the stability impacts are to the system.