What's it all about, Alfie?

I've spent most of today debugging a problem I saw in a vendor's API to a messaging system. It's not the best vendor I've ever worked with, in fact, I don't think they are even in the top 80%, but they are the vendor I have to work with at the time, and I've had to try and make the best of it. So here's the problem we ran into today.

We have a price injecting system. It gets price data over a custom socket interface (I wrote) from a price feed server and the reformats it into the format needed by the vendor's product and sends it on it's way. It's a simple transformation system. Nothing fancy. But the vendor's API is socket-based as well, and as we were to learn, not done nearly as well as mine.

When the transformer/feeder app was running in Chicago, and the price source was in Chicago and the database for the vendor's product was in New York, everything worked fine. When the feeder was in London and everything else was the same, my feeder missed the messages coming from the vendor's messaging system when I was injecting prices.

Inject prices - miss messages. Stop injecting prices - get messages. Make another test app subscribed for the same messages and it always gets them - because it injects no prices. This was getting crazy. So on a whim I decided to make use of a price source in London for the London feeder - Bingo! Now it worked. It appears that the vendor's API can't handle delays in the socket delivery from another completely different source. Yup. It's got nothing to do with the use of the vendor's API - it's the activity on a completely different socket that's effecting the vendor's API.

Note that in all this, my code is working fine. It's the vendor's that's stopped working properly. Nowhere in the documentation do they say that excessive waiting on socket communications will invalidate the delivery of messages - why should they? They probably never tested it, as they probably never had reason to. But when you charge $20 mil for something, you really ought to take a more pro-active view on things. For example, don't use a home-grown messaging buss when there are so many commercial ones that you can include in your $20 mil cost and not effect the bottom-line much.

In the end, I think I'm stable now, but there's really no way to know. They aren't going to fix this, I didn't expect them to. They took almost 2 months to fix the last bug we pointed out and that was a simple recompile with the right data type for a 64-bit version of the API. This would require real changes in how they do things, and change is not a word I'd use with this vendor.

I just have to say that I really hate the fact that it's expected that we figure this out. I'm not getting any part of that $20 mil, and yet I've saved their bacon by figuring out how to make it work in our environment. Crummy vendor.

Today I ordered a Kindle from Amazon. I tossed and turned over this for months. When I travel each day to work on the train, I read. A lot. So I have tons of books that are sitting on my shelves in my home office. I have been looking for an alternative, and the Sony feels nice, but it requires a PC - and I'm not about to go down that road. Also, I was very interested in the "shop and read" functionality in the Kindle. When I finish reading a book on the train, I have to sit for the rest of the trip because it's too bulky to carry two.

I'm not sure that the Kindle is going to be everything in eBooks, but I think it's got a decent chance. I was waiting for the second revision of the Kindle, but when Amazon came back and had them in stock, it was clear to me that the next version was not real soon now. So I decided to go ahead and get it.

It'll probably be like my iPod - got a 3G and used it for years before getting the Touch which is exactly what I want. I'm not convinced that Apple is coming out with a book reader - but it'd be nice if they did. I think that this is something that Amazon is in a unique opportunity to produce and provide. They have the bookseller reputation. So I'm going to give it a go.

I'll post updates as I get it and use it. Should arrive sometime this week.

At the Shop, I've been part of an integration team that's trying to get this large system into place and running. There's a ton of connections to our other systems, and it's a big job, but the part that I'm working with most is the price injection - because I'm the guy that wrote the price feed, after all. Makes sense.

Part of this product is a messaging system - which makes perfect sense to me - build a strong backbone and then hang things off it and let the middleware take care of the delivery and communication. Good plan. Horrible execution. I mean the worst I've seen in ages. And I've got data to back it up.

The entire messaging system is single-threaded. In this day and age of multi-core, multi-CPU machines, this alone makes this message system a problem. The entire thing is a bottle-neck, as opposed to enabling multiple messages to go between multiple senders and receivers, this guy acts like he's the 93 year old mail clerk in the post office. He's only going to do one thing at a time, and it's a government job, so he can't be replaced.

Case in point, I sent a message - a simple message to be sure, probably less than 1kB, and it took 23 minutes to arrive. Now this is going from one box in the server room to another box in the server room. There's no reason it shouldn't be measured in msec. - let alone 20+ minutes. While I understand that the speed of delivery is based, in part, on the messaging database and the other traffic it has to deliver, there's still no excuse to deliver to a customer a messaging solution that's 20 minutes to deliver a 1kB message.

The funny thing is, I could read all the relevant data from the database (about 3000 records) in about 15 seconds, so while it's clear that something is holding it up, it's not the underlying database that's the problem.

I've had problems with this product before, and as such, I'm not about to name names, but if you're doing anything with a custom-built vendor-suppled middleware, it pays to look under the hood and make sure you can plug in something that's got known good performance like Tibco or IBM MQ. Make sure you're not stuck with this company's horrible middleware that's causing you problems for years to come.

[5/1/08] UPDATE: after a restart of the subscription connections, things seems to be working better today. I'm not sure what the problem yesterday might have been, but it's working better today. I'm going to watch it for several days, but now it's delivering the messages in under 7 sec., and while that's not the fastest message system in the world, it's certainly acceptable for this project. I hope it keeps up.

I was chatting with a friend of mine on Twitter - through Twitterrific and he mentioned that he had gone into the NIB for Twitterrific and removed the tooltips for the mouse-overs of the name and the image. This makes the UI a lot cleaner as it already says who it is and it already says when it came in - so it's all redundant, and if you have the mouse over the window you're almost guaranteed to have a tooltip pop-up. So he cleared it out. I don't blame him, but I thought it'd be nicer to let the developers know what we wanted, and maybe they could fit in an option on the tooltips. Worth a try.

So I emailed them and got a reply back that was not surprising, but a touch disappointing. They are focused on the iPhone version, and when that's released in June they'll go back to the Mac version and they plan a significant upgrade to the UI.

So it's going to get addressed, but it's not going to be ready until the Fall. Which in itself is an interesting statement on the speed of their development - or at least the time they can devote to it. Then again, maybe they have significant changes in store and a little option would only take a few days (tops), but totally re-coding the app might take months. Hard to see, but it's possible.

In any case, if I want to strip out the tooltips, I'm going to have to fire up InterfaceBuilder and hack at it. Not sure that's what I want to do, but it's something to think about.

I realize it's hard to be a vendor to a lot of big companies. Some want stability and verification of easy little point release, and others want to see new features and fixes turned around as quickly as possible. Heck, I run into that every day here. Some groups want stability - which implies very few changes, and others want to see new things they've asked for as soon as possible - even if it means a little loss of stability. I get it... you can't please everyone.

But this can't be an excuse to do nothing. And by nothing I mean you can't have people on site helping to work on your multi-million-dollar product and have them (along with the users) identify bugs and needed features, and then just sit on them. That's not going to be 'OK' with anybody.

But it happens. Can't be helped. If a vendor is unresponsive, and you have to still make it work, you have to exhaust all possible options and avenues before throwing up your hands and saying "Well... we tried everything and still it's no-go. We have to wait for them to fix this."

I've had experience in both cases, and one is not more comforting than the other. You can't stop at the first sign of a vendor problem - you have to inform them, to be sure, but you can't stop there. You have to dig and dig, because you can't be sure that they are even going to look at the problem with the same intensity as you will. Plus, it's another pair of eyes - look and maybe you'll see something that they don't because they're too close to the problem.

At the same time, when you hit the end of the road, you have to accept that it is the end of the road and not try to change facts just because you want them to be different. You have to try and accept the facts. But you can't give up too quickly.

I've run into guys that have simply given up too quickly. Maybe that's all they had... maybe they couldn't see that there was another way to approach the problem. In any case, giving up is not going to make things work. You have to be more determined than that. If the answer can't be found, that's one thing, but you have to be sure that it's not that you're giving up too soon.

After all, you might be dealing with a vendor that's not going to be as responsive as you like, and you may only have a few weeks to get things working or you loose the management backing to continue with the project. You have to take things into your own hands a lot of the time. There's simply no other way to get things working.

I've been involved with computers since before the days of widespread Chat, ftp and telnet. It was initially modems with teletypes or (if you were lucky) a glass tty (terminal). There was nothing to download to - save the box you were logged into, and then it was pretty much your entire world. If you wanted to look at another system you hung up and dialed it. Period. It was all we had, and in relation to what came before (keycards and terminal rooms) it was heaven on earth. But things changed.

Ethernet changed a lot, we got networks of machines in college - not yet PCs, and the old hobby machines of the 70's didn't have ethernet cards in them. If you wanted to have them talk to one another you did a special serial cable with Tx/Rx crossed and then you were good to go. But by the time you had widespread adoption of ethernet, you had networks, and when you had networks it didn't take long to have telnet, ftp and chat.

I can remember first using ftp to get at things across the globe while in grad school. There were newsgroups that might publish the site and directory of something useful, and then with anonymous ftp, you could go get it. It wasn't fast, and there were no Google-like search engines in place... you had to log in, search the directory tree for the file you were looking for... download it and hope it made it down before you lost your connection. Still, this was big computer to big computer, it wasn't until the PCs came out that you really saw the growth of the Kermits, XModem, YModem, ZModem file transfer protocols. FTP was almost forgotten.

But IRC Chat staged a massive growth, and telnet was the way to get from one machine to another. So not everything was forgotten. But fast-forward to today. It's amazing to me to see that ftp, chat, and telnet (ssh) are as strong now as they were in the early days - not because something better hasn't come along, but because the work put into the documentation way back then.

Look at the RFC for FTP or Telnet someday... it's amazing the detail they went into. I've implemented both, and was immediately impressed that the RFC was right on point with the intended audience - people wanting to understand and implement the protocol. The docs are very well written, complete and detailed, but not overly verbose and wordy. These things were written by people wanting to pass on this knowledge to others in the industry and make sure that there were no lingering questions and problems.

FTP is, to this day, a great way to get files around, and is in every web browser around. Telnet (SSH) is virtually unchanged - save for the additional security, but is just as useful. It's Chat that seems to have taken on almost mythical proportions.

Look at all the Twittes, IMs, IRC Chat clients. It's all the same, basic premise - I type and you see it, you type and I see it. They first version I used was even called type on the Unix BSD4.x systems at Purdue. But look how many ways you can now communicate through this little concept. You've got the store-n-forward of SMS or Twitter... you've got the IRC Chat and a ton of different IMs. It's as if this - communication between people - was the real killer app of the network. Sharing a computer meant files and resources of one were available to another. Putting the machines on different desks took that away - only to be brought back by the network.

I use Chat (Colloquy), IM (Adium), and Twitter (Twitterrific) all day every day. It's fun to think about where things started and how they have evolved over the years. I'm sure the video chat will be bigger when the bandwidth is there, but right now, it's just not. Give it time, though, and it'll be just like all the Sci-Fi movies you've seen. It's a great time to be alive.

OK, this is a really odd feeling... I got an email that someone is following me on Twitter, which is not a big deal, but the odd thing is that I don't know this guy, and I can't imagine why he things the tweets I've sent are of such quality that they all deserve to be read.

I mean, it's not that I didn't want people to follow me - but I expected friends, co-workers, ex-co-workers, that kind of stuff. I didn't expect total strangers to think what I had to say 140 bytes at a time was worth really reading. Kind of wild. No... really wild. I guess this might be what it's like to have people want to read a comic you wrote. You did it for a reason, but didn't really expect some stranger to like it. Wild.

Who knows... it could be someone following 32,768 people, so it's not like there's something valuable I have to say. Could be someone that follows everyone. Anyway, it was odd enough to warrant a post.

Yesterday Apple updated Safari to 3.1.1 fixing two bugs but not putting in the new WebKit that's passing the Acid3 test 100/100. Too bad. I was looking forward to that, but they may have a lot more work to do on that guy before it's ready for release. Still, it's nice that they keep on top of the problems folks have found and release updates as often as they do. Sometimes it's still amazing how big a memory footprint Safari can get to be on my laptop - easily passing the 200MB size with only two tabs open. The cache and 'history' of the working browser has got to be enormous. Anyway, it's great to have that updated.

The next update I got this morning was Transmit 3.6.5, and while I really like the work the Panic guys do, I'm still a little surprised that they don't have a more 'minimal' interface to Transmit. What they have is fine, if you're using an FTP client like a file browser, but most of the time for me it's more like a small extension to the system. Make it as seamless as possible - no need for big borders - more like Cyberduck. That's a minimalistic interface that I like for file transfer. Nothing that gets in the way or takes up pixels that it doesn't need. I like all the features, and I appreciate what they are doing, but I think it'd just be nice to be able to have an alternate GUI that was less into borders fluff, and more into "just the goods".

The last update this morning was Camino 1.6. While I'm not a big user of Camino, I can appreciate what they are doing, and at times it's nice to be able to fire up another browser and see if the web site I'm having problems with works any better with Camino than Safari. It's fast, clean, and a nice Mac app, so you gotta keep up to date and make sure they didn't throw in something really cool that could make it the best browser on the platform. This release is supposed to have a few nice GUI improvements and I'm sure they're nice. I just haven't had a lot of time to play with it. Maybe this weekend.

Like any self-respecting computer goober, I love getting the updates and seeing what others have done with their code. In each of these cases, it's more than just an app - these are things that the developers are seriously proud of, and they should be. This is some of the best software out there, written by passionate, dedicated folks. And it shows. Gotta love that.

I really love Sun hardware and software. My first exposure to Sun workstations was the 3/50 in grad school and it was the most impressive box I'd ever seen. Oh, I'd had an Amiga, and it was nice, but when I was at school and could work on a 3/50 - or later a 3/60, it was an entirely different experience. Night and day. Nothing beat Sun stuff. Nothing.

Then I went to Auburn, and again, Sun was king. Oh, we had PCs, and I had a nice Mac II, and later a IIci, but if you wanted to crunch numbers and display them nicely, Mac OS 6 or 7, or Windows 3.11 just didn't cut it. You had to get Sun.

Little did I know at the time that had I seen an SGI workstation, my opinion of Sun would have changed, and I would probably never have looked at Macs the same again. At heart, I'm a scientist and a numbers guy. Scientific apps and data visualization is what I love doing most.

Anyway... Sun ruled the roost. Then they stopped innovating. Solaris 2.5 was nice, 2.6 was good... heck, they all are nice, but they aren't anything in comparison to linux or Mac OS X these days. And here's the case that brought it to a head this week.

I've been working on writing a C++ wrapper for libcurl and I wrote it on the Mac where I knew I had a recent version of libcurl, with 32 and 64-bit support. Then I moved it to linux -in this case Fedora Core 7 on x86_64 and it again compiled without a hitch. Then I moved it to my Sun box to make sure it built there. I have to do this, and usually do it in this order as it's the fastest and easiest way I've found to get the cross-platform builds working.

When I went to Solaris I had to add an include for bzero() which on the Mac and Linux is in the more standard includes, but on Solaris 8 (9 and 10), it's in strings.h. No big deal, I added that. Then started the build again. It compiled the 32-bit version just fine and then hit the wall at the 64-bit version.

Oh, my code compiled just fine - it's generic C++ with a little STL, but it's the linking phase that messed up, and that's where Sun has been standing still.

Sun was one of the first to support 32-bit and 64-bit on the same deployment on the same hardware. It was an elegant solution - two sets of files. True, Apple had a trick up it's sleeve in it's historical use of the quad-FAT binary (HP-UX, Solaris, Intel, Motorola). This allowed them to jump ahead to a single file with multiple architectures and word-sizes. Linux copied Sun's ideas and delivered multiple libraries pretty early on, which brings everyone up to about the same level.

Almost.

The problem is that Sun has failed to recognize that libraries like libcurl, openssl, zlib, and the others that I've gotten from SunFreeware.com need to be available in the default installation of Solaris. I've got Solaris 8 and 10 and what's missing form them is significant. I know they have SunFreeware, but even there, they don't seem to have multiple architecture binaries of the packages. If they had, I'd be a happy camper. I have already gotten most of SunFreeware installed on my machines, but it's all the single 32-bit versions of the packages.

I can't be the only one that wants 64-bit versions of these libraries. Maybe Sun doesn't have to include them in the OS, but it'd be nice if I could get a CD of all the libraries on a linux box built into packages for installation. Disk space is cheap... install all of them both 32 and 64-bit. Then, should I want to make something in 64-bit mode, I can.

So I'm left looking at the Sun box and thinking how nice it could have been had Sun just kept moving while it was ahead. Keep moving on the UltraSPARC chips... innovate... be the best of the best. But in the end, clever people with commodity hardware have essentially put Sun in the position of an also-ran, and that's sad.

Like when I heard SGI was filing Chapter 11. Sad.

So I've emailed Steven C. at SunFreeware and asked him what it might take to build OpenSSL and zlib and curl for 64-bit on Solaris 8 and 10 SPARC. I expect that it'll be more than I want to do at this time. It's getting harder and harder to justify Sun hardware when I can get an 8-core x86_64 box with amazing speed for next to nothing. Cost will drive the market.

I'd love to keep working on Sun, and I'll never get rid of my SPARC 20, but I can see a time very soon that work will be driven off linux, and it's a solid platform that I've enjoyed using for years. It's just too bad that Sun was in the lead, and they stood still.

UPDATE: I've received email from Steve of SunFreeware that the most recent versions of Solaris do have things like OpenSSL, zlib, and curl in 64-bit as part of the install. If that's the case, then I'm happy for Sun, but a little disappointed that it wasn't made backward compatible in the sense of Solaris 8 and 10.

For the last few days I've been trying to get an application to work with Java WebStart included with the JDK 1.6.0. It's been challenging, and what I've learned is that there's a lot you can do, and a lot you can't, and the trick is trying to turn the things you can't into things you can.

Packaging Shared Libraries

One of the things that isn't too hard to figure out is how to get shared libraries (so or dll) down to the client. Basically, you put them in a jar file and then sign that bad boy. There is one thing I learned: Sign all jars with the same key. The JWS is picky that way - they have to be the same key.

Delivering XML Files

Say you have an application (like I do) that uses XML files to configure itself properly. I didn't really pick this behavior, but it's something that I can see people dealing with. The Java code needs to be given the path to these XML files as arguments to the program, and then within the program it reads these XML files and deals with their contents. The problem is two-fold: getting the XML files there, and referencing their location.

Seems silly, but it's something I just lucked into. I had the XML files in a directory called cfg, and so I created a jar file with the cfg directory in it - just a bunch of XML files, and then signed that jar and placed it in the jnlp file as a nativelib resource. Something like this:

    <resources>
        <nativelib href="lib/cfg.jar" />
    </resources>

where I have ended up placing all my jar files in the lib subdirectory of the root of the JWS location. Then, if I use the arguments to the app indicating the location of the XML files are simply cfg/startUp.xml then the app will find them. Amazing, and interesting, and I wish someone had said the XML files worked like nativelib packages. I'm guessing just about anything can go into a nativelib and the code will find it.

Java VM Arguments

Again, this should be pretty obvious, but it wasn't in the examples I saw on the web. If you have a java command line like this:

    java -Xmx512m -DuseMyLog=true -DlogPath=logs/server.log -name Archie -type server

then the right way to split those up in the jnlp file is to put the -D arguments into the resources section and the regular args into the application-desc section. Something like this:

    <resources>
        <nativelib href="lib/cfg.jar" />
        <j2se version="1.5.0+" href="http://java.sun.com/products/autodl/j2se"
            max-heap-size="512m" />
        <property name="useMyLog" value="true" />
        <property name="logPath" value="logs/server.log" />
    </resources>
    <application-desc main-class="com.fun.MyServer">
        <argument>-name</argument>
        <argument>Archie</argument>
        <argument>-type</argument>
        <argument>server</argument>
    </application-desc>

I spent a ton of time trying to figure out what went where and for what reason. They certainly could have done a lot more in terms of the examples and explanations for this stuff. But, that's what I'm doing now, isn't it?

Finishing Touches

Once I got the thing working, I needed to get it updating and such. An addition in the 1.6.0 JWS you can add the update tag and that will let JWS know how to update the app based on changes on the server-side. I really liked this as I needed to make sure that I had this capability, and sure enough, it was there, just like I needed.

    <update check="always" policy="always" />

I have to admit that this has come a long way, and having it installed as a part of the JDK is really an important step in the right direction. I should be able to replace the third-party publish/subscribe application deployment system that's going away later this year with Java WebStart without too much trouble.

There are still things that I wish were there - variables. Make it so that you can put Java System property variables in the tags and then on invocation, have the JWS expand these. For example, say you wanted to put in the user name as an argument to the app. Since this is a static jnlp page served off an Apache web server, it's going to be very hard to get the username without placing a Java applet on the page that gets this data and sends it back to you in some form. Much too messy. Do this:

    <application-desc main-class="com.fun.MyServer">
        <argument>-name</argument>
        <argument>Archie</argument>
        <argument>-user</argument>
        <argument>@user.name</argument>
    </application-desc>

where the variable 'tag' @ is used to indicate that a Java System property variable name is following: user.name. Then, the JWS would replace the user's name in for this tag and pass the result to the application. This would solve a ton of issues as there are variables for most things you'd need, and the JWS is already running on the client machine, so it knows these things. It'd be really nice to have.

Thankfully, you can work that into your own code and do the same thing, but it's code we all don't need to write.