What's it all about, Alfie?

Today I spent most of the day trying to check the fix I made yesterday afternoon. I believe it's right, as I've stepped through the code and I can't see why it might be wrong, but I wanted the users to check the data. I was very hard to get them to verify the data.

The primary problem was that the data is sampled from one system, pushed into another system and then shown in a client. It's a bit of a travel, and the question is: How old is this data?

The problem I kept running into was that I could not prove the data right because there was much I could not tell about the processing. The original developer used Aspect logging, and while that did what he wanted (kept the logging out of the processing code), it meant that there were limitations on the logging because of that same fact.

I decided that it had to change, and started putting in the logging at all levels of the application so that when I was done, there would be one log file that would tell me:

• what machine gathered the data for the server (by hostname or IP address)
• what type of request was done
• when it was sent to the extraction process and how long he took
• time required to parse the packet received at the server
• time required to incorporate the data in the packet
• total time from start of grab to the successful incorporation of the data

This is the kind of logging this application needs to have. It's got to mean something and be useful.

In a way, I'm a little glad. After thinning out the old Aspect logging, it's starting to look like something I might have done from the start. I can quantify the delays so that I can tell the users exactly what they are seeing. It's a much better place to be.

They may not like it, but that's a different issue.

Today Apple released a lot of new hardware - new iMacs, new Mac Minis (including a server version with no DVD but two hard drives), new mouse, new remote, and updates to the wireless products. It was a pretty amazing update to the iMac: 27" display with quad-core, up to 16GB RAM, and 2TB disk. All for less than $4k. Amazing.

The mouse looks great because it's got the multi-touch interface on the top - no buttons at all. Very sweet. Definitely cool hardware. Can't wait to see it in the Apple Store and check it out.

I want a new desktop machine 🙂

Wow! I missed the final release of LaunchBar 5.0! This morning I noticed that LaunchBar 5.0.1 was released and I got it right away. There's so much in this app that I don't use, and should... the clipboard history is incredible... the search in Google Images is sweet... there's a ton of stuff that's sitting in this little app that I feel it's something I need to spend about a month on to really find all the value in it. Until I have the time, I'll just use it as I do, and be very happy.

[10/22] UPDATE: it turns out that I didn't miss the final release of 5.0 - this was the final release. They just didn't let 5.0 see the light of day. Interesting. I don't feel nearly as bad about this.

Today I was trying to make headway on a problem I was struggling with and got word from the users that there was a data error in the web app I inherited - in production. Not good.

The problem was that they believed that the data they were seeing was not correct, and while I had changed a lot in the code, I hadn't changed that. Assuming they were right, it would mean that the code has been wrong for 9 months! Yikes.

Nothing like getting blasted from something you never expected.

I started digging into the code (no comments, of course) and realized what I thought was the problem. I spent the afternoon putting in a fix for this problem, and got it running before the end of the day. Unfortunately, there's not enough time to really check what it's doing, so I'll have to check it tomorrow.

I sure hope this works.

Transmit 3.6.8 was updated this weekend with much improved Snow Leopard support. While it's not my favorite FTP client, it's a good one, and built into Coda, an excellent web site development tool. I still like to keep it around as it's got capabilities that are far more than simple FTP, and when you need those, it's usually a panic situation and you really need them. Good tools to have.

Shimo 2.2.2 was released this weekend, and I had to get the update. It's the slickest, cleanest, way to deal with VPN issues that I've seen on Mac OS X. I can connect, do work, and disconnect without any issues at all. It just works. That's a lot better than the old VPN software I used to use at my previous job. It wasn't nearly as nice. If you have VPNs to deal with - really consider this. It's incredible.

I wanted to like Miro, as I like the concept behind it - Community Commons licensed content scoured from the web and displayed in a nice application, but it's the content that has left me wanting more. What's there is nice, but it's just not the content I was looking for. Even so, Miro 2.5.3 is out, and I decided that I needed to get it just on the off-chance that I'll someday find that the content is there, and I'll have a nice tool for viewing it. Hope springs eternal.

I've got two large Java web apps that I'm working on - one I wrote from the ground up and another I inherited. Both have the problem that they are getting very large for the boxes we have, and I need to be doing something about their memory footprint. The problem is, in both cases, the values are relatively important and thinning the data footprint out is not as trivial as it seems.

Multi-Level LRU Caches

In one of the apps, I had recently put an LRU Cache so that I would not have to save every value, only those that had shown themselves to be of use to the users. Sounds like a good idea. But there's a limit to that. Each operation on the BKLRUCache is going to re-order the aging list, and if a lot of updates come into the system in short order, there's a possibility that they will purge the cache of "good" values.

Sure, you can increase the size of the cache, but that defeats the purpose. You can also buffer up the changes and treat them as a single batch, but that has negative effects as well. No, the idea is to somehow "flag" the requested entries so that they stay over the "untouched" ones in the cache. But what's the easiest way to accomplish that?

What I came up with, after much hassle and arguing with myself, was a two-level LRU cache. The primary level is the same, fixed size. When the user asks for a value, that value is first looked for in the secondary LRU Cache, and if it's not there, the primary cache is checked. If it's there, it's put into the secondary cache so that it's not competing with the other entries that may never be touched by a client.

I don't have to worry about the size issues - they are independent caches, and I can size them to suit my needs. I also don't need to worry about the flood of updates wiping out the one previously asked for. It only has to compete against the other requests. Very nice.

H2 File System Database Tests

In my other web app I've got a very large H2 in-memory database. This is getting to be a real problem. So much so that I'm starting to think about alternatives to the in-memory database. There's MySQL, PostgreSQL, SQLite, even H2 has a file-system database mode. So as I looked at each of these I realized that there's no way I would be able to tell the difference until I started running some tests.

Given that H2 has good specs and I already had it in the project, it seemed like a good enough place to start. What I did was just to make it not in-memory, but based on the local filesystem (simple disk), and run the application and see what the results were.

I was stunned to say the least.

I tried a lot of things, but I kept coming back to the factor of 6x to 10x in speed. It's just not possible to have a 3 second response time to the queries that I need to process. No way. So I must come up with something that makes it possible to get better speed while still not taking up a hundred GB of RAM.

No solution yet.

This afternoon I got a tweet about the newest release of Acorn 2.1 from Gus at Flying Meat. There's a lot of new stuff in this release - AppleScript support, color picker, rulers, etc. It's a nice update, and he's been busy. It's not been that long since 2.0 was released. Pretty impressive.

This morning I saw that iConquer 3.0 was released as a 64-bit app with full Snow Leopard support. I haven't played it in quite a while, but it's still my favorite implementation of Risk on the Mac. I've seen one on the iPhone, and have been tempted, but haven't taken the plunge. For now, this is how I play the game, and it's great to see that at least some work is being done on the code. For a while there, I was thinking it had been sold off and mothballed.