What's it all about, Alfie?

Seems there was an issue with a few of the serial numbers in NetNewsWire 3.2.2 so they cut a new version and it's out. No difference other than that, they say.

The beta releases of Adium are coming more quickly now. It seems that maybe they hit a critical point in the Snow Leopard conversion or got through a particularly nasty bug. In any case, 1.4b12 is out today and I had to get it. It's just too much a part of every day usage.

It's been a really awful morning today. When I got into work there were problems with one of the apps I inherited, and it was ugly. There were data issues that caused problems with the code changes I'd had to put in at the request of the users. It was my fault, but I'd been pressured to get the changes out as soon as possible by the users. I didn't get to do all the tests that I would have normally done and it got me. Period. No excuses. I had to make several emergency fixes and get things up and going, but in the end things were going OK.

It's not a fun way to start the day... no fun at all. But it happens when you rush things. I'm learning from this mistake and I'm not going to push things into production. Period.

The next problem was that there was a problem with one of the data feeder apps, but it wasn't at all clear where the problem was. I tried restarts... no good. I tried re-installing the little controlling apps that manipulate the feeder code... no good. I tried re-installing the feeder app itself... no good. I'm clearly getting desperate now because I have more than ten other boxes feeding good data - but there's this one that's not working and I'm running out of options.

But hey... problems are solved by those that show up, and I'm here and no one else is. So I have to find the solution.

I keep trying to reset things and in the end, I get a question from one of the users asking why the feeder was dead. I told him I was working on it, but it was dead. He mentioned that this wasn't the only problem - other apps were having problems this morning.

Bing!

There was nothing I could do, and there was nothing I had done wrong. Some bad data had gotten into the database and it was messing a lot of things up. When they fixed up the data, I was able to restart the feeder and all was OK. But for that first 90 mins of the day, I was in a near panic mode.

No fun.

But hey, problems are solved by people that show up.

Sometimes I just wished more people showed up. I could really use the help.

This morning I noticed that Twitterrific 3.2.1 was released with several nice little fixes:

• Added shortcut to view a tweet’s web page (shift-command-left arrow)
• Fixed bug with command-C not working on Snow Leopard
• Fixed a problem with tweets not being deleted
• Fixed a problem with window not appearing in all Spaces on Snow Leopard
• Fixed problems with URLs that use Unicode or CHOCKLOCK in the domain name

it's unusual to see these guys release the Mac version, so I had to get it - who knows when it'll be updated again? Still the best Twitter client on Mac and the iPhone I've used.

I was looking at the changes I'd made to the SQL in my web app to speed things up, which was really nice, but it ended up still being an issue. You see, the speed was nice when there were no INSERT statements hitting the database, but when I got into the thick of things today, I realized that the real issues were classic - locks, amount of data moved, etc. No way around the simple facts.

What ended up happening was the time climbed to the point that we got back to the 400 - 500 msec range. What I realized was that there's no "free lunch", and to get the times down to reasonable values I needed to look at what I was doing not just messing with the engine and tables.

What I had been doing was re-fetching the same data for the entire timespan and invalidating the cache when any data comes in from any of the sources. It's foolproof, it's just making me do the same work over and over and over again. The best I was able to do was a constant access time that increases during the day. But in reality, it's easy to think of a better solution.

Let's leave the table in-place and then "refresh" it by looking at the last acquired data point and then using that as the start time for the same request. I then could use that data to make an identical table, and then merge that table to the bottom of the existing table.

What I end up with is updating the last data point and then if there's more data, it's going to be added to the end of the table. If we have multiple requestors for the same table, the data will be added for each one. We put a nice little synchronization on the table being refreshed so that we don't have multiple edits at the same time, and all is OK.

What I've seen when I implemented this was a dramatic increase in speed. More than a factor of 10x on all the tables of data in the web app. It's going to make the load on the server a lot less, and we'll be able to handle a lot more users on the same hardware. Nice.

Really, this isn't a real surprise - you can work harder and get a little bit more, or you can work smarter and really make a huge difference. Should have done this earlier, but I didn't need it, so I didn't do it.

This morning I saw that NetNewsWire 3.2.2 was out, and yet when I tried to update from the in-app updater, I kept getting errors from the update server. I then tried to download it from the NetNewsWire site - that got me 3.2. Finally, I downloaded it from the VersionTracker link and I got it. I'm hoping that 3.2.2 handles updates better and I don't have to go through all this again.

OK, so maybe it's not really Black Magic but putting yourself in the place of the query planner/optimizer is a really big help. For instance, today I was working on trying to get a little more speed out of the following H2 (JDBC) SQL query:

  SELECT acquired, portfolio, leadFuturePrice, pnlSumNet
  FROM RAT_FirePortfolio
  WHERE acquired >= '2009-10-20 00:00:00'
  AND acquired <= '2009-10-20 23:59:59'
  AND portfolio NOT IN ('ED_NRML')

and the requests were returning in excess of 120,000 rows of data in about 600 - 775 msec. Not bad for an in-memory database, but I was thinking that the use of the "not in ()" was going to slow down the execution of the query because it would have to check each of the row's portfolio fields against each of the elements of the list. Not ideal.

So I got the idea to change it to a more direct version:

  SELECT acquired, portfolio, leadFuturePrice, pnlSumNet
  FROM RAT_FirePortfolio
  WHERE acquired >= '2009-10-20 00:00:00'
  AND acquired <= '2009-10-20 23:59:59'
  AND portfolio <> 'ED_NRML'

So I coded that up. At the same time, I realized that the exclusion of 'ED_NRML' was really a transitional artifact and was no longer needed. There are times when we drop data sources, and this keeps the new version from falsely showing the existing, but old, data. Now that we have completely transitioned off that, we didn't need it and the query simplified to:

  SELECT acquired, portfolio, leadFuturePrice, pnlSumNet
  FROM RAT_FirePortfolio
  WHERE acquired >= '2009-10-20 00:00:00'
  AND acquired <= '2009-10-20 23:59:59'

The times for this guy are in the 50 - 57 msec. Amazing!

We're looking at a factor of ten by removing the test. Now I had suspected that the "not in ()" clause was a little bit of an issue, but I had never guessed it was 90% of the time! That's amazing.

I'll never doubt again. I need to really hit these queries and make sure they are as clean and simple as possible. Simply amazing.

I was chatting with friends today and Adium let me know that 1.4b10 was available for updating. I had to say Yes to that! There's a lot of new things in this release - including a new release of libpurple. They are certainly be careful and getting a lot of things fixed in this release, and as long as it's running well on Snow Leopard, I'm content letting them get things done at their own pace. I just would like to know that things run well on Snow Leopard, and I'm not getting the sense of a lot of confidence in that area.

Soon, maybe. I hope.

I was looking this morning to see if Tomcat of some kind was available at HostMonster - just in case I was interested in doing a little servlet work or something like that, and found that while they don't support Tomcat, they do support PHP (knew that), and there was an update for WordPress to version 2.8.5.

WordPress is one of the most common platforms, and as such it's open to all kinds of hackers. This means that I need to stay as up to date as possible to make sure I don't get hacked. So I took the few minutes and got everything up to the stable release and now I'm ready to keep going.

Just feeling a little safer. I'll have to find a Tomcat install somewhere else.

They updated the Instant Hijack component of Audio Hijack Pro in 2.9.5 along with a few other bug fixes today. Nice to see the indie Mac development crowd doing so well.