This morning Apple updated iMovie '09 (8.0.6) to better handle video from Aperture. Now I'm not a big movie maker, but that seems a little on the edge, but what the heck - I'm glad they are making improvements. Who knows, maybe Marie will want to do this someday.
This morning I saw that Miro 3.0 has been released. It's the open source video access tool that can download content from a ton of places and update subscriptions, etc. similar to the iTunes Store for Open Source content. I first started looking at Miro when I wanted to be able to download episodes of House, M.D., but it turned out these were only available on Hulu.
Anyway, with this update, there are lots of interesting things but the question still remains: is it a better way to watch House? Don't know that's the case.
A few days ago I ran into a serious problem with my usage of the H2 database as an in-memory database for a ton of data. Like we're talking nearly 70 million rows in a day. It's a lot. The problem is that I need to be able to INSERT this same data into (optionally) two databases. This makes is most logical to build up a (large) SQL string and then send it to one, or more, databases through JDBC. Seems pretty simple.
Until it gets too big.
My errors were because I was adding a lot of data to this table. So much that it was blowing out the JDBC Connection. So I put in "breaks" into the SQL string and then chopped up the transactions by those breaks. What I didn't think about then, and did think about this morning was: What if I broke up the statements?
So I tried that.
// break up the SQL into it's component statements String[] parts = null; if (!error) { parts = sql.split("\n;"); if ((parts == null) || (parts.length == 0)) { error = true; log.warn("Unable to break up the SQL into statements!"); } } /** * Now let's make the JDBC Connection and Statement and then * execute each of the statements against it. */ try { ... for (int i = 0; i < parts.length; ++i) { stmt.executeUpdate(parts[i].trim()); } ... } catch (SQLException se) { ... }
I broke up the large SQL by the statement terminations, and then ran through all the statements one after another, until the complete package was done. The effect was dramatic: It worked!
Not just that, it was far better because now we were able to once again maintain the transactional integrity of the INSERTs so that I could use these tables with one another without having to worry about when certain rows were inserted into certain tables. Much better.
So it may still be a bug in H2, but it's easy enough to work around.
Today my web app started moving in a slightly different direction and I started working with the Google Visualization Line Chart. This is really exceptional. While I've liked the AnnotatedTimeLine, and all the things it allowed me to do very easily, it's the speed and ease of use that the Line Chart represents that really blew me away.
This guy is rendered in SVG graphics so there's no Flash to load, it's not got the fancy zoom, but maybe they'll add that at a later time if they get enough requests. Even so, it's fast, lightweight, and very nice visually.
What a great visualization tool!
This morning I noticed that Perian 1.2.1 was out with several bug fixes and supporting a new encoding. It's a great add-on for Quicktime, and it makes it a lot easier to look at unusual content in Safari. Nice to see the addition.
Today was an interesting experience in poor communication. I had put in a new section of code into Hemlock at the request of another developer, and implemented the logic as I understood it. I then said that he and Q/A needed to verify the accuracy of the implementation, and thought nothing more of it. If there was a problem, I'd hear about it, right? If not, then it was all good.
Yeah... right.
So I started the day by hearing that the values in the application were way out, and I spent quite a while trying to find the problem. I sent emails asking if there were any data changes made overnight - turns out there was. The data for this new feature was put in for one portfolio. Fancy that.
Now it was clear - they didn't test anything. They assumed it was OK, but didn't actually test the values to see if they were right. Lovely.
When ensued was a period of several hours where I was sitting next to the other developer trying to figure out what was wrong with the implementation I had. Now I've had this in source control for weeks, so he could have looked at it. But he didn't... that's OK, he was sitting right next to me, looking over my shoulder, and still he didn't see the problem.
I had to ask such incredibly detailed questions it was virtually explaining the code line-by-line. Yet, not really - for when I did that, he didn't catch the problem, and was unable to clearly communicate what he wanted me to do.
"Frustrating" doesn't even come close to describing it.
It's this "good enough" attitude that kills me. Hey, I have an idea - give me a sample code section in some language - I'll translate it, you don't have to do it for everything, but come on... throw me a bone here. Am I supposed to drag this out of you every time there's a change?
Don't get me wrong... this is a nice, decent, guy. He's personable, seems to make the traders happy, but I can't believe that his communication skills are so incredibly horrible. Really.
This morning, Unison 2.0.5 was released, and I know it's in good shape because once again, I was a beta tester for this release. It's not a really hard thing to do, but it does mean testing the code and making sure it's doing what they say, as well as what I need. It's really getting to be an incredible application. I'm very glad they decided to do ahead with the new codebase/design.
This morning I noticed that Firefox 3.6.2 is out and contains the requisite number of security fixes and performance enhancements. It's just another "point release" for the Firefox team, but good that they are making progress. I'm very interested in the new JavaScript engine they are planning, as I'd like to see Firefox overtake Google Chrome in that area. But until that day, I'll deal with these little improvements.
Today I've been fighting a bug in the H2 database that has been quite a puzzler. I'm not being overly critical of H2 as I think it's a great open source project, but when you see how non-obvious this bug is, I think you'll agree that there's a lot that still needs to be done with H2 in this regard.
To start, I am using H2 as an in-memory database for a web app that is collecting a lot of data - working on 9 million rows in memory. Not small. But the problem isn't about it's size, it's about a certain series of database inserts that cause, of all things, a Java StackOverflowException. Let's start with the table.
It's one of many tables in my in-memory schema, and it's pretty darn simple:
CREATE TABLE volatility ( acquired datetime NOT NULL, product VARCHAR(64) NOT NULL, expiration DATE NOT NULL, strike FLOAT NOT NULL, volatility DOUBLE, generationTime datetime NOT NULL, CONSTRAINT volatility_PK PRIMARY KEY (acquired, product, expiration, strike) );
This guy is getting hit with all the volatility data we have for a given portfolio - inserting hundreds of rows every few seconds. The primary key of acquired, product (or underlying), expiration, and strike make it very easy to look up this volatility data for just about any where clause I might write.
When I started putting together the SQL string for inserting the data, I started running into problems. Specifically, I don't use PreparedStatements because I can't be sure how many inserts I'll be doing and there's also the problem of being able to optionally do the same updates to the back-end database if the configuration is set-up that way.
I have my UAT box writing it's data to the back-end database, and all the others just read that when restarting. This flexibility requires that I have something that travels as an object, and can be used to update one or two databases without having to re-create anything. This leaves out the PreparedStatements as they are built off the connection, and that's database-specific. So I had to stick with straight SQL.
In general, that's not too bad, and I haven't had any problems with this scheme until today, but today I think I met my match.
What I found was that when I started feeding the system the data for the volatilities, I could have it all run smoothly if I limited the number of strikes to five or less. But if I limited it to 50 or less, or no limit at all, I'd get a SQLException with the message being that there was a 'general' exception: StackOverflowException. I had no idea why I had created something with excessive recursion, so I went through the code several times and couldn't find anything.
I saw that H2 had been updated to 1.2.132 (from 1.2.127) and thought that maybe it might be a bug in H2 that had been fixed. Don't get me wrong, this is a bug in H2, but pin-pointing it to some specific code snippet is not going to be trivial. If I have the time in the future, I might do it, but it'd be a long-shot and take the better part of a day to get the test data set up into a simple test app to send to the H2 developers.
Anyway, I got H2 1.2.132 and tried it - no difference. Crud.
So then I tried setting the auto-commit on the JDBC Connection objects to "ON". Initially, my code had it set to "OFF" with an explicit commit() called on the connection once the SQL statement is executed. But even with the auto-commit "ON", we had the same problems.
Next I tried breaking up the SQL itself into smaller blocks based on the number of volatility points per expiration. If there are more than five volatility points in an expiration, then I break the SQL into separate statements in the execution part of the code. Since these are somewhat stand-alone, it shouldn't cause any problems. But I had my doubts.
I was really surprised to see that this break-up based on the number of strikes for an expiration solved the problem completely. What a break!
What I don't understand is why.
I can see that 4 of the 6 columns are in the primary key - but that in and of itself shouldn't be the issue. Not for a recursion problem. I can see that I'm inserting several hundred of these at a time, but the problem appears somewhere below 50, so that doesn't make a lot of sense.
Nope, this is a real puzzle to me. If I get the time, I'll try to write a simple test app and ship it to the H2 developers to see if they have any idea as to why it crashed. Very odd indeed.
I got a tweet today that Acorn 2.2.2 was out with a nice list of fixes and features. It's my favorite image editor for the Mac primarily because I don't have to re-learn anything when I start to use it. GIMP is far too complex, and Photoshop is too, but I don't even try to get that one because it's so expensive for the little work I do.
Anyway, a great tool that works like a champ.