What's it all about, Alfie?

This morning I noticed that Panic had updated both Transmit 3.6.9 and Coda 1.6.6. The single note in the Transmit release notes indicates a bug in the replacing of folders. The Coda release notes are a little more extensive with fixes for Snow Leopard as well as a few FTP issues and "minor visual fixes".

In all, it's rare to see these guys push out two updates, but I'm happy they did. It always makes me smile to look at Coda - I'm going to be using it more as I move to Mac-based development of these AJAX web sites, and I'm looking forward to it.

I've been working with the Google Chrome Frame for IE this afternoon, and I noticed that it wasn't updating properly. In fact, it's not working well at all with the processing of the DataTable data sets. It used to work, but it seems not to be working now.

It could be my box, or my IE 8 install, but I've re-installed the Google Chrome Frame and that's not it. I've also re-installed Adobe Flash 10 as well - as that was coming up as an issue. I'm not sure about all the issues, but it seems that it might be related to the clicking on a URL from a link external to the web browser.

If I set FireFox 3.5 as my default browser, it works fine. Same for Google Chrome (the browser). But if I try to use the Chrome Frame in IE, I get some of the page rendered, and a few requests sent and returned, but there appears to be a bug in the Chrome script(s):

  Uncaught TypeError: undefined is not a function    native runtime.js:389

I tried to Google this error and see what might be at issue. I'm not sure if it's the Chrome install or the Chrome Frame or IE or the box. But it seems that the error only occurs on the Chrome Frame in IE. Very odd.

I'll keep an eye on it and see what I can see, but I'm guessing that it's just my box for some reason, and I'll give it a reboot one of these days and set everything back up.

[10/27] UPDATE: I posted something to the (moderated) Google group for the Chrome Frame. I'm not sure if I'll hear anything back - it's very new, and could be classified as experimental, and not supported. But it's worth a try. I found another box that had the problem that used to run, but no longer. So it's got to be something with the visualization widgets or the frame, or an incompatibility between the two. So maybe I'll get lucky.

[10/28] UPDATE: The responses I got were very nice. One needed an example, which isn't surprising, but the other pointed to a version of Chrome (the Browser) that's in the Chrome Frame install directory. He suggested that if it's in there, then it's not the Frame itself, but that version of Chrome. Indeed, it was. So I posted that to the group. I also put together a simple test HTML file and put it up on my site so that they can hit it and test their stuff. It'll be interesting to see what comes of all this because now it's clearly in their court.

[10/29] UPDATE: the latest news was that it was in the line:

  google.load('visualization', '1.1', {'packages': ['annotatedtimeline', 'table']});

but I have to put in both packages or the page isn't going to render properly, and I've tried using the '1' version of the package to no avail. So I wrote all this back and we'll see what they say. It's clearly all in their court as they have the browser, the frame, and the visualization stuff. It's just a matter of them getting things together and fixing what needs fixed.

I was messing around this morning trying to parse the HTTP GET variables in JavaScript so that my web app could "see" the arguments passed into it, and act on them accordingly. This was something that the users have asked for in the alerts from my web app - a link back to the web app so that they can easily "drill down" to the data to see what the alert was really all about.

So I did some Googling on this and I was very pleased to find something like this:

  /**
   * This function parses all the GET variables for the current page
   * and returns a map of the key/value pairs to the caller. This can
   * be used in some pages to get these GET variables for processing.
   */
  function parseGETVars() {
    var retval = [];
    var urlChunks = String(document.location).split('?');
    if (urlChunks[1]) {
      var pairs = urlChunks[1].split('&');
      for (var i = 0; i < pairs.length; ++i) {
        if (pairs[i]) {
          var parts = pairs[i].split('=');
          retval[parts[0]] = unescape(parts[1]);
        }
      }
    }
    return retval;
  }

with this, I can then simply parse the GET variables into a global (page) variable and then use it in the code to see what I need to see.

With this, I was able to pass into one of my pages, the default portfolio to show - as opposed to the first one in the available list. It's a pretty slick little thing. Very nice.

I've been thinking a lot lately about the ideal structure for application development. It's more than just what makes the developer productive, it's also what makes a great application. There are components of infrastructure, user-interface design, support, reliability, stability, responsiveness. All the things that work together to create truly exceptional software.

I've been thinking about this because recent jobs have had vastly different models for how things should be done. Is there a QA department? Is there an Operations group that must take ownership of the developed product? Who supports the app with the end users? How does feedback make it's way to the developer from the users?

There are a lot ways to do this - probably as many as there are companies on the map. But I'm looking at the few examples I've been exposed to up-close, for an extended period of time.

The Cube Farm - Stick to Your Assignment, Developer

Probably one of the most Dilbert-esque, and therefore, most common, forms of organization is to function under the belief that management knows best. That people can really be modeled as ants in an ant farm - interchangeable, fungible, assets that can be told what to do and they will execute on those instructions as well as if they had any input or control in the decision-making process. It's born out of the idea of the old Peter Principle:

In a Hierarchy Every Employee Tends to Rise to His Level of Incompetence.

Developers that are really good developers must be good managers, so they are put in charge of a small team. Do that well, and he must be a good manager, and so put him in charge of a few teams. This continues until he's no longer really good at his job, and the promoting stops. However, the damage is already done.

You now have a great developer in management, and so he believes that he knows how to do this development, and he's right - he knows. But only about how he does it. If there's a person that's really effective because they write the docs first and then use that as a template for the system, that's great. But it's not a universal imperative. Not everyone will work best that way. Maybe some people see a design unfold best when they start writing code and see where it takes them. This might be along the lines of the Extreme Programming movement. No two people are really interchangeable. Not really.

In this kind of environment, there are Teams for everything. There's a Team for requirements gathering. Another for design. Another for development. Another for documentation. Another for Q/A. All these groups are designed to be very good at their jobs, and individually, they are. But that doesn't alter the fact that the line of communication between the users and any one individual is very complex, and error-prone. It's like the school-kids game of "operator" - each kid whispering the word they heard to the next, and at the end of the class what comes out is often times not at all like what went in.

To get the users the very best application, every person in that complex system needs to be aware of what is going on in all the different groups. If they hear of something that should be told to another group, they need to be able to get that information to that group. Unfortunately, that's not how these groups typically work. The larger the group the less effective the communication. When in order to work, it needs to be just the opposite.

So while this seems like a good idea for development, it's never been really good at delivering a great product to the users. It can get useful stuff out there, but it's at the cost of time. This kind of structure runs very slowly.

The Tiny Shop - Jack-of-All-Trades Developer

One could think that the other end of the spectrum is the indie developer. Let's surround him with a few partners and call it the start-up or small business developer. This guy has to do everything. This includes doing the sales, marketing, support, requirements gathering, design, development, deployment, etc. There's just no one else to help. In my opinion, there's a lot to like about this situation, having been in it before - there's no one else to worry about dropping the ball. You have to understand the problem domain in order to write the system, but that means you have to really understand your customers - never a bad thing. You have to be able to balance the infrastructure "cool" with the practical and the user-interface - it has to perform acceptably, but it's got to look nice as well. You have to be good at almost everything in the pipeline.

There are plenty of downsides to the person that's a little weak in one or more of these areas. If you're horrible at UIs, then you're going to be very uncomfortable with the fact that there's no one else to lean on for a UI for your back-end. The same would be true for sales or support or domain knowledge. If you're not willing to come up to speed on these areas - or shop them out to someone who is, then you're going to be uncomfortable. But for those people that are used to this, it represents a wonderfully freeing opportunity.

You can write your own code because you understand the domain. Be it a drawing app, a photography app, a simulation of some kind, or something as focused as a cellular image processing app. If you understand the domain, and can code, this model really allows you to control the entire project - much like the old masters painters. They sat in a room, conceived of a painting and then did it. Period. There's a lot to be said for those that can do this, to enjoy this more than working as a single member of a much larger team.

But it's not without risks as you're only as good as your weakest link, and if you're doing it all, your product is only as good as your weakest subject. This isn't for everyone, that's for sure. But it's something I've had experience with, and found very satisfying.

The Partnership - A Hybrid Approach

In engineering, there's always the continuous spectrum where an optimization is done to find the most effective solution to the problem. Given that this is my background, I have to look at this same problem with this same bias. Where is the most effective balance?

My current thinking is that the best balance is a partnership with the right partner. You need to complement each other. I know how important this is as I've been in partnerships where this wasn't the case, and it ended horribly. You have to be willing to take control and take over for your partner when they are down, and likewise, you have to be able to trust your partner to do the same for you.

This doesn't mean that you need to identical skill sets, or even complementary ones. It's enough to overlap in some areas and be weak in others. It's part of the dynamics of the partnership. Have someone to lean on, but not someone that's going to ask you where the cover letter is for your TPS Report.

While I think there is value in a larger organization, and there's considerable freedom in the indie developer, an appropriate balance of the two is really where you have to live. If I had a bunch of money and never needed to make another dime, I'd be an indie developer and that's it. But if I need to actually sell something to someone, I think I'm shooting for the partnership.

It's been a really awful morning today. When I got into work there were problems with one of the apps I inherited, and it was ugly. There were data issues that caused problems with the code changes I'd had to put in at the request of the users. It was my fault, but I'd been pressured to get the changes out as soon as possible by the users. I didn't get to do all the tests that I would have normally done and it got me. Period. No excuses. I had to make several emergency fixes and get things up and going, but in the end things were going OK.

It's not a fun way to start the day... no fun at all. But it happens when you rush things. I'm learning from this mistake and I'm not going to push things into production. Period.

The next problem was that there was a problem with one of the data feeder apps, but it wasn't at all clear where the problem was. I tried restarts... no good. I tried re-installing the little controlling apps that manipulate the feeder code... no good. I tried re-installing the feeder app itself... no good. I'm clearly getting desperate now because I have more than ten other boxes feeding good data - but there's this one that's not working and I'm running out of options.

But hey... problems are solved by those that show up, and I'm here and no one else is. So I have to find the solution.

I keep trying to reset things and in the end, I get a question from one of the users asking why the feeder was dead. I told him I was working on it, but it was dead. He mentioned that this wasn't the only problem - other apps were having problems this morning.

Bing!

There was nothing I could do, and there was nothing I had done wrong. Some bad data had gotten into the database and it was messing a lot of things up. When they fixed up the data, I was able to restart the feeder and all was OK. But for that first 90 mins of the day, I was in a near panic mode.

No fun.

But hey, problems are solved by people that show up.

Sometimes I just wished more people showed up. I could really use the help.

I was looking at the changes I'd made to the SQL in my web app to speed things up, which was really nice, but it ended up still being an issue. You see, the speed was nice when there were no INSERT statements hitting the database, but when I got into the thick of things today, I realized that the real issues were classic - locks, amount of data moved, etc. No way around the simple facts.

What ended up happening was the time climbed to the point that we got back to the 400 - 500 msec range. What I realized was that there's no "free lunch", and to get the times down to reasonable values I needed to look at what I was doing not just messing with the engine and tables.

What I had been doing was re-fetching the same data for the entire timespan and invalidating the cache when any data comes in from any of the sources. It's foolproof, it's just making me do the same work over and over and over again. The best I was able to do was a constant access time that increases during the day. But in reality, it's easy to think of a better solution.

Let's leave the table in-place and then "refresh" it by looking at the last acquired data point and then using that as the start time for the same request. I then could use that data to make an identical table, and then merge that table to the bottom of the existing table.

What I end up with is updating the last data point and then if there's more data, it's going to be added to the end of the table. If we have multiple requestors for the same table, the data will be added for each one. We put a nice little synchronization on the table being refreshed so that we don't have multiple edits at the same time, and all is OK.

What I've seen when I implemented this was a dramatic increase in speed. More than a factor of 10x on all the tables of data in the web app. It's going to make the load on the server a lot less, and we'll be able to handle a lot more users on the same hardware. Nice.

Really, this isn't a real surprise - you can work harder and get a little bit more, or you can work smarter and really make a huge difference. Should have done this earlier, but I didn't need it, so I didn't do it.

OK, so maybe it's not really Black Magic but putting yourself in the place of the query planner/optimizer is a really big help. For instance, today I was working on trying to get a little more speed out of the following H2 (JDBC) SQL query:

  SELECT acquired, portfolio, leadFuturePrice, pnlSumNet
  FROM RAT_FirePortfolio
  WHERE acquired >= '2009-10-20 00:00:00'
  AND acquired <= '2009-10-20 23:59:59'
  AND portfolio NOT IN ('ED_NRML')

and the requests were returning in excess of 120,000 rows of data in about 600 - 775 msec. Not bad for an in-memory database, but I was thinking that the use of the "not in ()" was going to slow down the execution of the query because it would have to check each of the row's portfolio fields against each of the elements of the list. Not ideal.

So I got the idea to change it to a more direct version:

  SELECT acquired, portfolio, leadFuturePrice, pnlSumNet
  FROM RAT_FirePortfolio
  WHERE acquired >= '2009-10-20 00:00:00'
  AND acquired <= '2009-10-20 23:59:59'
  AND portfolio <> 'ED_NRML'

So I coded that up. At the same time, I realized that the exclusion of 'ED_NRML' was really a transitional artifact and was no longer needed. There are times when we drop data sources, and this keeps the new version from falsely showing the existing, but old, data. Now that we have completely transitioned off that, we didn't need it and the query simplified to:

  SELECT acquired, portfolio, leadFuturePrice, pnlSumNet
  FROM RAT_FirePortfolio
  WHERE acquired >= '2009-10-20 00:00:00'
  AND acquired <= '2009-10-20 23:59:59'

The times for this guy are in the 50 - 57 msec. Amazing!

We're looking at a factor of ten by removing the test. Now I had suspected that the "not in ()" clause was a little bit of an issue, but I had never guessed it was 90% of the time! That's amazing.

I'll never doubt again. I need to really hit these queries and make sure they are as clean and simple as possible. Simply amazing.

Today I spent most of the day trying to check the fix I made yesterday afternoon. I believe it's right, as I've stepped through the code and I can't see why it might be wrong, but I wanted the users to check the data. I was very hard to get them to verify the data.

The primary problem was that the data is sampled from one system, pushed into another system and then shown in a client. It's a bit of a travel, and the question is: How old is this data?

The problem I kept running into was that I could not prove the data right because there was much I could not tell about the processing. The original developer used Aspect logging, and while that did what he wanted (kept the logging out of the processing code), it meant that there were limitations on the logging because of that same fact.

I decided that it had to change, and started putting in the logging at all levels of the application so that when I was done, there would be one log file that would tell me:

• what machine gathered the data for the server (by hostname or IP address)
• what type of request was done
• when it was sent to the extraction process and how long he took
• time required to parse the packet received at the server
• time required to incorporate the data in the packet
• total time from start of grab to the successful incorporation of the data

This is the kind of logging this application needs to have. It's got to mean something and be useful.

In a way, I'm a little glad. After thinning out the old Aspect logging, it's starting to look like something I might have done from the start. I can quantify the delays so that I can tell the users exactly what they are seeing. It's a much better place to be.

They may not like it, but that's a different issue.

Today I was trying to make headway on a problem I was struggling with and got word from the users that there was a data error in the web app I inherited - in production. Not good.

The problem was that they believed that the data they were seeing was not correct, and while I had changed a lot in the code, I hadn't changed that. Assuming they were right, it would mean that the code has been wrong for 9 months! Yikes.

Nothing like getting blasted from something you never expected.

I started digging into the code (no comments, of course) and realized what I thought was the problem. I spent the afternoon putting in a fix for this problem, and got it running before the end of the day. Unfortunately, there's not enough time to really check what it's doing, so I'll have to check it tomorrow.

I sure hope this works.

I've got two large Java web apps that I'm working on - one I wrote from the ground up and another I inherited. Both have the problem that they are getting very large for the boxes we have, and I need to be doing something about their memory footprint. The problem is, in both cases, the values are relatively important and thinning the data footprint out is not as trivial as it seems.

Multi-Level LRU Caches

In one of the apps, I had recently put an LRU Cache so that I would not have to save every value, only those that had shown themselves to be of use to the users. Sounds like a good idea. But there's a limit to that. Each operation on the BKLRUCache is going to re-order the aging list, and if a lot of updates come into the system in short order, there's a possibility that they will purge the cache of "good" values.

Sure, you can increase the size of the cache, but that defeats the purpose. You can also buffer up the changes and treat them as a single batch, but that has negative effects as well. No, the idea is to somehow "flag" the requested entries so that they stay over the "untouched" ones in the cache. But what's the easiest way to accomplish that?

What I came up with, after much hassle and arguing with myself, was a two-level LRU cache. The primary level is the same, fixed size. When the user asks for a value, that value is first looked for in the secondary LRU Cache, and if it's not there, the primary cache is checked. If it's there, it's put into the secondary cache so that it's not competing with the other entries that may never be touched by a client.

I don't have to worry about the size issues - they are independent caches, and I can size them to suit my needs. I also don't need to worry about the flood of updates wiping out the one previously asked for. It only has to compete against the other requests. Very nice.

H2 File System Database Tests

In my other web app I've got a very large H2 in-memory database. This is getting to be a real problem. So much so that I'm starting to think about alternatives to the in-memory database. There's MySQL, PostgreSQL, SQLite, even H2 has a file-system database mode. So as I looked at each of these I realized that there's no way I would be able to tell the difference until I started running some tests.

Given that H2 has good specs and I already had it in the project, it seemed like a good enough place to start. What I did was just to make it not in-memory, but based on the local filesystem (simple disk), and run the application and see what the results were.

I was stunned to say the least.

I tried a lot of things, but I kept coming back to the factor of 6x to 10x in speed. It's just not possible to have a 3 second response time to the queries that I need to process. No way. So I must come up with something that makes it possible to get better speed while still not taking up a hundred GB of RAM.

No solution yet.