Cleaning Up Data for Public Relations

January 8th, 2010

Every app I've worked on has data problems. Period. In my recent web app, I have the ability to edit the in-memory database contents that forms the basis of the displays the user sees. I typically clean-up the data in the dev, test and production apps a few times a day, but there was concern that someone might hit test or prod and see bad data and have a bad impression of the system. So for public relations reasons, I needed to put in some way for the Test and Prod versions to automatically clean themselves.

I looked at cleaning the data. It's possible. I can add a servlet context parameter that contains the name of the machine so I can have different instances behaving differently, but I was still concerned about the state of the data. It might be "bad", but is that a reason to delete it from the back-end persistent store? I'd like to think not.

I worried about only cleaning up the in-memory database, but that had a lot of problems because the same INSERT statements that are used for the back-end database are used for the in-memory database. So what to do...

Then it hit me - a far simpler solution: Use the value in the servlet context as the limit in the SQL statement on the selected machines!

Silly me. The views would always look "clean", but the underlying data is still in the databases in it's raw state. Nice.

Sure, it's silly, but when you spend 20 mins trying to come up with a clever way to auto-clean the data - always thinking of it in the "clean" sense, you get stuck in a rut and something as simple as re-casting the idea to a filter seems amazing.

No one said I was a genius.

Starting to Add Hierarchy to Google Table Visualization

January 7th, 2010

GoogleVisualization.jpg

The primary user of a new web page I created came back to me with a nightmare request: take the table and make it a tiered drill-down table. The table is the Google Visualization Table, and it's not got a single provision for handling roll-ups of the data, or hiding/expanding groups - or even of groups at all. This was going to have to be something I implemented from scratch.

Ick.

I had all the data, and I could get the organization, but how to store the organization? How to expand/collapse it? There were a lot of things I needed.

The thing I wanted to get started today was the aggregation into groups. It's a pretty simple idea: there's a new row in the table that is a very simple attribution of the data on the rows of it's members. So I needed to be able to identify the groups in some order so that they would be calculated in a consistent manner. If there were going to be several levels in this new system, we needed to be able to correctly aggregate the data for all groups.

What hit me was a simple use of the JavaScript objects and arrays to organize the groups. Each group would have a name and list of members (for now), and they would be placed into an array in the order of processing, which would guarantee that the lowest-level groups were calculated first, and then those depending on these next, and so on.

Breakthrough.

The group definitions started out looking like:

  var g = 0;
  var groups = new Array();
  groups[g++] = { name: 'Housing',
                  members: ['ABC', 'DEF', 'GHI'] };
  groups[g++] = { name: 'Tech',
                  members: ['AAPL', 'IBM', 'GOOG'] };
  groups[g++] = { name: 'Retail',
                  members: ['BBUY', 'HD', 'LOW'] };

which means that if you want to re-order the calculation, just move the definitions in the JavaScript file - the g++ takes care of putting them in the right order. Also, with this we can then look at writing something like this to add the aggregated value rows to the end of the table:

  // create all the groups - in the right order
  for (var g = 0; g < groups.length; ++g) {
    createGroup(answer, groups, portMap);
  }
 
  /**
   * This function creates the aggregate row in the table
   */
  function createNewGroup(tbl, grp, map) {
    // create a new row at the end of the table - sort later
    var row = tbl.addRow();
    // add it to the map
    map[grp.name] = row;
    // ...set the name of the group in the right spot
    tbl.setValue(row, 0, grp.name);
    // now get the values for the group from the members present
    var val = 0.0;
    var colCnt = tbl.getNumberOfColumns();
    for (var c = 1; c < colCnt; ++c) {
      // reset the value for each column
      val = 0.0;
      // aggregate the numerics values - pick first non-numeric
      if (tbl.getColumnType(c) == 'number') {
        // sum up all the available numeric values
        for (var e = 0; e < grp.members.length; ++e) {
          if (map[grp.members[e]] != undefined) {
            val += tbl.getValue(map[grp.members[e]], c);
          }
        }
      } else {
        // get just the first available value
        for (var e = 0; e < grp.members.length; ++e) {
          if (map[grp.members[e]] != undefined) {
            val = tbl.getValue(map[grp.members[e]], c);
            break;
          }
        }
      }
      // save the value we have for the column
      tbl.setValue(row, c, val);
      // ...and set the formatted value as well
      if (tbl.getColumnType(c) == 'number') {
        if (val != null) {
          tbl.setFormattedValue(row, c, val.numberFormat('#,##0;(#,##0)');
        }
      }
    }
  }

The trick here is the mapping of the portfolio (or member) to row in the table. I created a function that did this for me:

  function mapNamesToIndex(tbl) {
    var map = new Array();
    var port = null;
    for (var i = 0; i < tbl.getNumberOfRows(); ++i) {
      // get the portfolio for this row
      port = tbl.getValue(i, 0);
      // map it into the array
      map[port] = i;
    }
    return map;
  }

With this, I'm able to make one scan through the table and map the row indexes for all values - making it much faster to do the aggregations.

With this, I'm able to get the aggregations created in the table. With the previous posting about how to sort on an arbitrary key, I can then pass this into that function to have the groups above their contents in the table.

Big day... not done, but a good start to what I need to get done.

Setting the Read Timeout on a URL Request in Java

January 6th, 2010

I got a request from a co-worker late yesterday to add in a timeout to the AJAX gathering code in my web app. It wasn't immediately obvious, but I spent just a few minutes and it turned out to be pretty simple after all. It's used when you need to pull in an XML file from a URL for parsing into a DOM. Not hard, but very important to get right, and in a nicely flexible way.

If you start with the code I had originally:

  InputSource    retval = null;
  URL   source = ...;
 
  try {
    retval = new InputSource(new InputStreamReader(source.openStream()));
  } catch (IOException ioe) {
    if (!ioe.getMessage().equals("Connection refused") &&
        !ioe.getMessage().startsWith("Server returned HTTP response code: 500 ")) {
      log.error("While trying to get the data from the URL, an IOException occurred: "
          + ioe.getMessage();
    }
  }

and then finally read in the JavaDocs that the call:

  source.openStream()

is really just:

  source.openConnection().getInputStream()

So the code can quickly become:

  InputSource    retval = null;
  URL   source = ...;
 
  try {
    URLConnection  conn = source.openConnection();
    if (conn != null) {
      if (aTimeoutMSec > 0) {
        conn.setReadTimeout(aTimeoutMSec);
      }
      retval = new InputSource(new InputStreamReader(conn.getInputStream()));
    }
  } catch (IOException ioe) {
    if (!ioe.getMessage().equals("Connection refused") &&
        !ioe.getMessage().startsWith("Server returned HTTP response code: 500 ")) {
      log.error("While trying to get the data from the URL, an IOException occurred: "
          + ioe.getMessage();
    }
  }

In the end, we added in more logging - even a stack trace by a co-worker to help them find out what the problem was with a server-side error. But I have to admit, Java isn't as clear as it could be on things like this. But the JavaDocs did have what I needed, and that was enough to find out what I needed.

Unison 2.0 is Out

January 5th, 2010

Unison.jpg

This afternoon, I noticed that Unison 2.0 was out. To be honest, I've been a beta-tester for several months, and I have to say that it's one of the best newsreaders out there. In my opinion, they lost a few things in this version - like the count of unread articles on each group, but the gains have really been nice. I've submitted a feature request to add those numbers back in, and I hope they decide to do it, but even if they don't it's OK.

There are a few little things I've mentioned to the Panic guys about Unison 2.0 - most are related to the space used on screen. I'm sure they'll either get to them, or decide that they have a better UI vision than I do, and perhaps they are right. Still... the Unison 1 UI was small and clean and nearly perfect. But let's see where they take this version.

I'm just glad it's out, I can get a license for good to show my support, and they're going to keep making progress with it. Excellent tool.

Neat Little Sorting Idea for Google DataTables

January 5th, 2010

I've been working on a new page in my web app - something that will display the latest data (or any point in time, actually) and the users wanted to have it sorted by a very unusual key - asset class, then region, then desk. It's nothing simple, and I was worrying about how I'd get this done in a clean way. I didn't want to implement another sorting scheme... and I didn't want to mess with the reorganization of the table. But I just wasn't seeing how I would be able to make this happen easily.

Then it came to me: the DataTable from Google (in JavaScript) and it's Java counter-part written by me, has the sort() method, and that could be used to organize the rows in any way I want if I provided it a suitable key. The trick was to give it that key.

If I classified the row labels into their asset class, region, and desk - by name, then I could use this to create the index I needed.

Asset Class Code
Equities A
Rates B
Commodities C

and then:

Region Code
USA A
Europe B

so that I can then (in JavaScript) map the portfolios to these regions and asset classes (same will go for the desks, but it's an unnecessary complication here):

  var assetClass = new Array();
  assetClass['One'] = 'Equities';
  assetClass['Two'] = 'Equities';
  assetClass['Three'] = 'Rates';
  assetClass['OneMore'] = 'Rates';
  assetClass['Huey'] = 'Commodities';
  assetClass['Louie'] = 'Commodities';
 
  var region = new Array();
  region['One'] = 'USA';
  region['Two'] = 'Europe';
  region['Three'] = 'USA';
  region['OneMore'] = 'USA';
  region['Huey'] = 'USA';
  region['Louie'] = 'Europe';

With all this static data established for each portfolio, I can then create the JavaScript function to sort a passed in table on a known column. Let's say for my page, the portfolio name is in the first column. I could look for it but the getColumnLabel() method on the DataTable, but I know it's there, and that saves a little time.

  function sortByAssetClassAndRegion(tbl) {
    // make a new column that we'll use to sort
    var col = tbl.addColumn('string');
    // populate it with the mapping of the names based on the attributes
    for (var row = 0; row < tbl.getNumberOfRows(); ++row) {
      var portfolio = tbl.getValue(row, 0);
      tbl.setValue(row, col, assetClass[port] + '-' + region[port] + '-' + port);
    }
    // now just sort by the new column
    tbl.sort([{ column: col }]);
    // delete it now as it's served it's purpose
    tbl.removeColumn(col);
  }

This is pretty nice in that it leaves the table sorted by this other key, but it's very quickly done. The key's composition can be changed in any number of ways to make this even more flexible. For example, I expanded on this to make a sorting order that was specifically set by the user - and had nothing to do with asset class or region.

First, make an array with the sorting order you want to have:

  var sortingOrder = new Array();
  var i = 0;
  sortingOrder['One'] = i++;
  sortingOrder['Two'] = i++;
  sortingOrder['Three'] = i++;
  sortingOrder['OneMore'] = i++;
  sortingOrder['Huey'] = i++;
  sortingOrder['Louie'] = i++;

then you can create the sorting function:

  function sortByDefinedOrder(tbl, order) {
    // make a new column that we'll use to sort
    var col = tbl.addColumn('number');
    // populate it with the mapping of the names based on the attributes
    for (var row = 0; row < tbl.getNumberOfRows(); ++row) {
      tbl.setValue(row, col, order[tbl.getValue(row, 0)]);
    }
    // now just sort by the new column
    tbl.sort([{ column: col }]);
    // delete it now as it's served it's purpose
    tbl.removeColumn(col);
  }

and then it can be called:

  sortByDefinedOrder(answerData, sortingOrder);

It's not rocket science, but it is a nice little tool to have in hand to make sorting easy.

Google Chrome 4.0.249.49 is Out

January 5th, 2010

This morning I was coming off vacation and noticed that Google Chrome for Mac OS X had been updated to 4.0.249.49. Not a major update, but it's nice to see them making progress. Typically, I'm on the 3.x series of Chrome, but that's on Windows, and thankfully I don't have to use that unless it's work-related.

It's great to see Safari get a little competition.

Fun Skiing Vacation

December 30th, 2009

We just got back from a great skiing vacation. Liza and the kids took Ski School and learned to ski - Angelina on her new snowboard. We learned a lot about each other - who's got the balance and who doesn't take a fall very well.

Totem Park

The big surprise for me was Marie. She was amazing on the slopes once she learned how to ski. She was shooshing with the best of them. Very fun to see her excel at this.

There were casualties. Liza's hip may never be the same. OK... maybe it will, but the swelling from two consecutive falls to the same spot is not going away anytime soon. Still, it's something she's really glad she learned in the post-40 age bracket.

Great fun.

It’s Tough to Stay Upbeat Working on Horrible Code

December 22nd, 2009

Today I got tossed into having to work with my nemesis project today. This is the code that I can never seem to get rid of, and will most likely follow me as long as I'm here. It's painful. Well, today I thought that I wouldn't have to implement a certain change because another project (written by a co-worker) was going to take over this part of the code, which has been the goal for a while, but I just wasn't aware that now was the time.

So I talked to the guy doing the work on the new project, and realized that he wasn't anywhere close to getting it done. This meant that I needed to do the work if it was going to get done in a timely manner.

So back into the muck and crud I went... I spent about half a day on it, and in the end, it's working, and should work just fine, but it's left me feeling very un-Christmas-like. Very.

It's been a very tough couple of months - November and December. Liza's been sick with migraines, and work has been end-of-the-year stressful, as it can be. I'm not feeling like I'd like to feel, and yet there's really no time to just say "Timeout!" and get into the mood. I've got a few days, and I'll be working on Christmas Eve because I'll be taking time off for a family vacation the week between Christmas and my birthday.

I know there's no way to outrun this codebase. There just isn't.

I also know there's no way to get the other members of my team to get their stuff done faster. They're doing the best they can, it's just not as fast as I can work. I get loaded down at the end of the month because I hit all my objectives, which is another reason it's stressful. I just wish I knew that I'd never have to work on this codebase again.

But I know better.

Sigh.

Wouldn’t Nib-ware be Great for Web Pages?

December 21st, 2009

Today I spent a good chunk of the day creating a page in my web app from two other pages. It's a request I received from a user that wants to see both views of the data, but wants to pack it in on the display - screen real estate being what it is on most desks. What I kept thinking while I was doing this is: Why isn't this easier?.

I'd love to see nib-ware for the web. Something where I can build a self-contained "widget" that I can drop into pages with a few personalizations. I suppose it's possible with the Google Visualization Widget toolkit, but I didn't think of that until just now. Maybe I'll look into it. What did come to mind was Interface Builder nib-ware: drop-in boxes of functionality that can be customized and then used over and over. The low-level details are all handled in the implementation of the widget.

This gets back to the idea of having an Interface Builder for HTML/JavaScript/AJAX systems in the first place. Something where I can create functions (or at least stubs of functions) and then hook the outlets and actions up so that a clicking of a checkbox fires of some action like hiding an element, or clicking on a button grabs the value of a textbox and adds it to a list... stuff that you can do in IB, but can't really do it HTML.

Of course, the trick is that the rendering is the biggie. That's where all these JavaScript and HTML toolkits and frameworks come in. I guess it's getting there - slowly, but it'd sure be nice to be able to use something like IB for this. It sure would make it a lot easier.

Upgraded to WordPress 2.9 at HostMonster

December 21st, 2009

Saw a tweet this morning from Daniel J. that WordPress 2.9 was out. So I headed over to HostMonster and used SimpleScripts to update my installs. It's easy and fast. About the only critique I have is that the backup files are strewn about in my public_html directory and I really have no idea what to do with them. I think I'm going to just delete the old ones and compress the most recent ones. That's got to be good enough for now.

In any case, this little video about the new features is really quite informative, and while I can't imagine doing image editing in WordPress, it looks nice for those that need this capability.