What's it all about, Alfie?

OK... here's something that I know I'm horrible at: Accepting other's sloppy work.

Yup, I know I'm no good at it.

When I've spent several months creating a new system based on solid design goals and refactoring where and when it's necessary, it really bugs me when someone with far less attention to detail starts slapping through the codebase like a drunk with a steel-bladed weed wacker.

I do my very best to make sure that when I go into someone else's code I stick to their coding conventions - whether I like them or not. This goes past the coding style to the design, flow, control, and even architecture of the app. It's not easy, but it's respectful of the work the person did to get the codebase to this point.

It's certainly possible that all this effort is misplaced. It could be that the codebase is a pile of junk, and the original author didn't put this much effort into it's original creation - but that's not the point. The point is that I am a visitor in this codebase, and as such, I should at least ask before I start moving functionality around, and when I do, I better make it look like the original author did it.

Well... as you can imagine, I've been the recipient of some help in one of my projects, and the author didn't talk to me about it (I would have done it entirely differently), and executed it with the same grace and skill as a drugged-out bull knitting a sweater.

I've done my best to not tell them to never do this again, but my displeasure is clearly evident. When I have to learn about major shifts in the code from svn update, and then to see the changes implemented so horribly, well... it's a hot button for me.

So take a few tips from me. If you're a visitor in someone else's code try to follow these simple rules:

• Talk about the changes you're planning before you do them. Seems simple, but you'd be surprised how many people sit less than ten feet from one another and this doesn't happen. Take the time. It may not be a big deal, but it's going to make what you do seem a lot more like help, and a lot less like a savage beating.
• When you write new code, stick to the established coding style. Again, seems simple enough, but for those that use IDEs, the vertical spacing that's essential for someone that doesn't use the same IDE can be really screwed up. Take the few seconds to make sure that the file you're about to check in looks like the original author wrote it.
• If you move functionality, discuss alternatives first. This is an extension of the first point, but the importance is even greater. It may not be apparent by looking at the code what the author's choice might have been for the code you're moving, so it's really important that you discuss what they would have done, if they were doing it. For all you know, there's a reason or pattern you're missing and it should stay right where it is.
• Rise to the level of their code. This might be impossible for some, but to as great an extent as possible, try to be that coder. You may not have the skills, but make the attempt. Look at the existing code... study it... learn from it. If this coder is better than you are, you'll know it. It'll show in their code and yours. Don't do a sloppy job. Rise to the occasion. Be better than you have to be. Be Excellent. In the end, you'll be a better coder.

Step two in my creation of a modified version of the Google Visualization Table widget is to add in the indentation, sorting and collapsing of the data so that on each recalculation of the data table it looks like it's supposed to look to the user. Some of these are a lot easier than others, but they all share a common theme that was started with the creation of the aggregate groups in the previous post - that is, there's a list (array) of groups and that dictates how to do each of these steps.

Adding Indentation

The first thing I wanted to tackle is the indentation required when all the groups are expanded. Since I'd calculated the aggregate rows, and sorted them, what I needed to do was to make it look nice when all these groups are expanded. It's going to be a simple manner of putting non-breaking spaces in front of the label on the row, but the question is how to do this with the existing structure I have for the groups? Answer: pretty simply.

In keeping with the portfolio-to-row index mapping, and a simple loop over all the groups in the array, we can use the recursive function:

  function indentGroupMembers(tbl, grp, map) {
    for (var e = 0; e < grp.members.length; ++e) {
      if (map[grp.members[e]] != undefined) {
        var row = map[grp.members[e]];
        // ...add in the necessary space
        tbl.setValue(row, 0, '&nbsp;&nbsp;&nbsp;' + tbl.getValue(row, 0));
        // see if this member is, in fact, a group itself
        for (var g = 0; g < groups.members.length; ++g) {
          if (groups[g].name == grp.members[e]) {
            indentGroupMembers(tbl, groups[g], map);
            break;
          }
        }
      }
    }
  }

The beauty of this is that we have already defined all the structure we need to do the complete indenting - no matter how deep it goes. All we needed to do is to identify if the member is a group, and if so, call it again. Works perfectly.

Adding Group-Level Sorting

Probably the most difficult part of this was getting the sorting done correctly. Each group had to be sorted properly - with the other groups at the same level. Then within each group, the members had to be sorted - but stay within the limits of the group. All this while still appearing to work with the Google Table widget. I was concerned.

The idea I settled on was a hybrid of the sorter and the indenter - what if I went through the groups - sorted their members and assigned numbers 1, 2, 3, ... for their position in the group. Then, we'd scale up the values based on their "depth" in the scheme.

OK, an example. If we had in the table the following data:

then we'd add in the 'sorting column', and then scan each group - picking out the values, placing them in a JavaScript array, sorting that, and then assigning values. The trick is to once again use JavaScript objects and take not only the value of the row, but the row index so it's easy to place them in the right order.

So that now you can see that each group has it's order assigned. The 'indenting' trick is to then scale up the non-leaf nodes by a factor of 10 for each level and add it to the members so that the members are sorted with the groups. This only works if you have less than 10 members in a group. If you have more, then simply increase the factor so that the largest group's membership is covered.

When you're done applying the scale factor, you'll have something like this:

At this point, it's pretty clear that the simple table sort on this column will get us what we're looking for. Pretty neat. It took me a while to figure out what I needed to do here, and the code for the factor application and sorting isn't trivial, but it's not hard, and it's all driven by the group definitions so it's very flexible.

Adding the Collapsing

Once the aggregation and sorting are done, the final step is the collapsing of the non-expanded groups. It's easy to see that we can have a JavaScript array with the names of the expanded groups in it - removed when it's collapsed, added when it's expanded, so that it really comes down to removing the members of the groups that aren't expanded.

Based on the same ideas as the indenting this function does the trick:

  function removeGroupMembers(tbl, grp) {
    for (var e = 0; e < grp.members.length; ++e) {
      // remove this guy (row) from the table
      removeRow(grp.members[e], tbl);
      // see if this member is, in fact, a group itself
      for (var g = 0; g < groups.members.length; ++g) {
        if (groups[g].name == grp.members[e]) {
          removeGroupMembers(tbl, groups[g]);
          break;
        }
      }
    }
  }

the difference here being that because the table is dynamic at this stage, we can't use the portfolio-to-row index map, and have to, instead, have the function:

  function removeRow(name, tbl) {
    var rowCnt = tbl.getNumberOfRows();
    for (var i = 0; i < rowCnt; ++i) {
      if (name == tbl.getValue(i, 0).replace(/&nbsp;/g,'')) {
        tbl.removeRow(i);
        break;
      }
    }
  }

where this function looks in the table for the portfolio name and then removes it from the table. This is going to be an expensive step, but it's the only way I know to do what's needed without doing more than what's needed. After all, the groups may be expanded, and for those, there's nothing to delete.

Put together with a little JavaScript and tags, clicking on the group in the table toggles it's inclusion in the expanded list, updates the data and re-generates the table. It's pretty slick. Pretty fast, and exactly what I needed.

Every app I've worked on has data problems. Period. In my recent web app, I have the ability to edit the in-memory database contents that forms the basis of the displays the user sees. I typically clean-up the data in the dev, test and production apps a few times a day, but there was concern that someone might hit test or prod and see bad data and have a bad impression of the system. So for public relations reasons, I needed to put in some way for the Test and Prod versions to automatically clean themselves.

I looked at cleaning the data. It's possible. I can add a servlet context parameter that contains the name of the machine so I can have different instances behaving differently, but I was still concerned about the state of the data. It might be "bad", but is that a reason to delete it from the back-end persistent store? I'd like to think not.

I worried about only cleaning up the in-memory database, but that had a lot of problems because the same INSERT statements that are used for the back-end database are used for the in-memory database. So what to do...

Then it hit me - a far simpler solution: Use the value in the servlet context as the limit in the SQL statement on the selected machines!

Silly me. The views would always look "clean", but the underlying data is still in the databases in it's raw state. Nice.

Sure, it's silly, but when you spend 20 mins trying to come up with a clever way to auto-clean the data - always thinking of it in the "clean" sense, you get stuck in a rut and something as simple as re-casting the idea to a filter seems amazing.

No one said I was a genius.

The primary user of a new web page I created came back to me with a nightmare request: take the table and make it a tiered drill-down table. The table is the Google Visualization Table, and it's not got a single provision for handling roll-ups of the data, or hiding/expanding groups - or even of groups at all. This was going to have to be something I implemented from scratch.

Ick.

I had all the data, and I could get the organization, but how to store the organization? How to expand/collapse it? There were a lot of things I needed.

The thing I wanted to get started today was the aggregation into groups. It's a pretty simple idea: there's a new row in the table that is a very simple attribution of the data on the rows of it's members. So I needed to be able to identify the groups in some order so that they would be calculated in a consistent manner. If there were going to be several levels in this new system, we needed to be able to correctly aggregate the data for all groups.

What hit me was a simple use of the JavaScript objects and arrays to organize the groups. Each group would have a name and list of members (for now), and they would be placed into an array in the order of processing, which would guarantee that the lowest-level groups were calculated first, and then those depending on these next, and so on.

Breakthrough.

The group definitions started out looking like:

  var g = 0;
  var groups = new Array();
  groups[g++] = { name: 'Housing',
                  members: ['ABC', 'DEF', 'GHI'] };
  groups[g++] = { name: 'Tech',
                  members: ['AAPL', 'IBM', 'GOOG'] };
  groups[g++] = { name: 'Retail',
                  members: ['BBUY', 'HD', 'LOW'] };

which means that if you want to re-order the calculation, just move the definitions in the JavaScript file - the g++ takes care of putting them in the right order. Also, with this we can then look at writing something like this to add the aggregated value rows to the end of the table:

  // create all the groups - in the right order
  for (var g = 0; g < groups.length; ++g) {
    createGroup(answer, groups, portMap);
  }
 
  /**
   * This function creates the aggregate row in the table
   */
  function createNewGroup(tbl, grp, map) {
    // create a new row at the end of the table - sort later
    var row = tbl.addRow();
    // add it to the map
    map[grp.name] = row;
    // ...set the name of the group in the right spot
    tbl.setValue(row, 0, grp.name);
    // now get the values for the group from the members present
    var val = 0.0;
    var colCnt = tbl.getNumberOfColumns();
    for (var c = 1; c < colCnt; ++c) {
      // reset the value for each column
      val = 0.0;
      // aggregate the numerics values - pick first non-numeric
      if (tbl.getColumnType(c) == 'number') {
        // sum up all the available numeric values
        for (var e = 0; e < grp.members.length; ++e) {
          if (map[grp.members[e]] != undefined) {
            val += tbl.getValue(map[grp.members[e]], c);
          }
        }
      } else {
        // get just the first available value
        for (var e = 0; e < grp.members.length; ++e) {
          if (map[grp.members[e]] != undefined) {
            val = tbl.getValue(map[grp.members[e]], c);
            break;
          }
        }
      }
      // save the value we have for the column
      tbl.setValue(row, c, val);
      // ...and set the formatted value as well
      if (tbl.getColumnType(c) == 'number') {
        if (val != null) {
          tbl.setFormattedValue(row, c, val.numberFormat('#,##0;(#,##0)');
        }
      }
    }
  }

The trick here is the mapping of the portfolio (or member) to row in the table. I created a function that did this for me:

  function mapNamesToIndex(tbl) {
    var map = new Array();
    var port = null;
    for (var i = 0; i < tbl.getNumberOfRows(); ++i) {
      // get the portfolio for this row
      port = tbl.getValue(i, 0);
      // map it into the array
      map[port] = i;
    }
    return map;
  }

With this, I'm able to make one scan through the table and map the row indexes for all values - making it much faster to do the aggregations.

With this, I'm able to get the aggregations created in the table. With the previous posting about how to sort on an arbitrary key, I can then pass this into that function to have the groups above their contents in the table.

Big day... not done, but a good start to what I need to get done.

I got a request from a co-worker late yesterday to add in a timeout to the AJAX gathering code in my web app. It wasn't immediately obvious, but I spent just a few minutes and it turned out to be pretty simple after all. It's used when you need to pull in an XML file from a URL for parsing into a DOM. Not hard, but very important to get right, and in a nicely flexible way.

If you start with the code I had originally:

  InputSource    retval = null;
  URL   source = ...;
 
  try {
    retval = new InputSource(new InputStreamReader(source.openStream()));
  } catch (IOException ioe) {
    if (!ioe.getMessage().equals("Connection refused") &&
        !ioe.getMessage().startsWith("Server returned HTTP response code: 500 ")) {
      log.error("While trying to get the data from the URL, an IOException occurred: "
          + ioe.getMessage();
    }
  }

and then finally read in the JavaDocs that the call:

  source.openStream()

is really just:

  source.openConnection().getInputStream()

So the code can quickly become:

  InputSource    retval = null;
  URL   source = ...;
 
  try {
    URLConnection  conn = source.openConnection();
    if (conn != null) {
      if (aTimeoutMSec > 0) {
        conn.setReadTimeout(aTimeoutMSec);
      }
      retval = new InputSource(new InputStreamReader(conn.getInputStream()));
    }
  } catch (IOException ioe) {
    if (!ioe.getMessage().equals("Connection refused") &&
        !ioe.getMessage().startsWith("Server returned HTTP response code: 500 ")) {
      log.error("While trying to get the data from the URL, an IOException occurred: "
          + ioe.getMessage();
    }
  }

In the end, we added in more logging - even a stack trace by a co-worker to help them find out what the problem was with a server-side error. But I have to admit, Java isn't as clear as it could be on things like this. But the JavaDocs did have what I needed, and that was enough to find out what I needed.

I've been working on a new page in my web app - something that will display the latest data (or any point in time, actually) and the users wanted to have it sorted by a very unusual key - asset class, then region, then desk. It's nothing simple, and I was worrying about how I'd get this done in a clean way. I didn't want to implement another sorting scheme... and I didn't want to mess with the reorganization of the table. But I just wasn't seeing how I would be able to make this happen easily.

Then it came to me: the DataTable from Google (in JavaScript) and it's Java counter-part written by me, has the sort() method, and that could be used to organize the rows in any way I want if I provided it a suitable key. The trick was to give it that key.

If I classified the row labels into their asset class, region, and desk - by name, then I could use this to create the index I needed.

and then:

so that I can then (in JavaScript) map the portfolios to these regions and asset classes (same will go for the desks, but it's an unnecessary complication here):

  var assetClass = new Array();
  assetClass['One'] = 'Equities';
  assetClass['Two'] = 'Equities';
  assetClass['Three'] = 'Rates';
  assetClass['OneMore'] = 'Rates';
  assetClass['Huey'] = 'Commodities';
  assetClass['Louie'] = 'Commodities';
 
  var region = new Array();
  region['One'] = 'USA';
  region['Two'] = 'Europe';
  region['Three'] = 'USA';
  region['OneMore'] = 'USA';
  region['Huey'] = 'USA';
  region['Louie'] = 'Europe';

With all this static data established for each portfolio, I can then create the JavaScript function to sort a passed in table on a known column. Let's say for my page, the portfolio name is in the first column. I could look for it but the getColumnLabel() method on the DataTable, but I know it's there, and that saves a little time.

  function sortByAssetClassAndRegion(tbl) {
    // make a new column that we'll use to sort
    var col = tbl.addColumn('string');
    // populate it with the mapping of the names based on the attributes
    for (var row = 0; row < tbl.getNumberOfRows(); ++row) {
      var portfolio = tbl.getValue(row, 0);
      tbl.setValue(row, col, assetClass[port] + '-' + region[port] + '-' + port);
    }
    // now just sort by the new column
    tbl.sort([{ column: col }]);
    // delete it now as it's served it's purpose
    tbl.removeColumn(col);
  }

This is pretty nice in that it leaves the table sorted by this other key, but it's very quickly done. The key's composition can be changed in any number of ways to make this even more flexible. For example, I expanded on this to make a sorting order that was specifically set by the user - and had nothing to do with asset class or region.

First, make an array with the sorting order you want to have:

  var sortingOrder = new Array();
  var i = 0;
  sortingOrder['One'] = i++;
  sortingOrder['Two'] = i++;
  sortingOrder['Three'] = i++;
  sortingOrder['OneMore'] = i++;
  sortingOrder['Huey'] = i++;
  sortingOrder['Louie'] = i++;

then you can create the sorting function:

  function sortByDefinedOrder(tbl, order) {
    // make a new column that we'll use to sort
    var col = tbl.addColumn('number');
    // populate it with the mapping of the names based on the attributes
    for (var row = 0; row < tbl.getNumberOfRows(); ++row) {
      tbl.setValue(row, col, order[tbl.getValue(row, 0)]);
    }
    // now just sort by the new column
    tbl.sort([{ column: col }]);
    // delete it now as it's served it's purpose
    tbl.removeColumn(col);
  }

and then it can be called:

  sortByDefinedOrder(answerData, sortingOrder);

It's not rocket science, but it is a nice little tool to have in hand to make sorting easy.

Today I got tossed into having to work with my nemesis project today. This is the code that I can never seem to get rid of, and will most likely follow me as long as I'm here. It's painful. Well, today I thought that I wouldn't have to implement a certain change because another project (written by a co-worker) was going to take over this part of the code, which has been the goal for a while, but I just wasn't aware that now was the time.

So I talked to the guy doing the work on the new project, and realized that he wasn't anywhere close to getting it done. This meant that I needed to do the work if it was going to get done in a timely manner.

So back into the muck and crud I went... I spent about half a day on it, and in the end, it's working, and should work just fine, but it's left me feeling very un-Christmas-like. Very.

It's been a very tough couple of months - November and December. Liza's been sick with migraines, and work has been end-of-the-year stressful, as it can be. I'm not feeling like I'd like to feel, and yet there's really no time to just say "Timeout!" and get into the mood. I've got a few days, and I'll be working on Christmas Eve because I'll be taking time off for a family vacation the week between Christmas and my birthday.

I know there's no way to outrun this codebase. There just isn't.

I also know there's no way to get the other members of my team to get their stuff done faster. They're doing the best they can, it's just not as fast as I can work. I get loaded down at the end of the month because I hit all my objectives, which is another reason it's stressful. I just wish I knew that I'd never have to work on this codebase again.

But I know better.

Sigh.

Today I spent a good chunk of the day creating a page in my web app from two other pages. It's a request I received from a user that wants to see both views of the data, but wants to pack it in on the display - screen real estate being what it is on most desks. What I kept thinking while I was doing this is: Why isn't this easier?.

I'd love to see nib-ware for the web. Something where I can build a self-contained "widget" that I can drop into pages with a few personalizations. I suppose it's possible with the Google Visualization Widget toolkit, but I didn't think of that until just now. Maybe I'll look into it. What did come to mind was Interface Builder nib-ware: drop-in boxes of functionality that can be customized and then used over and over. The low-level details are all handled in the implementation of the widget.

This gets back to the idea of having an Interface Builder for HTML/JavaScript/AJAX systems in the first place. Something where I can create functions (or at least stubs of functions) and then hook the outlets and actions up so that a clicking of a checkbox fires of some action like hiding an element, or clicking on a button grabs the value of a textbox and adds it to a list... stuff that you can do in IB, but can't really do it HTML.

Of course, the trick is that the rendering is the biggie. That's where all these JavaScript and HTML toolkits and frameworks come in. I guess it's getting there - slowly, but it'd sure be nice to be able to use something like IB for this. It sure would make it a lot easier.

Well... I've been doing a little playing with Coda, and while I have to say it's very nice, and it's got a ton of flexibility. I think it's the tool I've been looking for to do the larger scale, faster AJAX building and deployment. You can put the CSS for a web page in the page, which is quite nice, but you can also put it in a general CSS file, and share it among many pages. Nice flexibility.

Of course, with preview is Safari (WebKit), so the JavaScript is working in the preview, and while it's not exactly like Safari because I can't set my default fonts in the preview pane, it's probably a lot more representative of the typical user than I've been thinking my Safari and Firefox sessions are. For example, in Google Chrome, by WordPress blog looks very different - primarily in the font selection. I'm not sure which is right, because Safari, Firefox and Camino are all looking the same and it's only Chrome that's odd. Maybe it's just Chrome.

Anyway, it's probably a decent thing that the preview is very generic, as it'll force the proper use of CSS. That's important for a whole host of reasons, not the least of which is that these days you have to have CSS or the page looks horrible on just about any browser.

So I was thinking that with Coda, I had a really nice tool to build the pages. But then there's the complete object-oriented development tools. The plus there is that I have the Google DataTable in Java, and JavaScript, and that serialization is done very nicely, but in general, I really need to think about building up some JSON serialization tools for things like Maps and Lists in Java. It's not a complete 1:1 mapping, but there are a few things I could write that would make it a lot easier to serialize out some standard Java datatypes to the JavaScript world, and that would make it faster to build pages.

Still... writing the Java code isn't the long pole for AJAX development for me. It's more the initial building of the HTML page based on all the ways of putting it together. All the different tags and structures are more than a little challenging on multiple browsers, but it's something I'm getting used to, and in time, I feel, it's something that will come as second nature. But for now, it's the one thing that would benefit from a builder tool.

In the end, it's still one of the most enjoyable development environments I've coded in in a while. Glad that I have the tools to do it easily.

Today I spent quite a bit of time putting together a new page in my web app. I had a model to work off of, but there were enough new things I needed to work in that I got to stretch a little artistic muscle - thankfully, I didn't pull anything. But it got me thinking how much I'd like to have a layout editor that is sort of like the WebKit Inspector panel - in reverse.

I spent the majority of the time messing with the HTML to get the layout looking the way I wanted. I had to make it look nice in FireFox and WebKit (Google Chrome), which wasn't horrible, but it did take a lot more time than glueing it all together with JavaScript and the servlet backend. So it's got me to thinking a few thoughts.

First, I'm very interested in hitting Coda again and seeing how they handle the CSS. That could make it a lot faster right there. There's no way to do web design on multiple platforms without CSS. I can't imagine it. So... faster CSS means faster development. I'd like to hit Coda harder and see how it handles more complex pages.

Second, I'm interested in the idea that there might be a more accurate preview system. Maybe this is Coda, I'm not exactly sure, but I'm thinking that spending 15 sec. editing JavaScript followed by a minute of redeploying it, refreshing the web browser, checking it, going to the other box, repeating the process, all adds up. If the preview was more like the Interface Builder preview, then I'd know it was wired up and working exactly as I expected.

Third, I can see the great value in the JavaScript Frameworks like DoJo, etc. and now the buzz coming out about Apple's private framework for making iPhone web apps. These all make things nicer, but I'm looking for one to emerge as the standard - at least for the Mac community. There's too much to handle on a low-level to be doing this all over from scratch each time. Cross-browser compatibility among the biggies here. I just haven't taken the plunge - yet.

I guess that's about it for today. It's been a lot of fun because in general, once you get the HTML hammered out (the view), the controller (JavaScript) and model (servlets) are not too bad. Not a bad way to develop an app, really.