Bad Code Just Keeps on Giving…

January 19th, 2010

GottaWonder.jpg

I know it's been a recurring theme of late, but I just can't help from commenting on the fact that a Good Coder can take Bad Code and keep it running far, far longer than it's probably good for the organization. Additionally, it keeps on giving to the coder long after he's tried to fix things up. Primarily because you can't really re-write the bad code - then it'd be a new project. No, management wants you to fix it, and quickly.

Well today I came across another lovely little "gotcha" with this application I've inherited. The designer tried to make a nice inheritance system for Stocks, Futures, Options, and to his credit, the ideas are basically nice, but the implementation is... well... let's just say not what I'd have done. But it's there. That's not the problem.

No, today I realized that a new portfolio has basically a bunch of stocks, but in order to get the pricing and calculated data for the instruments, I had to do a special type of query for futures that have no options on them. It's like the data source was written to supply data for stocks, futures, and options -- but only if you had options on the underlyings. If you just had a stock - too bad.

So they made this hack that a future was basically a stock. Not really - that whole expiration thing swept under the rug... but it's what I'm stuck with. I had to write code to make stocks from futures, and put it into the code base and write all the JUnit tests, etc. in order to get this one portfolio properly evaluating.

Not my idea of fun. And I'm sure it's not going to be my last realization about this codebase.

Dense Visualizations in the Finance Industry

January 15th, 2010

GoogleVisualization.jpg

I think one of the things I really like about the Finance Industry - certainly creating applications and visualizations for it, is the density of the visualizations and applications. Most finance applications are running on machines that have a ton of other things running, and the users want to see as much data as possible. I've seen traders with dual 30" monitors and an additional three 19" monitors all tied to a raft of machines - all just to get the data they need in front of their eyes for the trades. It's pretty impressive.

I have to say I'm the same way. I love the visual density of the information in graphs and well designed tables. So when the users in London asked me to essentially fuse two pages together - and two of the more complex, active, pages at that, I had to rise to the challenge.

This fusion page has the Google AnnotatedTimeLine widget on it for the intraday values of several portfolios, and it also has a Google Table widget to contain the product-level values for the components of those portfolios. There are a few other things, but these two are going to update independently of each other with the trigger being the same timer event. So they will be close, but not really in sync.

Today I did most of the HTML/CSS layout to get the components on the page. This is necessary because the AnnotatedTimeLine is really two stacked on top of each other to function as a double-buffered system. We draw to the 'back' one, then flip them. Pretty simple, but necessary because of the delays in updating the ATL with new data.

I'll be able to put the AJAX behind this when I get back on Tuesday. Nice to have a three-day weekend!

Optimizing jTDS packetSize for MS SQL Server

January 14th, 2010

While doing some network testing/optimization recently, one of the network guys suggested I look at the jTDS parameter packetSize. He thought it might be something to look at if all else failed.

Since I had pretty much gotten to that point, I decided this morning to do those tests, and at the same time take a look at what H2 might say about performance tuning - as that was the destination of the data, after all.

The first step was to change the datatype in the database. According to the H2 docs:

Each data type has different storage and performance characteristics:

  • The DECIMAL/NUMERIC type is slower and requires more storage than the REAL and DOUBLE types.
  • Text types are slower to read, write, and compare than numeric types and generally require more storage.
  • See Large Objects for information on BINARY vs. BLOB and VARCHAR vs. CLOB performance.
  • Parsing and formatting takes longer for the TIME, DATE, and TIMESTAMP types than the numeric types.
  • SMALLINT/TINYINT/BOOLEAN are not significantly smaller or faster to work with than INTEGER in most modes.

The DBA I'd worked with to set up the back-end database that I read from didn't like using the double datatype primarily due to rounding. I said it was OK, but relented when he pressed. I then used the same DECIMAL(19,6) in the H2 in-memory database as existed in the MS SQL Server database. Seems reasonable, but it flies in the face of the suggestion from the H2 docs.

Since it's all Java, and a Java Double is OK with me, I decided to change all the DECIMAL(19,6) columns in the in-memory database to double. The results were amazing. I was able to achieve more than a 50% increase in the rows/sec processed by this simple change. Additionally, I was able to see a significant reduction in the memory used for the web app after making this change.

All told, a wonderful suggestion.

Then I took to running tests with different values of packetSize. I got:

packetSize Portfolio Product
512 11,496 11,904
1024 11,636 13,650
2048 11,902 13,941
4096 11,571 12,703
8192 12,560 14,774
16384 12,447 14,744
32768 12,753 14,017
65536 12,680 15,038

where the data is (rows/sec) processed from the back-end database into the in-memory database. Faster is clearly better.

What I found was that a size of 8192 was the smallest value that got good performance. So that's what I went with. With these changes, my 7 minute restart is down to about 2:20 - an impressive improvement.

Skitch 1.0b8.5 is Out

January 14th, 2010

This morning I noticed that Skitch 1.0b8.5 was out with a few Snow Leopard fixes, but an additional extension on the beta period. I'm guessing this will continue for a while, but eventually they may have to charge for storage. If they do, I'll sign up as it's just amazingly valuable.

Nothing else like it in my opinion.

Performance Tuning jTDS Hitting MS SQL Server

January 13th, 2010

Today I've spent a lot of the day trying to get the restart time of my web app down to a reasonable time. The problem is that I need to load upwards of two million rows from a back-end MS SQL Server back-end database into an H2 in-memory database on restart using the fastest JDBC driver for MS SQL Server I've heard of - jTDS. The reason for this is access speed of the web app. There's just so much data that needs to be available to the servlets that if I were to get it from the database, I'd have an access time that's 10x what I have now.

So I need to load a lot of rows from the back-end database. In the past, I had a restart time that was about a minute. Not horrible. Today I realized that I'm looking at something more like seven minutes. That's too long.

So I pulled in the network guys to see if they could find anything in the wiring in the server room, or settings on the box, because there were machines where the SELECT statements were executed significantly faster than my box. The question was Why?

To their credit, the network guys did an impressive job of digging into the problem. Really amazing analysis of the problem. Unfortunately, in the end, they didn't have anything that was going to significantly change the performance of the query processing for me. But I wasn't too surprised, either. There had to be things I could do to clean things up, and I had always suspected it was going to be up to me in the end.

Acorn 2.2 is Out

January 12th, 2010

I just got a tweet about Acorn 2.2 being out. The release notes are impressive and there's a lot to play with - gestures, new UI features, bug fixes... it's a major upgrade. Gotta love that stuff.

Upgraded to WordPress 2.9.1 at HostMonster

January 12th, 2010

I saw today that when I upgraded to WordPress 2.9 a while back, it broke the TimeZone plugin, and all the posts I'd made were 6 hours off. I also found that there was a new version of WordPress out - 2.9.1, so I went to HostMonster and upgraded my installs to 2.9.1.

I then went to the General settings tab and set the TimeZone there to be 'Chicago'. At this point, the TomeZone plugin correctly reported the time offset, and I should be OK. But I'll have to check and see what happens the next time we come to a time change.

I’m Horrible at Accepting Other’s Sloppy Work

January 12th, 2010

cubeLifeView.gif

OK... here's something that I know I'm horrible at: Accepting other's sloppy work.

Yup, I know I'm no good at it.

When I've spent several months creating a new system based on solid design goals and refactoring where and when it's necessary, it really bugs me when someone with far less attention to detail starts slapping through the codebase like a drunk with a steel-bladed weed wacker.

I do my very best to make sure that when I go into someone else's code I stick to their coding conventions - whether I like them or not. This goes past the coding style to the design, flow, control, and even architecture of the app. It's not easy, but it's respectful of the work the person did to get the codebase to this point.

It's certainly possible that all this effort is misplaced. It could be that the codebase is a pile of junk, and the original author didn't put this much effort into it's original creation - but that's not the point. The point is that I am a visitor in this codebase, and as such, I should at least ask before I start moving functionality around, and when I do, I better make it look like the original author did it.

Well... as you can imagine, I've been the recipient of some help in one of my projects, and the author didn't talk to me about it (I would have done it entirely differently), and executed it with the same grace and skill as a drugged-out bull knitting a sweater.

I've done my best to not tell them to never do this again, but my displeasure is clearly evident. When I have to learn about major shifts in the code from svn update, and then to see the changes implemented so horribly, well... it's a hot button for me.

So take a few tips from me. If you're a visitor in someone else's code try to follow these simple rules:

  • Talk about the changes you're planning before you do them. Seems simple, but you'd be surprised how many people sit less than ten feet from one another and this doesn't happen. Take the time. It may not be a big deal, but it's going to make what you do seem a lot more like help, and a lot less like a savage beating.
  • When you write new code, stick to the established coding style. Again, seems simple enough, but for those that use IDEs, the vertical spacing that's essential for someone that doesn't use the same IDE can be really screwed up. Take the few seconds to make sure that the file you're about to check in looks like the original author wrote it.
  • If you move functionality, discuss alternatives first. This is an extension of the first point, but the importance is even greater. It may not be apparent by looking at the code what the author's choice might have been for the code you're moving, so it's really important that you discuss what they would have done, if they were doing it. For all you know, there's a reason or pattern you're missing and it should stay right where it is.
  • Rise to the level of their code. This might be impossible for some, but to as great an extent as possible, try to be that coder. You may not have the skills, but make the attempt. Look at the existing code... study it... learn from it. If this coder is better than you are, you'll know it. It'll show in their code and yours. Don't do a sloppy job. Rise to the occasion. Be better than you have to be. Be Excellent. In the end, you'll be a better coder.

Sorting in Hierarchal Google Table Visualization

January 11th, 2010

GoogleVisualization.jpg

Step two in my creation of a modified version of the Google Visualization Table widget is to add in the indentation, sorting and collapsing of the data so that on each recalculation of the data table it looks like it's supposed to look to the user. Some of these are a lot easier than others, but they all share a common theme that was started with the creation of the aggregate groups in the previous post - that is, there's a list (array) of groups and that dictates how to do each of these steps.

Adding Indentation

The first thing I wanted to tackle is the indentation required when all the groups are expanded. Since I'd calculated the aggregate rows, and sorted them, what I needed to do was to make it look nice when all these groups are expanded. It's going to be a simple manner of putting non-breaking spaces in front of the label on the row, but the question is how to do this with the existing structure I have for the groups? Answer: pretty simply.

In keeping with the portfolio-to-row index mapping, and a simple loop over all the groups in the array, we can use the recursive function:

  function indentGroupMembers(tbl, grp, map) {
    for (var e = 0; e < grp.members.length; ++e) {
      if (map[grp.members[e]] != undefined) {
        var row = map[grp.members[e]];
        // ...add in the necessary space
        tbl.setValue(row, 0, '&nbsp;&nbsp;&nbsp;' + tbl.getValue(row, 0));
        // see if this member is, in fact, a group itself
        for (var g = 0; g < groups.members.length; ++g) {
          if (groups[g].name == grp.members[e]) {
            indentGroupMembers(tbl, groups[g], map);
            break;
          }
        }
      }
    }
  }

The beauty of this is that we have already defined all the structure we need to do the complete indenting - no matter how deep it goes. All we needed to do is to identify if the member is a group, and if so, call it again. Works perfectly.

Adding Group-Level Sorting

Probably the most difficult part of this was getting the sorting done correctly. Each group had to be sorted properly - with the other groups at the same level. Then within each group, the members had to be sorted - but stay within the limits of the group. All this while still appearing to work with the Google Table widget. I was concerned.

The idea I settled on was a hybrid of the sorter and the indenter - what if I went through the groups - sorted their members and assigned numbers 1, 2, 3, ... for their position in the group. Then, we'd scale up the values based on their "depth" in the scheme.

OK, an example. If we had in the table the following data:

Portfolio Delta
Tech 101.00
   AAPL 41.00
   MSFT -10.00
   GOOG 70.00
Retail 0.00
   HD 55.00
   LOW -55.00

then we'd add in the 'sorting column', and then scan each group - picking out the values, placing them in a JavaScript array, sorting that, and then assigning values. The trick is to once again use JavaScript objects and take not only the value of the row, but the row index so it's easy to place them in the right order.

Portfolio Delta Sort
Tech 101.00 2
   AAPL 41.00 2
   MSFT -10.00 1
   GOOG 70.00 3
Retail 0.00 1
   HD 55.00 2
   LOW -55.00 1

So that now you can see that each group has it's order assigned. The 'indenting' trick is to then scale up the non-leaf nodes by a factor of 10 for each level and add it to the members so that the members are sorted with the groups. This only works if you have less than 10 members in a group. If you have more, then simply increase the factor so that the largest group's membership is covered.

When you're done applying the scale factor, you'll have something like this:

Portfolio Delta Sort
Tech 101.00 20
   AAPL 41.00 22
   MSFT -10.00 21
   GOOG 70.00 23
Retail 0.00 10
   HD 55.00 12
   LOW -55.00 11

At this point, it's pretty clear that the simple table sort on this column will get us what we're looking for. Pretty neat. It took me a while to figure out what I needed to do here, and the code for the factor application and sorting isn't trivial, but it's not hard, and it's all driven by the group definitions so it's very flexible.

Adding the Collapsing

Once the aggregation and sorting are done, the final step is the collapsing of the non-expanded groups. It's easy to see that we can have a JavaScript array with the names of the expanded groups in it - removed when it's collapsed, added when it's expanded, so that it really comes down to removing the members of the groups that aren't expanded.

Based on the same ideas as the indenting this function does the trick:

  function removeGroupMembers(tbl, grp) {
    for (var e = 0; e < grp.members.length; ++e) {
      // remove this guy (row) from the table
      removeRow(grp.members[e], tbl);
      // see if this member is, in fact, a group itself
      for (var g = 0; g < groups.members.length; ++g) {
        if (groups[g].name == grp.members[e]) {
          removeGroupMembers(tbl, groups[g]);
          break;
        }
      }
    }
  }

the difference here being that because the table is dynamic at this stage, we can't use the portfolio-to-row index map, and have to, instead, have the function:

  function removeRow(name, tbl) {
    var rowCnt = tbl.getNumberOfRows();
    for (var i = 0; i < rowCnt; ++i) {
      if (name == tbl.getValue(i, 0).replace(/&nbsp;/g,'')) {
        tbl.removeRow(i);
        break;
      }
    }
  }

where this function looks in the table for the portfolio name and then removes it from the table. This is going to be an expensive step, but it's the only way I know to do what's needed without doing more than what's needed. After all, the groups may be expanded, and for those, there's nothing to delete.

Put together with a little JavaScript and tags, clicking on the group in the table toggles it's inclusion in the expanded list, updates the data and re-generates the table. It's pretty slick. Pretty fast, and exactly what I needed.

Trouble with Java Default Values

January 11th, 2010

java-logo-thumb.png

I got a note from the London users of my web app - specfically the email alerts system I've built into the web app so that people don't have to watch any one display all day long - they can get a chat or email alert telling them a certain condition has occurred. In any case, this morning I got word that the specific alert wasn't working quite right. So I started digging into the code. And hit one of my biggest annoyances with Java... the timing of the setting of the default values for class instance variables.

Let's look at a class and it's subclass. The problem will be in the subclass, but it's not obvious at looking at the code:

  public class BaseAlert extends Object {
    /**
     * These are the constructors - make the default protected so it's
     * not called accidentally, and make the general form of the
     * constructor take the important params and set things up right.
     */
    protected BaseAlert() {
      // always do the super's constructor
      super();
      // now do the base initialzation
    }
 
    public BaseAlert(String aName, Properties aProp) {
      // do the base constructor
      this();
      // now do the initialization of my args
      setName(aName);
      // update the configuration from the Properties
      updateConfiguration(aProp);
    }
 
    /**
     * This method picks out all the values from the Properties to
     * configure this instance.
     */
    public synchronized void updateConfiguration(Properties aProp) {
      /**
       * Read the values I need from the map and use them. Simple.
       */
    }
  }

and now the subclass that uses the updateConfiguration() method call for the same purpose, but has independent instance variables:

  public class MyAlert extends BaseAlert {
    /**
     * This is meant to be an attribute of the specific class. The
     * default value should work, but it doesn't.
     */
    private String      _code = null;
 
    /**
     * These are the constructors - make the default protected so it's
     * not called accidentally, and make the general form of the
     * constructor take the important params and set things up right.
     */
    protected MyAlert() {
      // always do the super's constructor
      super();
    }
 
    public MyAlert(String aName, Properties aProp) {
      // always do the super's constructor
      super(aName, aProp);
    }
 
    /**
     * This method picks out all the values from the Properties to
     * configure this instance.
     */
    public synchronized void updateConfiguration(Properties aProp) {
      // do all the super's value picking first
      super.updateConfiguration(aProp);
 
      /**
       * Read the values I need from the map and use them. Simple.
       */
      setCode(aProp.getProperty("Code"));
    }
  }

Here's what should happen:

  • Creating an instance of MyAlert causes the subclass to be created.
  • The call to updateConfiguration() is done and the values are read in.

Here's the catch: the default values are applied after the constructor is called. No joke.

The upshot: the value of _code is always going to be null all the debugging isn't going to help, as I found out. It's in the way the JDK handles the creation of the classes. Very annoying.

Fix? Have no default instance variable values. That's about it.

Grrr...