What's it all about, Alfie?

For the last two days I've been hip-deep in a ton of little UI changes and additions to my web app. I can understand the reasons for the changes - it's the fist time they have actually sat down and thought about it for more than 2 seconds. When they did, and started to think about what they really wanted, they had some changes. Unfortunately, it was a lot harder to do this all a week (or more) after the initial creation of the page(s), but I suppose better late than never, eh?

There were things like re-arranging the columns in a table, and because of the nature of the Google Visualization Table, you can't re-arrange the columns in the view - they're fixed. So until they add this feature, I'm stuck putting the data in the right order before it gets to the client. Not horrible, but when it hits several pages, it's just a little time-consuming.

I also had requests for a few new fields. Nothing major, but putting it on four different pages meant a lot of little updates and making sure to try and pull together as much code as possible so that it's not copy-n-paste reuse. It's the easiest way to add the same thing in several places, but it would lead to a mess later on. I'll probably do a bit of work tomorrow to try and coalesce this even more. It's getting to be big.

In the end, I got the changes made and emails sent to clients so they knew the fields were there. Good enough.

Over the weekend, I stopped by the Apple Store in Naperville to pick up two of their small aluminum keyboards to use with two machines at work that used to share a keyboard/mouse with a switch-box, but last week were "split" so that I needed two keyboards in a pretty small desk.

For this desk, I need to get rid of the numeric keyboards to save the space. Just have to.

So the question is "How do I make it work with linux?"

Thankfully, xmodmap is going to do everything we need, and all I need to do is to get the keycodes from the keyboard and put them into the .Xmodmap file that I've already got to swap CapsLock and Ctrl.

Open up a terminal and type:

  $ xev

What you'll see is a new window with every X event getting printed to the standard out. Lots of information there, but in addition, you'll see every key press and release - and therein you can find the keycodes.

What I found was the following:

From this I was able to create the .Xmodmap that made the "Apple" keys act as "Alt" keys (their default appears to be to act as "Windows" keys) so it's easier to do what I need:

  !
  ! Swap Caps_Lock and Control_L
  !
  remove Lock = Caps_Lock
  remove Control = Control_L
  keysym Control_L = Caps_Lock
  keysym Caps_Lock = Control_L
  add Lock = Caps_Lock
  add Control = Control_L

  !
  ! Make the Apple keys Alt keys
  !
  keycode 115 = Alt_L Meta_L
  keycode 116 = Alt_R Meta_R
  add mod1 = Alt_L Meta_L Alt_R Meta_R

I may try to map some of the function keys to PgUp and PgDn keys, as those would be nice, but for now, it's enough this morning to get the Alt keys working on the new keyboards as that's how I switch between workspaces - which is essential.

I really like the additional desk space... that's a real plus. Just a little more fiddling with the location of things, and I should be settled in with the new hardware.

Lately I've been considering going back out on my own. Maybe with a few friends, but much, much smaller than what Port-to-Port became. When you have a lot of employees it changes what you're responsible for, and I want to stay doing things, as opposed to managing the business of those doing it. And the reason for this contemplation is that we have been discussing things at the Shop - things like vacation policy, sick days, and the like.

It all started when a co-worker brought up the fact that many places had no vacation policy. Actually, that's not entirely true. Their policy was that they didn't need a complex policy. Rather, they used the simple statement:

Achieving a balance between work and rest is important for everyone. Work has to get done and responsibilities covered. Good people know how to manage both.

and while this is heavily paraphrased, it's the essence of the policy that I liked.

Responsibility. Personal Responsibility, on a corporate scale.

I don't take a lot of vacation time, but I'd like not to even think about it. I work exceptionally hard and deliver results and products that are far above the norm. It's just the way it is. I'm not going to apologize for working 11 hr days, and working faster and smarter than most of the folks around me. Not all, mind you, and when I get to work with really sharp people, it's a treat. Really.

But it's the concept that I have to count these days. Moreover, when there is a "pool" of vacation and sick days, most people will look at sick days and say "I'm not that sick today to use a vacation day" - and so they bring their germs to work. Thus putting all of us with families at risk. It's the culture that there's a limited resource (days off), and the use of them needs to be personally justified.

This can get so bad that when there are "use it or loose it" days, people take off when they don't really want to because they feel they have to use this resource. Silly, yes. But very real. I've seen it happen at the Shop and I've only been here nine months.

As this email 'conversation' developed, the Netflix corporate culture docs (recently released) came up. And they are amazing. It's been talked about quite a bit, and I have to agree with all of them. What gets me most today are these:

• Adequate performers get generous severance packages.
• Evaluate performance based on effectiveness and results.

They go on to say that they see themselves as a Major League baseball team. Pay well, treat well, but expect stellar performance. If not, then be gracious, and show them the door. It's perfect.

Recently, I've been struggling with the very slip-shod work of people not really paying attention. It's not that they are really doing their best and just aren't capable of doing better - it's that they are being careless because they think it doesn't matter. They are wrong. It all matters.

So while I don't hold out a lot of hope that things will change at the Shop, I can hold out hope that I'll find the magic ingredients to cut out on my own, or with a few friends, and set up some place that adheres to these same guidelines. It's about the only new job I'm interested in looking at.

This morning I noticed that CoRD 0.5.2 was out, and with it, a hope that I may never have to have another Windows box again. This is an incredible app - works every bit as well as Windows' own Remote Desktop client for everything I need: get to a machine, run what I need, fiddle with files and settings, and get out. It's amazing.

With the update it's running in 64-bit on Snow Leopard, and several crashes have been fixed. It's also got the self-install/on-disk-location checks which have been talked about a lot in the web lately for Mac apps. I'm encouraged when I see this kind of thing picked up by really nice open source projects. Clearly, these guys are good, and that's great news.

I'll keep using this as long as it's available because I never want to have another Windows box, and there's no way business is going to give them up.

I've been working pretty hard today - and for me that's saying something. There were a few things that were very near to completion, and I wanted to get them all out to the users today so I could write them off my list. I had a deployment to do, then I needed to incorporate a few more data sources to my web app, then I needed to update a few pages with additional features the users had asked for. Nothing took more than an hour or so, but there were a lot of those hour-long projects I wanted to get through.

By the time early afternoon rolled around I found that I was in a pretty crummy mood. I started to think about a conversation we had this past week about my need for appreciative feedback. She had suggested that I volunteer at the Apple Store (or get paid, either way) because the people coming in there are (usually) very appreciative of the Genius Bar's ability to fix things up.

I didn't have the heart to tell her about the bad stories I'd heard from the Genius Bar workers on the net, but that's not the point. She was saying that the work I do is nice, and it's fulfilling, but everyone wants a little positive feedback for their efforts. Otherwise, it's going to lead to burn-out.

As I was thinking of this I realized she was totally right. What's more, while I may have felt that my last job was far more hostile, it was also far more appreciative.

The harsh remarks of my last place were, at times, clearly over the top. But primarily confined to a few folks. It was certainly not universal. And when they came, it was clear they were issued by a person that was very upset about something - maybe the problem they were talking about, and maybe it was something else entirely. But in every case, I was able to clearly identify the issues involved and indicate if there was something that I could do to resolve the issue.

No question, there were folks there that could go over the top with verbal anger at times.

What I hadn't realized was that there were other folks - the majority of folks, in fact, that would go over the top with appreciation.

The majority of the support folks... the majority of the operations staff... even the majority of the traders were very appreciative of my work. True, I had been there many years and built tools that were the cornerstone of the risk analysis, but still... there were web systems I built, and legacy apps I maintained and all those people were appreciative of the efforts.

Even the loudest of the 'loud angry' folks was very appreciative at times.

Which brings me to why I feel so crummy this afternoon - there's virtually no appreciation in this position now. There's no "loud angry" people either, but if I had to accept the latter to get the former, I think I'd do it every day of the week. If I'm going to be working 11 hr days, and making a 90 min commute - each way, I'd like to know that what I do matters.

In the last four years of my old place, I almost never got yelled at. Plenty of times in the first three, but less and less as time went on. I was good at what I did, and when I said I'd get it done, it got done. No one had a reason to get angry at me. My stuff just worked. So while I was working in a hostile environment, I wasn't the target of much hostility. I was, however, the recipient of a good bit of the appreciation.

I miss that. I really do.

If I'm going to work for no appreciation, I'll still do it. I'll just pull up quite a bit short of where I've been working because there's no one giving back to me. Also, the faux appreciation of some folks doesn't count. You can tell when it's genuine, and when it's "just words".

So today has been an important realization for me. I realized that if I'm going to be leaving it on the track every day, I'm going to need to hear a genuine "good job" now and then. If I'm not going to get it, I can't really leave it all on the track -- I'll burn out. And I've been very close to burn-out the last few weeks.

So it's time to work as hard as I can - for myself. If I get more from the users, that's great, and I'll be able to give a little more, but if not -- and they are by no means required to do so, then I'll at least have enough left over for the family in the evening.

And I know they deserve it.

I spent the latter part of today getting all the AJAX and client (JavaScript) post-processing on the 'Fusion' page with the graph and table I started the other day. Getting the data wasn't bad, but unlike the previous table I created with the collapsing groups, these groups aren't clearly defined. In fact, the groups are contained in the table itself. This allows me to not have to worry about the group definitions, but at the same time, I needed to change everything about the old scheme because I didn't have the group definitions.

It wasn't horrible, but it took a little bit of thinking to get the summations, indentation, sorting, tagging and collapsing all working. A couple of hours of quiet work, which of course, turned out to be an entire afternoon because I don't have quiet, and making it (with my fingers in my ears) doesn't make for easy typing.

Yes... I wish I had quiet. But I don't.

In the end, I've got the code I needed, and it's off the the London users for them to kick around and send me any changes they might have.

I know it's been a recurring theme of late, but I just can't help from commenting on the fact that a Good Coder can take Bad Code and keep it running far, far longer than it's probably good for the organization. Additionally, it keeps on giving to the coder long after he's tried to fix things up. Primarily because you can't really re-write the bad code - then it'd be a new project. No, management wants you to fix it, and quickly.

Well today I came across another lovely little "gotcha" with this application I've inherited. The designer tried to make a nice inheritance system for Stocks, Futures, Options, and to his credit, the ideas are basically nice, but the implementation is... well... let's just say not what I'd have done. But it's there. That's not the problem.

No, today I realized that a new portfolio has basically a bunch of stocks, but in order to get the pricing and calculated data for the instruments, I had to do a special type of query for futures that have no options on them. It's like the data source was written to supply data for stocks, futures, and options -- but only if you had options on the underlyings. If you just had a stock - too bad.

So they made this hack that a future was basically a stock. Not really - that whole expiration thing swept under the rug... but it's what I'm stuck with. I had to write code to make stocks from futures, and put it into the code base and write all the JUnit tests, etc. in order to get this one portfolio properly evaluating.

Not my idea of fun. And I'm sure it's not going to be my last realization about this codebase.

I think one of the things I really like about the Finance Industry - certainly creating applications and visualizations for it, is the density of the visualizations and applications. Most finance applications are running on machines that have a ton of other things running, and the users want to see as much data as possible. I've seen traders with dual 30" monitors and an additional three 19" monitors all tied to a raft of machines - all just to get the data they need in front of their eyes for the trades. It's pretty impressive.

I have to say I'm the same way. I love the visual density of the information in graphs and well designed tables. So when the users in London asked me to essentially fuse two pages together - and two of the more complex, active, pages at that, I had to rise to the challenge.

This fusion page has the Google AnnotatedTimeLine widget on it for the intraday values of several portfolios, and it also has a Google Table widget to contain the product-level values for the components of those portfolios. There are a few other things, but these two are going to update independently of each other with the trigger being the same timer event. So they will be close, but not really in sync.

Today I did most of the HTML/CSS layout to get the components on the page. This is necessary because the AnnotatedTimeLine is really two stacked on top of each other to function as a double-buffered system. We draw to the 'back' one, then flip them. Pretty simple, but necessary because of the delays in updating the ATL with new data.

I'll be able to put the AJAX behind this when I get back on Tuesday. Nice to have a three-day weekend!

While doing some network testing/optimization recently, one of the network guys suggested I look at the jTDS parameter packetSize. He thought it might be something to look at if all else failed.

Since I had pretty much gotten to that point, I decided this morning to do those tests, and at the same time take a look at what H2 might say about performance tuning - as that was the destination of the data, after all.

The first step was to change the datatype in the database. According to the H2 docs:

Each data type has different storage and performance characteristics:

• The DECIMAL/NUMERIC type is slower and requires more storage than the REAL and DOUBLE types.
• Text types are slower to read, write, and compare than numeric types and generally require more storage.
• See Large Objects for information on BINARY vs. BLOB and VARCHAR vs. CLOB performance.
• Parsing and formatting takes longer for the TIME, DATE, and TIMESTAMP types than the numeric types.
• SMALLINT/TINYINT/BOOLEAN are not significantly smaller or faster to work with than INTEGER in most modes.

The DBA I'd worked with to set up the back-end database that I read from didn't like using the double datatype primarily due to rounding. I said it was OK, but relented when he pressed. I then used the same DECIMAL(19,6) in the H2 in-memory database as existed in the MS SQL Server database. Seems reasonable, but it flies in the face of the suggestion from the H2 docs.

Since it's all Java, and a Java Double is OK with me, I decided to change all the DECIMAL(19,6) columns in the in-memory database to double. The results were amazing. I was able to achieve more than a 50% increase in the rows/sec processed by this simple change. Additionally, I was able to see a significant reduction in the memory used for the web app after making this change.

All told, a wonderful suggestion.

Then I took to running tests with different values of packetSize. I got:

where the data is (rows/sec) processed from the back-end database into the in-memory database. Faster is clearly better.

What I found was that a size of 8192 was the smallest value that got good performance. So that's what I went with. With these changes, my 7 minute restart is down to about 2:20 - an impressive improvement.

Today I've spent a lot of the day trying to get the restart time of my web app down to a reasonable time. The problem is that I need to load upwards of two million rows from a back-end MS SQL Server back-end database into an H2 in-memory database on restart using the fastest JDBC driver for MS SQL Server I've heard of - jTDS. The reason for this is access speed of the web app. There's just so much data that needs to be available to the servlets that if I were to get it from the database, I'd have an access time that's 10x what I have now.

So I need to load a lot of rows from the back-end database. In the past, I had a restart time that was about a minute. Not horrible. Today I realized that I'm looking at something more like seven minutes. That's too long.

So I pulled in the network guys to see if they could find anything in the wiring in the server room, or settings on the box, because there were machines where the SELECT statements were executed significantly faster than my box. The question was Why?

To their credit, the network guys did an impressive job of digging into the problem. Really amazing analysis of the problem. Unfortunately, in the end, they didn't have anything that was going to significantly change the performance of the query processing for me. But I wasn't too surprised, either. There had to be things I could do to clean things up, and I had always suspected it was going to be up to me in the end.