What's it all about, Alfie?

Yesterday I was adding a feature to one page in my web app, and I wasn't happy with it at all. It was excessively crowded, and to me clearly looked like a forced fit. Yet, this is just what I was asked to do, and until I really completed it, I couldn't be 100% sure that it'd be such a horrible mess. But it was. Holy Cow! What a mess.

So this morning I decided that I was going to leave that page as-is and add the feature to the web app as a new page - specifically used to show historical data for some portfolios. I know it wasn't what my manager wanted, as he'd "talked" to me trying to understand what I was building. I told him to wait and see, and still he kept at me about trying to understand. I hope I made it clear that I wasn't interested in back-seat designers, but I'm guessing he didn't get the hint at all.

In any case, I decided that I was going to do this, and it didn't matter what he said. I was going to do it right, because I just knew there was a better way to visualize the data. So I didn't stop.

When I got done with the page - heck, I didn't need to get done with it, but just to the main visualization. It was clearly a vastly superior interface for this kind of data. Far cleaner, far clearer. Wonderful. I wish I had a screen capture of the two to compare, but it's not in the cards at this time. So it goes.

What impresses me most about this experience is that I did just what I wanted to do and came out on the other side with something that was vastly superior for the end user. I'm not going to be bullied into making a slap-dash product again.

Today I have been adding in the ability to see more than just a day in my web app using the Google Visualization Annotated Time Line (ATL). The problem is, the ATL is a good time visualization tool, and it's not really producing "good" looking plots for the weekly, monthly, and yearly data. Oh, it works, but the x-axis remains true to the date/time which means that when I show a week, two of the seven days are flat, and very uninteresting. But there's no way to "compress" the "blank" space in time.

The other problems are that some of the data is 'reset' every day - like the P/L. That means that every day it's going to start back at 0, and work it's way up (or down) that day. But in the end, it's going to go back to 0 at the start of the next day. While this is exactly what they want from a business and accounting point of view, it's not leading to very interesting looking graphs as they seem to just "jump up" a little and then hop back down to 0.

For a lot of the datasets, the trading day is about eight hours, which means that the majority of the graph really doesn't have anything "interesting" on it. This is exactly what you'd expect, but it leads to a view that's really not very nice to look at.

So in the end, while it works, and does exactly what I asked it to do, it's not really what I want, nor is it reasonably speedy. It's going through a ton of data, so when looking at the year, it's a good 30 sec to get new data. Not really "zippy", that's for sure.

I'm glad that I got the code to work - it's quite a bit of non-trivial code, but I can't really say that I like the way it looks. It's doing the right thing, but it's not what I think the users are going to want to see when they think "week", "month", and "year".

I've kicked it out to the group and we'll see what everyone has to say. I'm betting that the guy who asked for this is going to have a mixed reaction to it, and write a 42-page opus on what to do next to make it better.

Goody.

I'm not the easiest person to work with, this is not news to me. But as long as I've been at my current job, I've been working very hard not to be as blunt as I might wish to be with co-workers. Specifically, co-workers that are doing a really bad job. So imagine my surprise when today I was able to exchange emails with The Smartest Man Alive. TSMA, as I'll call him, is a developer at The Shop and he's got a pretty well deserved reputation for being a prima dona. But hey, he's also TSMA, so it's OK, right?

This morning I got the following email from a fellow developer in my group that was trying to get TSMA's stored procedure to work:

   From: Mary
   Sent: Today
     To: TSMA, Steve, Bob
Subject: RE: lock problem again

Steve said the lock is free now.

Bob looked in the logs and sees the lock acquire error happening 4-5
times a day. TSMA, this sounds like expected behavior, right?

Mary

I had heard about these locking problems yesterday, and had reported to Mary how many times I was seeing them happen. It was a lot. When these locks happen, we're losing data, and that's not good. So I have to give it to Mary for sending this back to TSMA.

His response was a little surprising to me:

   From: TSMA
   Sent: A Little Later
     To: Mary, Steve, Bob
Subject: RE: lock problem again

my code retries (up to several times with a short sleep in betwen)
on one of these deadlocks and I have not yet seen a case where all
retries fail -- so honestly do not know how often the deadlock hav
happened.  But a few times a day does not sound unacceptable to me.

TSMA

(I've left the typos in the emails just to remain honest to the communication)

And here's where I have to step in. I am able to separate myself from a really bad job - so long as it's not brought up in front of my face. You want to do a crummy job? Fine. Just don't expect me to depend on it, and don't do it in front of me. If I let it slide, then I look like I knew it was happening and did nothing to stop it. That's not OK with me.

So I wrote back:

   From: Bob
   Sent: A Little Later
     To: TSMA, Mary, Steve
Subject: RE: lock problem again

TSMA,
  While I don’t claim to know the details of this project, or the
database involved, I would think that any design that allows a
failure to occur through normal usage is a bad design.
  Deadlocks are something that needs to exist in database usage
to keep the database running in those rare conditions that multiple
locks are involved in a single query or transaction.
  When some process or request needs to ensure some kind of data
integrity, there are many perfectly acceptable ways to achieve this
without the process failing – multiple times in a day.
  Does your car fail to start multiple times and day in the course
of regular usage?
  Does your computer fail? Your phone? Would it be acceptable to
you if they did?
  I think you get my point.
  While it’s not my database, the idea that this kind of level of
service is justified because it only happens a couple of times in
a day is really a stretch, don’t you think?
  Let’s follow our CEO’s words: “Raise the bar.” Be better than
this.

      Bob

This was, of source, a huge mistake, and I clearly didn't realize that I was talking to The Smartest Man Alive, after all. His response was quick and decisive:

   From: TSMA
   Sent: Right Away
     To: Mary, Steve, Bob
Subject: RE: lock problem again

Bob,
You would have done much better if you stopped with the first part
of your first sentence since that's the only piece of your email
that's shows any understanding.  The deadlock in question is the
one the sp announces it is preventing.
 
Save your advice to cases where you actually first bother to ask
questions before making grandiose statements.

TSMA

and then just a minute later:

   From: TSMA
   Sent: One Minute Later
     To: Mary, Steve, Bob
Subject: RE: lock problem again

and by the way Bob, get rid of the patrionizing tone, although
*you* may not be better than this.

Well... that certainly showed me, didn't it?

I sat totally shocked by this response. I'll admit, he's a prima dona, but really... do we need to have the attitude? Maybe he felt I was over the top. Maybe, to him, I was. But when I've talked to several people about - including the Senior Partners in the firm, they think that the idea that a critical risk system fails 4 or 5 times a day is anything but OK.

The Shop is filled with people that have no accountability - or such limited accountability that they can all but break the law and get away with it. The problem isn't in the individual, it's in the organization. If the organization did, in fact, hire The Smartest Man Alive, then great for us. But if the cost is that he's incapable of delivering a reliable product, and when pressed on the issue insults, well... then maybe it's too high a price to pay.

It's not my Shop, so it's not my decision. And I have no illusions about the fact that there are literally dozens of these guys at this place. Each one associated with a group of traders that protect them so that they can nearly get away with murder.

The real long-term cost is that people don't want to work with this kind of individual, and that means that should something happen to him, the organization is at a real loss. There are also people like me that see this as the prime reason to avoid this person and, if necessary, duplicate his work because he's shown himself to be incapable of creating solid, reliable products with a decent support arrangement.

But I can sleep easier tonight knowing that I exchanged emails with The Smartest Man Alive... well... at least the most arrogant developer in The Shop.

Today I spent a little time looking at how to make my existing web app able to view data historically. The first cut through a lot of these pages has been to show the data for today and then not worry so much about showing it for the previous week or month. Well... I wanted to take a swing at that, and I was pretty happy with the results.

Once I figured out which database the data was in, and I did that by looking at the supplied date (thank goodness I had that in the query), it was a simple matter of direction: if it's historical data, hit the back-end database, if it's current data, hit the in-memory database.

Couldn't be any simpler than that.

Now there's still the issue of some of the conversion factors - those aren't currently loaded historically, and then there's the goal to have the pages span more than a day - like a week, or a month, or a year... but I'll get to those starting tomorrow. It's enough for me to be happy about the progress I made on all these pages in a very short amount of time.

Nice job.

This morning I noticed that the Flash 10.1 Release Candidate was out - 10.1.53.7. While I typically use ClickToFlash to not view Flash in Safari (my primary browser), when I have to use it, I want to make sure that it's the best and latest version of Flash. This branch of Flash is supposed to be much better performing on Mac OS X, and it's about time - the vast majority of creative professionals are on Mac, why hasn't Adobe made Flash a top-notch player on the Mac?

Well... it's here, and for all I need, it'll do nicely.

I fired up Google Chrome this morning and noticed that it seems to update itself on each start-up, if necessary. A nice feature, to be sure. The current release is 5.0.366.0 (dev) and I have to say that a lot of my previous concerns about Chrome are slowly, but surely, starting to disappear as the browser matures.

Isn't that always the case? Getting the basics is relatively easy. Getting it polished, well... that's another thing entirely. My latest concerns were really focused on the responsiveness of the browser. It seemed to take a lot longer to get the data from the server than, say, Firefox or Safari. I'm not sure if they have done anything special with the sockets, or if their "every page a process" scheme is causing some additional overhead that the other two don't have, but there was a definite lag in Chrome that wasn't in the other two.

Thankfully, 5.0.366.0 (dev) seems to have cleared up a lot of this. It's actually about on par with Firefox now. Very interesting. I may have to use it for a day or two to see what it's like in more significant usage.

Today I spent the entire day changing the way in which data sets are selected for display on the graph. Doesn't sound like much - but Holy Cow! it was a ton of work. The page was a relatively simple volatility by strike viewer, but the users wanted to be able to see any set of product/expiration pairs on the graph. At the start of this, I was allowing them to see any set of expirations for a given product.

Big difference.

The problem was really two parts: creating the GUI for the selection process, and then modifying the SQL to get the sets of data, and then coalesce them into one final data set.

The way I solved the first problem was to have a traditional selection drop-down set: the portfolio, the product and the expirations. These all made referential sense: change the portfolio and the products would be updated. Change a product, and the expirations would be updated to reflect that product. It allows for the selection of a single, 'tuple'.

I then had 'Add' and 'Remove' buttons between this on the right, and a single-select multi-line HTML SELECT on the left. The user could then 'create' a tuple, 'add' it to the list, and if there was one thing in the list they didn't want, they could select it and 'remove' it. It works pretty well, I have to say. Not amazing, but it's clean, simple, and almost everyone will understand how to use it immediately.

The second problem was how to get all this data into a single, coherent, data set. The answer was pretty simple: don't. What came to me was the idea that each of the tuples could be easily turned into a DataTable, and then I can put all those tables into a list and iterate over them, and then through the table. It's really a lot like the code I had to run through the JDBC ResultSet, but this time it's a nested loop.

In the end, I had just what I needed, but it took a lot of time to get all the features of the GUI right, and to get the servlet working properly. There was an additional wrinkle in that one of the users wanted to be able to specify the way in which the strikes would be placed on the same x-axis. Hey... it's silly, but it's a user, and that's what I do.

Today I spent quite a bit of time doing work for a single desk because their desk developers were all busy building new infrastructure for the firm. Of course, the infrastructure team was busy doing something else totally unrelated to the new infrastructure, but that's really the point, isn't it?

There is a serious bit of disfunction here. While the firm's stated goals are communication, cooperation, and excellence, many groups are so completely walled off - either on accident, by the business, or more likely through the use of incredible egos, that they simply have no need to communicate with others. They are the pinnacle of excellence, and any cooperation they offer is out of the goodness of their incredible generosity.

For a place that says it doesn't like egos, it's all about egos.

So we play this silly game of shuffle responsibilities. Since the infrastructure group is not talking to the desk developers, the desk developers are building their own infrastructure. Since the desk developers aren't providing the product they are supposed to be building for the desks, the desks come looking to the global risk group for that work. My manager is interested in looking good to all the groups, so he allows them to ask me to build and maintain things.

Unfortunately, the buck stops with me, as I have to do their work and mine as well. It's a shame I don't have a group that could pick up my work - Hey, I know, maybe the infrastructure team can manage to do this stuff!

It's just amazing that this place is run like a playschool. Managers know of problems in communication, and they do nothing about it. They give their tacit approval because having a job here is having a job for life. It's sad that professionalism is so lacking from people that are supposed to be professionals.

In truth, a lot of these folks wouldn't last like this in another job. They'd have to actually work, and they don't here. It's very frustrating for me as I end up doing other people's jobs, and nothing is being done to stop it.

Things are really heating up between some groups in Google and some groups in Apple. From what I've read, it seems like it's more of an Executive War, as opposed to a real "can't stand to look at you" war in the trenches. But hey, that's what makes the game of Executive Watching so darn much fun.

Well, this morning I was reading Slashdot and saw that Google announced that they would be working with Adobe on a new plugin API for browsers to support Flash, and including Flash in Chrome. This not only covers the initial install, but also any subsequent updates. This is a major slap in the face to Apple and it's push for HTML5 as the technology moving forward.

In a sense, I'm really happy that Google is doing this. I use Google's Visualization widgets daily at The Shop, and a better Flash in Chrome is going to do nothing but help me. That's fantastic. But in a larger sense, I think the better solution is to see the components I use ported to HTML5 Canvas or SVG graphics like some of the Visualization widgets.

My ultimate goal is to have a visualization platform that does what I need without crashing, or bringing the machine to it's knees. CUrrently, I have to do a lot of little "tweaks" to get a reasonably stable platform, and even that's a bit dodgey at times. Anything that Google does to make this better is a good move in my book.

That being said, if I saw that someone came out with an HTML5 Canvas-based graphing package like the Google Visualizations, I'd immediately check it out. If it had a better system of support and less problems in the deployment (I can't deploy on Firefox, and IE only with the Google Chrome Frame plugin), then I'd switch over and never look back.

I don't think Google is doing the right thing in the long-term, and that's what I've always thought they would do. Engineers giving into silly Executive battles is just silly. It's beneath them. Beneath us.

Stop it. Convert the AnnotatedTimeLine to HTML5 Canvas, and then stop messing with Flash. That's a winning strategy.

I'm having a difficult time trying to come up with a good, solid, plan for dealing with a serious micromanaging manager. He's a decent guy, but I'm not sure if it's "just his way", or if it's an issue of trust that causes him to be so involved in every single detail of the work I do. After a full year of delivering good stuff, and even asking me what I thought he should do in a lot of situations, I'd think we had established the trust necessary for him to be able to give me something to be done, and then walk away until I had something to show him.

But that is clearly not the case.

So I'm beginning to think that it's not trust at all that's the issue. Problem is, I really don't have any other idea what it could possibly be. I'm going to try and ask him about it in our monthly planning meeting coming up in a few days, but I wonder if he's even aware of the issue?

Probably not.

In fact, I'm guessing that he thinks this is what everyone likes. That, in fact, this is the preferred way a manager acts to his people. If he likes it, then why wouldn't we?

I think I may have my work cut out for me on this one.

May not even be possible, but I think it's worth at least a few questions to see if he's aware of the situation.