What's it all about, Alfie?

I just got done talking to my two teammates about the value of defensive coding. Specifically, that there is value in Java of checking the value returned from every method call - including the JVM's new operator. I'm stunned that these guys believe that it's "silly" to check for things like that. I say, that's the bloody point of checking things in code. Have they not read Code Complete? Have they never really built large, critical systems that simply didn't have the option of failing?

My guess is, the answer is No.

In my code, you'll see a lot of code that looks like this:

  /**
   * This method returns the reference to the BKIRCProtocol that is
   * going to be monitoring the traffic from the IRC Server while we are
   * connected to it. If there is no listener defined at the time this is
   * first called, then create one because we really need to have it.
   */
  public synchronized BKIRCProtocolListener getListener() throws BKException {
    if (_listener == null) {
      _listener = new BKIRCProtocolListener(this);
      if (_listener == null) {
        throw new BKDebugException("BKIRCProtocol.getListener() - no protocol
          listener was present and one could not be created. This is a serious
          Java allocation error and needs to be looked into.");
      }
    }
 
    return _listener;
  }

Where I follow a new with a test for the resultant against null. It's standard coding style for me. When I was at First Chicago, I had the great fortune to work with an incredible group of guys. They were pivotal in having me look at coding as something a group can do as opposed to a series of individuals. In that group, I learned the value of having the code look like one mind wrote it. Debugging and fixing was as easy for one as it was for all. The style wasn't exactly mine, but we agreed on one for the group, and it stuck. It was great.

I have been trying to re-create that kind of experience and to some extent, I've been able to succeed in very limited cases. But I have seen it again. When you look at coding as not your work, but the team's work, you start to see that there's more to it than what you think. You care about the next guy getting the call to fix a problem you might have created. But when you write as a team, it's impossible to tell who wrote it, and that makes debugging vastly easier.

I guess I'm just surprised that I'm talking with developers making six-figures and they are talking about "how often" in the error checking. It's just stunning. You only have to write it once, and then it's always going to be there. That's the "how often" you need to focus on.

Expect more. Build it better than it has to be.

Yesterday I was adding a feature to one page in my web app, and I wasn't happy with it at all. It was excessively crowded, and to me clearly looked like a forced fit. Yet, this is just what I was asked to do, and until I really completed it, I couldn't be 100% sure that it'd be such a horrible mess. But it was. Holy Cow! What a mess.

So this morning I decided that I was going to leave that page as-is and add the feature to the web app as a new page - specifically used to show historical data for some portfolios. I know it wasn't what my manager wanted, as he'd "talked" to me trying to understand what I was building. I told him to wait and see, and still he kept at me about trying to understand. I hope I made it clear that I wasn't interested in back-seat designers, but I'm guessing he didn't get the hint at all.

In any case, I decided that I was going to do this, and it didn't matter what he said. I was going to do it right, because I just knew there was a better way to visualize the data. So I didn't stop.

When I got done with the page - heck, I didn't need to get done with it, but just to the main visualization. It was clearly a vastly superior interface for this kind of data. Far cleaner, far clearer. Wonderful. I wish I had a screen capture of the two to compare, but it's not in the cards at this time. So it goes.

What impresses me most about this experience is that I did just what I wanted to do and came out on the other side with something that was vastly superior for the end user. I'm not going to be bullied into making a slap-dash product again.

Today I have been adding in the ability to see more than just a day in my web app using the Google Visualization Annotated Time Line (ATL). The problem is, the ATL is a good time visualization tool, and it's not really producing "good" looking plots for the weekly, monthly, and yearly data. Oh, it works, but the x-axis remains true to the date/time which means that when I show a week, two of the seven days are flat, and very uninteresting. But there's no way to "compress" the "blank" space in time.

The other problems are that some of the data is 'reset' every day - like the P/L. That means that every day it's going to start back at 0, and work it's way up (or down) that day. But in the end, it's going to go back to 0 at the start of the next day. While this is exactly what they want from a business and accounting point of view, it's not leading to very interesting looking graphs as they seem to just "jump up" a little and then hop back down to 0.

For a lot of the datasets, the trading day is about eight hours, which means that the majority of the graph really doesn't have anything "interesting" on it. This is exactly what you'd expect, but it leads to a view that's really not very nice to look at.

So in the end, while it works, and does exactly what I asked it to do, it's not really what I want, nor is it reasonably speedy. It's going through a ton of data, so when looking at the year, it's a good 30 sec to get new data. Not really "zippy", that's for sure.

I'm glad that I got the code to work - it's quite a bit of non-trivial code, but I can't really say that I like the way it looks. It's doing the right thing, but it's not what I think the users are going to want to see when they think "week", "month", and "year".

I've kicked it out to the group and we'll see what everyone has to say. I'm betting that the guy who asked for this is going to have a mixed reaction to it, and write a 42-page opus on what to do next to make it better.

Goody.

I'm not the easiest person to work with, this is not news to me. But as long as I've been at my current job, I've been working very hard not to be as blunt as I might wish to be with co-workers. Specifically, co-workers that are doing a really bad job. So imagine my surprise when today I was able to exchange emails with The Smartest Man Alive. TSMA, as I'll call him, is a developer at The Shop and he's got a pretty well deserved reputation for being a prima dona. But hey, he's also TSMA, so it's OK, right?

This morning I got the following email from a fellow developer in my group that was trying to get TSMA's stored procedure to work:

   From: Mary
   Sent: Today
     To: TSMA, Steve, Bob
Subject: RE: lock problem again

Steve said the lock is free now.

Bob looked in the logs and sees the lock acquire error happening 4-5
times a day. TSMA, this sounds like expected behavior, right?

Mary

I had heard about these locking problems yesterday, and had reported to Mary how many times I was seeing them happen. It was a lot. When these locks happen, we're losing data, and that's not good. So I have to give it to Mary for sending this back to TSMA.

His response was a little surprising to me:

   From: TSMA
   Sent: A Little Later
     To: Mary, Steve, Bob
Subject: RE: lock problem again

my code retries (up to several times with a short sleep in betwen)
on one of these deadlocks and I have not yet seen a case where all
retries fail -- so honestly do not know how often the deadlock hav
happened.  But a few times a day does not sound unacceptable to me.

TSMA

(I've left the typos in the emails just to remain honest to the communication)

And here's where I have to step in. I am able to separate myself from a really bad job - so long as it's not brought up in front of my face. You want to do a crummy job? Fine. Just don't expect me to depend on it, and don't do it in front of me. If I let it slide, then I look like I knew it was happening and did nothing to stop it. That's not OK with me.

So I wrote back:

   From: Bob
   Sent: A Little Later
     To: TSMA, Mary, Steve
Subject: RE: lock problem again

TSMA,
  While I don’t claim to know the details of this project, or the
database involved, I would think that any design that allows a
failure to occur through normal usage is a bad design.
  Deadlocks are something that needs to exist in database usage
to keep the database running in those rare conditions that multiple
locks are involved in a single query or transaction.
  When some process or request needs to ensure some kind of data
integrity, there are many perfectly acceptable ways to achieve this
without the process failing – multiple times in a day.
  Does your car fail to start multiple times and day in the course
of regular usage?
  Does your computer fail? Your phone? Would it be acceptable to
you if they did?
  I think you get my point.
  While it’s not my database, the idea that this kind of level of
service is justified because it only happens a couple of times in
a day is really a stretch, don’t you think?
  Let’s follow our CEO’s words: “Raise the bar.” Be better than
this.

      Bob

This was, of source, a huge mistake, and I clearly didn't realize that I was talking to The Smartest Man Alive, after all. His response was quick and decisive:

   From: TSMA
   Sent: Right Away
     To: Mary, Steve, Bob
Subject: RE: lock problem again

Bob,
You would have done much better if you stopped with the first part
of your first sentence since that's the only piece of your email
that's shows any understanding.  The deadlock in question is the
one the sp announces it is preventing.
 
Save your advice to cases where you actually first bother to ask
questions before making grandiose statements.

TSMA

and then just a minute later:

   From: TSMA
   Sent: One Minute Later
     To: Mary, Steve, Bob
Subject: RE: lock problem again

and by the way Bob, get rid of the patrionizing tone, although
*you* may not be better than this.

Well... that certainly showed me, didn't it?

I sat totally shocked by this response. I'll admit, he's a prima dona, but really... do we need to have the attitude? Maybe he felt I was over the top. Maybe, to him, I was. But when I've talked to several people about - including the Senior Partners in the firm, they think that the idea that a critical risk system fails 4 or 5 times a day is anything but OK.

The Shop is filled with people that have no accountability - or such limited accountability that they can all but break the law and get away with it. The problem isn't in the individual, it's in the organization. If the organization did, in fact, hire The Smartest Man Alive, then great for us. But if the cost is that he's incapable of delivering a reliable product, and when pressed on the issue insults, well... then maybe it's too high a price to pay.

It's not my Shop, so it's not my decision. And I have no illusions about the fact that there are literally dozens of these guys at this place. Each one associated with a group of traders that protect them so that they can nearly get away with murder.

The real long-term cost is that people don't want to work with this kind of individual, and that means that should something happen to him, the organization is at a real loss. There are also people like me that see this as the prime reason to avoid this person and, if necessary, duplicate his work because he's shown himself to be incapable of creating solid, reliable products with a decent support arrangement.

But I can sleep easier tonight knowing that I exchanged emails with The Smartest Man Alive... well... at least the most arrogant developer in The Shop.

Today I spent a little time looking at how to make my existing web app able to view data historically. The first cut through a lot of these pages has been to show the data for today and then not worry so much about showing it for the previous week or month. Well... I wanted to take a swing at that, and I was pretty happy with the results.

Once I figured out which database the data was in, and I did that by looking at the supplied date (thank goodness I had that in the query), it was a simple matter of direction: if it's historical data, hit the back-end database, if it's current data, hit the in-memory database.

Couldn't be any simpler than that.

Now there's still the issue of some of the conversion factors - those aren't currently loaded historically, and then there's the goal to have the pages span more than a day - like a week, or a month, or a year... but I'll get to those starting tomorrow. It's enough for me to be happy about the progress I made on all these pages in a very short amount of time.

Nice job.

Today I had one more in a very long series of conversations with a manager that seems to want to understand the technical details, but I suspect really only wants to be able to appear to understand them. Basically, he wants to write a good email, and appear to really have a complete and deep understanding of the technical issues surrounding the problem. While I admire his desire, the exercise really comes off as The Great Pretender.

He's not technically trained, and while he may have some Excel "programming" or Matlab experience, it's not the same as really writing a system. Which means he believes he can understand the ins and outs of all technical decisions, and to a point, he probably can. But there has to come a point when he accepts his limitations on understanding and trusts those people he has asked for their opinions.

For example, I shouldn't ask the car dealer about the specifics of the on-board computer with regards to the spark advance during high-torque intervals when the humidity is greater than 90%. It's certainly a technical question, but the real issue for me, as a driver, is Will the car drive as I expect it to? I don't need to know every single detail, but I'm sure there are automotive engineers that do know all this, and good for them.

I'm not arrogant enough to think my opinion is essential, but if you ask for it, and then try to argue with me about it because you believe you have the salient technical facts, then it's a bit... oh... let's call it "excessive". Let's say he's asking about how to build a database. Specifically, what hardware should he buy to get what he wants? In my opinion, he needs to ask what storage he needs, then balance that with the costs of a few different alternatives. In the end, he shouldn't be grilling people about the difference between 1GB drives and 2GB drives - that's a detail that he really doesn't need to be involved in because his area of expertise and responsibility is not which drive to pick - it's the basics of balancing the technical needs with the business needs.

This happens at least a few times every week.

Today it was an email about the packet contents Hemlock receives from it's satellite processes. He asked if it was XML? Nope, it's tab-delimited ASCII. He then asks about the XML he's seen on my monitor. Yes... that's XML, but that's coming out of Hemlock, not going in? OK, so what's going in, a text file? No... it's tab-delimited ASCII data stream. And so on...

I'm not trying to make it difficult, I started saying "it's text", but he kept digging, knowing that wouldn't sound technical enough. And while I admire his desire to use the proper terminology, if it were only that, he'd have written down ""tab-delimited ASCII" the first time I told him. Not wait for me to explain away his confusion and then write it.

If you want an answer, ask. But don't then argue about it. Just take it and go. Because i know he's not using this as a traditional classroom situation where he's not going to ask again. It's like this several times on each point.

In the end, I'm just annoyed.

I remember my Econ class at Purdue - the professor used this quote:

Don't try to teach a pig to sing, it'll only frustrate you and annoy the pig.

and that's exactly what I feel I"m doing with this manager. Very frustrating.

I was in a meeting this afternoon and realized that I've really lost faith in The Shop's ability to pull itself out of it's current problems in a reasonable timeframe. Moreover, I'm not really sure that I'm interested in trying to fix the problems here. Maybe that makes me less committed than those willing to stay, but I think that a lot of people here are here simply because it's a relatively easy paycheck. I just don't see the kind of commitment and passion that I've seen in the past that leads to real, substantive, change.

It's not just The Shop - it's also that I have management that's trying to be technically savvy, but really should just step out of the way and realize that their technical skills are really non-existent, and if they want second opinions to my advice (which is always OK), they need to ask someone else. Me explaining the pros and cons of a technical decision isn't really helping because there's no real opposing position. Having the non-technical manager trying to be that "devil's advocate" isn't feasible because he doesn't know the right questions to ask.

I'm just finding myself frustrated by all these encounters.

We have what appears to be a completely dysfunctional communication system between managers of essential groups... we have a pandemic of mediocrity in the development groups... and there seems to be no one in the upper management that is either aware of these issues, or interested in tackling them. While I can understand that changes to these things will take time, I don't see any evidence that any change is occurring at all.

When I voice my concerns to my manager, he says there are changes, but what I can't understand, and he can't explain is why some simple issues like the egos causing communication problems is allowed to exist for even one more day. While I know it's not my company, this is having an effect on everyone as it seems very clear that this situation is allowed to exist, even perpetuate, and it's soon going to be time to start choosing sides.

Why? It makes no sense to me.

If you have a major problem, set forth a plan saying how you plan on attacking it, and then do your best to do that. If you have two highly-paid professionals that are acting more like 5-year olds than professionals, get them in a room and treat them like the 5-year olds they appear to want to be.

Or offer them both a generous severance package and show them the door. That would certainly get other people's attention. The message would be loud and clear: No more. Period.

But this place doesn't believe in that - clearly. So I'm really left with no choice but to admit that I don't understand why this place functions as it does. That disconnects me from this place in a very fundamental way. From there, it's not far to the loss of faith.

I'm sad to see this come to pass. I'd love to see something give me hope that real, substantive change is in the offing. I want to see that spark - even if it's just from a small group of people. Something to make me believe again.

Today I spent the entire day changing the way in which data sets are selected for display on the graph. Doesn't sound like much - but Holy Cow! it was a ton of work. The page was a relatively simple volatility by strike viewer, but the users wanted to be able to see any set of product/expiration pairs on the graph. At the start of this, I was allowing them to see any set of expirations for a given product.

Big difference.

The problem was really two parts: creating the GUI for the selection process, and then modifying the SQL to get the sets of data, and then coalesce them into one final data set.

The way I solved the first problem was to have a traditional selection drop-down set: the portfolio, the product and the expirations. These all made referential sense: change the portfolio and the products would be updated. Change a product, and the expirations would be updated to reflect that product. It allows for the selection of a single, 'tuple'.

I then had 'Add' and 'Remove' buttons between this on the right, and a single-select multi-line HTML SELECT on the left. The user could then 'create' a tuple, 'add' it to the list, and if there was one thing in the list they didn't want, they could select it and 'remove' it. It works pretty well, I have to say. Not amazing, but it's clean, simple, and almost everyone will understand how to use it immediately.

The second problem was how to get all this data into a single, coherent, data set. The answer was pretty simple: don't. What came to me was the idea that each of the tuples could be easily turned into a DataTable, and then I can put all those tables into a list and iterate over them, and then through the table. It's really a lot like the code I had to run through the JDBC ResultSet, but this time it's a nested loop.

In the end, I had just what I needed, but it took a lot of time to get all the features of the GUI right, and to get the servlet working properly. There was an additional wrinkle in that one of the users wanted to be able to specify the way in which the strikes would be placed on the same x-axis. Hey... it's silly, but it's a user, and that's what I do.

Today I spent quite a bit of time doing work for a single desk because their desk developers were all busy building new infrastructure for the firm. Of course, the infrastructure team was busy doing something else totally unrelated to the new infrastructure, but that's really the point, isn't it?

There is a serious bit of disfunction here. While the firm's stated goals are communication, cooperation, and excellence, many groups are so completely walled off - either on accident, by the business, or more likely through the use of incredible egos, that they simply have no need to communicate with others. They are the pinnacle of excellence, and any cooperation they offer is out of the goodness of their incredible generosity.

For a place that says it doesn't like egos, it's all about egos.

So we play this silly game of shuffle responsibilities. Since the infrastructure group is not talking to the desk developers, the desk developers are building their own infrastructure. Since the desk developers aren't providing the product they are supposed to be building for the desks, the desks come looking to the global risk group for that work. My manager is interested in looking good to all the groups, so he allows them to ask me to build and maintain things.

Unfortunately, the buck stops with me, as I have to do their work and mine as well. It's a shame I don't have a group that could pick up my work - Hey, I know, maybe the infrastructure team can manage to do this stuff!

It's just amazing that this place is run like a playschool. Managers know of problems in communication, and they do nothing about it. They give their tacit approval because having a job here is having a job for life. It's sad that professionalism is so lacking from people that are supposed to be professionals.

In truth, a lot of these folks wouldn't last like this in another job. They'd have to actually work, and they don't here. It's very frustrating for me as I end up doing other people's jobs, and nothing is being done to stop it.

I'm having a difficult time trying to come up with a good, solid, plan for dealing with a serious micromanaging manager. He's a decent guy, but I'm not sure if it's "just his way", or if it's an issue of trust that causes him to be so involved in every single detail of the work I do. After a full year of delivering good stuff, and even asking me what I thought he should do in a lot of situations, I'd think we had established the trust necessary for him to be able to give me something to be done, and then walk away until I had something to show him.

But that is clearly not the case.

So I'm beginning to think that it's not trust at all that's the issue. Problem is, I really don't have any other idea what it could possibly be. I'm going to try and ask him about it in our monthly planning meeting coming up in a few days, but I wonder if he's even aware of the issue?

Probably not.

In fact, I'm guessing that he thinks this is what everyone likes. That, in fact, this is the preferred way a manager acts to his people. If he likes it, then why wouldn't we?

I think I may have my work cut out for me on this one.

May not even be possible, but I think it's worth at least a few questions to see if he's aware of the situation.