Archive for the ‘Cube Life’ Category

First Full Day in The New Space

Monday, March 29th, 2010

cubeLifeView.gif

Well today has been a whirlwind of activity. It's the first full day in the new space and there are a lot of things that aren't quite right. I had the benefit of unpacking over the weekend, so I'm ready to go, but there are a lot of people with issues ranging from "not too bad" to "pretty serious" and there is a small army of support folks here to make sure that we can trade today out of the new space.

At the same time, today I'm doing a lot of application changes and re-configurations that is supposed to help the London folks better manage their risk. Will it? Not sure. They say it will, so I'm doing it, but in the end, I've often wondered if these kinds of changes are really proven to be worth it, or if they just believe they need it, and want to see it and play with it to see if it really lives up to the hope.

It's about thinking about what's really needed as opposed to simply doing something with the hope that it's going to be good enough to get the job done. If not, then try again. While it's possible to achieve success like this, it's also just as likely that you'll take enough 'stabs' at the problem that it'll be dead. It should only be a last resort - not a preferred way of development and design (which is to say no design).

But I'm not being asked to correct these problems - only update the apps. So I shake my head and wonder how many of these I can send to The Daily WTF before I'm found out.

CoRD 0.5.4 is Out

Monday, March 29th, 2010

This morning I got a tweet saying that CoRD 0.5.4 was out, and as long as I might need access to a Windows box at work, this is the best way I've seen to make it usable on the Mac.

Finishing Up a Few Things and then The Big Move

Friday, March 26th, 2010

cubeLifeView.gif

Today I spent most of the time working on the latest page I put together for my web app - it used the Google Visualization AnnotatedTimeLine to display the at-the-money volatility versus time of a series of expirations for a given product. It allows the risk managers to see the change in underlying price and how it effects all these volatility values.

Interesting, but the big thing today was The Move.

Yes, indeed. The Shop is moving from it's current location to the next building over where some of the folks have already been working for the last few weeks. Today, this weekend actually, is when all the trading floor folks move. The space is big, nice, clear, clean, and new. I've seen it, but I'm not sure how it's going to feel once we all get in there.

I'll get to see how things really feel this weekend as I need to come in on Sunday to set things up and make sure everything survived the move. It'll be an interesting start to the week, that's for sure.

Working Around H2 Database “Bug”

Thursday, March 25th, 2010

H2_logo.png

A few days ago I ran into a serious problem with my usage of the H2 database as an in-memory database for a ton of data. Like we're talking nearly 70 million rows in a day. It's a lot. The problem is that I need to be able to INSERT this same data into (optionally) two databases. This makes is most logical to build up a (large) SQL string and then send it to one, or more, databases through JDBC. Seems pretty simple.

Until it gets too big.

My errors were because I was adding a lot of data to this table. So much that it was blowing out the JDBC Connection. So I put in "breaks" into the SQL string and then chopped up the transactions by those breaks. What I didn't think about then, and did think about this morning was: What if I broke up the statements?

So I tried that.

  // break up the SQL into it's component statements
  String[]  parts = null;
  if (!error) {
    parts = sql.split("\n;");
    if ((parts == null) || (parts.length == 0)) {
      error = true;
      log.warn("Unable to break up the SQL into statements!");
    }
  }
 
  /**
   * Now let's make the JDBC Connection and Statement and then
   * execute each of the statements against it.
   */
  try {
    ...
    for (int i = 0; i < parts.length; ++i) {
      stmt.executeUpdate(parts[i].trim());
    }
    ...
  } catch (SQLException se) {
    ...
  }

I broke up the large SQL by the statement terminations, and then ran through all the statements one after another, until the complete package was done. The effect was dramatic: It worked!

Not just that, it was far better because now we were able to once again maintain the transactional integrity of the INSERTs so that I could use these tables with one another without having to worry about when certain rows were inserted into certain tables. Much better.

So it may still be a bug in H2, but it's easy enough to work around.

Starting to Work with the Google Visualization LineChart

Wednesday, March 24th, 2010

GoogleVisualization.jpg

Today my web app started moving in a slightly different direction and I started working with the Google Visualization Line Chart. This is really exceptional. While I've liked the AnnotatedTimeLine, and all the things it allowed me to do very easily, it's the speed and ease of use that the Line Chart represents that really blew me away.

This guy is rendered in SVG graphics so there's no Flash to load, it's not got the fancy zoom, but maybe they'll add that at a later time if they get enough requests. Even so, it's fast, lightweight, and very nice visually.

What a great visualization tool!

Developers and Good Communication

Tuesday, March 23rd, 2010

Today was an interesting experience in poor communication. I had put in a new section of code into Hemlock at the request of another developer, and implemented the logic as I understood it. I then said that he and Q/A needed to verify the accuracy of the implementation, and thought nothing more of it. If there was a problem, I'd hear about it, right? If not, then it was all good.

Yeah... right.

So I started the day by hearing that the values in the application were way out, and I spent quite a while trying to find the problem. I sent emails asking if there were any data changes made overnight - turns out there was. The data for this new feature was put in for one portfolio. Fancy that.

Now it was clear - they didn't test anything. They assumed it was OK, but didn't actually test the values to see if they were right. Lovely.

When ensued was a period of several hours where I was sitting next to the other developer trying to figure out what was wrong with the implementation I had. Now I've had this in source control for weeks, so he could have looked at it. But he didn't... that's OK, he was sitting right next to me, looking over my shoulder, and still he didn't see the problem.

I had to ask such incredibly detailed questions it was virtually explaining the code line-by-line. Yet, not really - for when I did that, he didn't catch the problem, and was unable to clearly communicate what he wanted me to do.

"Frustrating" doesn't even come close to describing it.

It's this "good enough" attitude that kills me. Hey, I have an idea - give me a sample code section in some language - I'll translate it, you don't have to do it for everything, but come on... throw me a bone here. Am I supposed to drag this out of you every time there's a change?

Don't get me wrong... this is a nice, decent, guy. He's personable, seems to make the traders happy, but I can't believe that his communication skills are so incredibly horrible. Really.

Interesting Bug in H2 Database 1.2.132

Monday, March 22nd, 2010

H2_logo.png

Today I've been fighting a bug in the H2 database that has been quite a puzzler. I'm not being overly critical of H2 as I think it's a great open source project, but when you see how non-obvious this bug is, I think you'll agree that there's a lot that still needs to be done with H2 in this regard.

To start, I am using H2 as an in-memory database for a web app that is collecting a lot of data - working on 9 million rows in memory. Not small. But the problem isn't about it's size, it's about a certain series of database inserts that cause, of all things, a Java StackOverflowException. Let's start with the table.

It's one of many tables in my in-memory schema, and it's pretty darn simple:

  CREATE TABLE volatility (
    acquired        datetime    NOT NULL,
    product         VARCHAR(64) NOT NULL,
    expiration      DATE        NOT NULL,
    strike          FLOAT       NOT NULL,
    volatility      DOUBLE,
    generationTime  datetime NOT NULL,
    CONSTRAINT volatility_PK PRIMARY KEY (acquired, product, expiration, strike)
  );

This guy is getting hit with all the volatility data we have for a given portfolio - inserting hundreds of rows every few seconds. The primary key of acquired, product (or underlying), expiration, and strike make it very easy to look up this volatility data for just about any where clause I might write.

When I started putting together the SQL string for inserting the data, I started running into problems. Specifically, I don't use PreparedStatements because I can't be sure how many inserts I'll be doing and there's also the problem of being able to optionally do the same updates to the back-end database if the configuration is set-up that way.

I have my UAT box writing it's data to the back-end database, and all the others just read that when restarting. This flexibility requires that I have something that travels as an object, and can be used to update one or two databases without having to re-create anything. This leaves out the PreparedStatements as they are built off the connection, and that's database-specific. So I had to stick with straight SQL.

In general, that's not too bad, and I haven't had any problems with this scheme until today, but today I think I met my match.

What I found was that when I started feeding the system the data for the volatilities, I could have it all run smoothly if I limited the number of strikes to five or less. But if I limited it to 50 or less, or no limit at all, I'd get a SQLException with the message being that there was a 'general' exception: StackOverflowException. I had no idea why I had created something with excessive recursion, so I went through the code several times and couldn't find anything.

I saw that H2 had been updated to 1.2.132 (from 1.2.127) and thought that maybe it might be a bug in H2 that had been fixed. Don't get me wrong, this is a bug in H2, but pin-pointing it to some specific code snippet is not going to be trivial. If I have the time in the future, I might do it, but it'd be a long-shot and take the better part of a day to get the test data set up into a simple test app to send to the H2 developers.

Anyway, I got H2 1.2.132 and tried it - no difference. Crud.

So then I tried setting the auto-commit on the JDBC Connection objects to "ON". Initially, my code had it set to "OFF" with an explicit commit() called on the connection once the SQL statement is executed. But even with the auto-commit "ON", we had the same problems.

Next I tried breaking up the SQL itself into smaller blocks based on the number of volatility points per expiration. If there are more than five volatility points in an expiration, then I break the SQL into separate statements in the execution part of the code. Since these are somewhat stand-alone, it shouldn't cause any problems. But I had my doubts.

I was really surprised to see that this break-up based on the number of strikes for an expiration solved the problem completely. What a break!

What I don't understand is why.

I can see that 4 of the 6 columns are in the primary key - but that in and of itself shouldn't be the issue. Not for a recursion problem. I can see that I'm inserting several hundred of these at a time, but the problem appears somewhere below 50, so that doesn't make a lot of sense.

Nope, this is a real puzzle to me. If I get the time, I'll try to write a simple test app and ship it to the H2 developers to see if they have any idea as to why it crashed. Very odd indeed.

Maybe It’s All the Same – Everywhere… Sure Hope Not, Though

Thursday, March 18th, 2010

Management.jpg

I was chatting with a friend today about my current disillusionment with the management decisions that I seem to be more aware of in these past few months. To be honest, it's the clumsy, inept management that makes Dilbert a comic that everyone believes is written with their workplace in mind. And to a large extent this is the case - people are the same no matter what the workplace. There are executives, bosses, managers, and workers, and no matter what the company does, it's the interactions of these people that are remarkably similar.

So goes the theory.

But at what point does this similarity stop being a commonality, and become a difference? Specifically, if you're having a difficult time with the current management, what makes you think it'll be significantly different at a different workplace? After all, that's one of the big reasons for changing jobs, isn't it? To get to a better environment.

So I was thinking... How much of what I'm feeling about this management is just universal management dissatisfaction, and how much is truly unique to this place? Based on this, is the difference really all that bad, or is it just frustration I'm feeling in a lot of little ways, and rather than push back, my first instinct is to walk away because I don't feel it's my place in a large organization to change it. It has evolved to this point, and who am I to say it should adjust/adapt to me when the reverse is probably what every one else has done?

It's a lot of good questions, and there's no way I'll be able to find out all the answers because it's all very subjective, and you can't know what every place is like until you work there. For instance, getting here, I believed that this place would be significantly better than my last position, and in a few ways, it is. In some others, it's decidedly worse. Other areas, it's just very different. So is the move to this workplace a move up, down, or just a move?

I'm coming to the conclusion that it's just a move. But it can be a very important move. If the differences are over one or two issues, then that's not going to really change, but if there are serious disagreements over how work should be done, then maybe this you're in a workplace that's not as good a fit as another.

At my old company we came up with a saying: Some times, good people just don't fit. And it's not a statement that there are a lot of good people in the workforce, and there's a lot of good workplaces. But not every good person fits well with a good place. There's a lot more to a good fit than the goodness of the person and the place.

Are the goals compatible?

One of the things I've found most difficult to match up properly are the real goals of the place and the person. Someone may be looking to create some of the greatest works of their career. Some places may be looking to be the very best in their field. Others may be very content to work at the level they have achieved and leave the superlatives to the others. There's nothing right or wrong about one set of goals - they just are. The problem comes in when the goals are different and the friction builds.

Someone looking to create great things can be very frustrated when faced with an environment that's only interested in the next incremental change. They accept that the feeds are bad, or the data is iffy, because it's "OK", and to the person looking to do great work, "OK", is just not "OK". They are looking for great. The individual can think that the workplace is just not interested in doing the level of work they are interested in doing, and to a great extent, that's exactly it - but not because they don't want to - it could be as simple as the fact that they are very comfortable with the flow of work they have evolved to, and aren't really interested in changing. If the great work can be done while retaining the work flow they have come to depend on, then fantastic! But if not, then they'd rather keep going like they have rather than venture into territory they might not like.

Is the pace of work compatible?

This is one of the easy ones, really. Virtually every place expects folks to get in around the same time and work a certain number of hours. Additionally, they may expect a certain level of effort from a group which translates to hours based on the development cycle. In any case, there are expectations of the amount and level of effort from the people at the workplace.

It's not always fair - the executives may work less hours than the workers, or maybe more, but the expectations won't likely be spelled out in the interview process. It's not the best way to attract top talent by telling them something they may already know, or will learn within the first few weeks anyway.

And there's the problem for me - if one person is ready to put in significantly more effort than the others around him, what's likely to happen is that managers will become reliant on this person and will allow others to slack off a bit knowing that this 'engine' will pull the majority of the work anyway. Then, when they take a vacation, or need some time off it's nearly a revolt because the others got lazy and don't know what to do.

The reverse is typically weeded out by the workplace for someone hot showing up for the 'standard' hours and level of effort. Still... it's not completely fair to the person to withhold this information from them in the interview process. Yet, at the same time, how many people want to go to a place where they hear "Everyone works 55 to 60 hour weeks, and if you don't you'll be let go at your first review." Answer: not a lot.

Are the demands of the work acceptable?

Given that the duration and the pace of work are compatible, what about the other demands of the place? Are there going to be calls after-hours? Are there games, events, parties that have an implied expectation of attendance? Again, these might not be clear in the beginning, but after a few months, there may be a significant level of this type of 'work' associated with the position.

While many may say this type of commitment cannot be required of an employee, everyone in that position knows that there's a million different ways to mask the truth of the matter. Re-assignments, difficult goals, there are always ways to make it appear that there's no bias towards the person, but the talk at the water cooler is that this person didn't get this promotion or opportunity because they didn't go to the Company picnic and let the Boss win the sack race.

Everyone has a different tolerance level for politics, and while it shouldn't matter in an intelligent, mature, professional workplace, we all know that's just idealistic.

Do the rewards of the work feed your soul?

There's more to a job than the paycheck, and while that's got to be acceptable to the person and the place, there is a lot more to this than just dollars and cents. Look at all the volunteerism around you - that's got nothing to do with a paycheck, it's all about the satisfaction they receive from doing the job they are doing. It can externally motivated like power or acclaim, or it can be completely internally motivated - simply knowing they were capable of accomplishing the feats they set before themselves.

Tapping into this is hard because it presupposes that everyone knows what internally motivates them - assuming something does, which is not universally true. But for those few that have this knowledge, there can be a lot lacking in the other parts of the fit as this will allow them to ignore some other problems and still feel very good about themselves.

Looking for a Better Fit

I've done a lot of thinking about this. I'm coming to the opinion that I'm not a great fit for The Shop. Lots of little things, but all pointing to the same thing: a significant difference from the 'average' developer here. It's not bad or good - it's just different. In the long run, some times good people don't fit.

I think that's me.

MarsEdit 2.4.1 is Out

Thursday, March 18th, 2010

MarsEditIcon128.jpg

I got a tweet this morning that MarsEdit 2.4.1 was out with an interesting collection of features and fixes:

  • Support editing slug on Tumblr posts
  • Support editing the date of Squarespace posts
  • Fix issue that caused Autosaved posts to get "stuck" and recover on every launch
  • Avoid showing spell checking marks inside HTML tags
  • Improved Accessibility for VoiceOver and other assistive interfaces
  • Fix accessibility attributes of main post editor fields
  • Fix accessibility of weblog list utility buttons
  • Expose per-weblog busy indicators to accessibility
  • Quiet noisy console warnings on Snow Leopard 10.6

and while I haven't run into any of the stated problems, it's something I clearly use every single day, and I love what it's been able to allow me to do.

In all fairness, my own journaling tool wasn't bad... it was just far too early and non-standard. I was glad to move to WordPress at HostMonster and MarsEdit. It's been a fantastic release for me.

Professional Suicide – Working with Old Visual Basic Apps

Wednesday, March 17th, 2010

NoMoreVB.jpg

Well... today I heard something I never thought I'd hear again: "You'll probably have to get into the VB Form to look at how it's working." What!?

I was talking to the original author of Hemlock - asking him for a little help with the lovely concoction he'd created. He wasn't really interested in getting his hands dirty, but he had the advice that it worked - once. His single idea was to check for double-counting. Yup, already did that. After that, he was fresh out, and his suggestion was to get into the VB Form code for that feature in the legacy app that everything's compared to.

While it's a good suggestion, it's also professional suicide.

Can you imagine the conversation when looking for my next job?

"So... what's this Hemlock project all about?"

"Well, it was a risk tool that was developed by one desk and then I took it over and it grew to be a firm-wide tool."

"Hmmm... sounds interesting. What languages did you work in?"

"Java, C#, and mrmph-mrmph."

"Sorry, I didn't get that..."

"Visual Basic"

"Thanks for coming in."

I can't believe this place is this dependent on VB at this point in time. It's like they don't care that their most essential application in the entire Firm is written on something that I stopped using for anything critical more than a decade ago. While it was once the platform for anyone on Windows needing more power than Excel and not enough time to make a VC++ app, it's now just one of many languages that fill that sector.

In fact, there are a lot better languages these days - C#, Java, any of the .Net group. And you don't need to stop there - Java on Windows is a very good platform and can access almost everything you need.

This is not about getting the most out of a good, workable solution written several years ago - everyone in the organization says they want to retire the old system - it's too hard to maintain, too many problems in the core of the system. Heck, a while back, they wanted to replace MS MQ with 29West as the messaging system for the app, but it was impossible. The developers tried, but they could not get rid of MSMQ because something broke here, or there. So they left in MSMQ and had 29West feed MSMQ. Completely whacked out.

The idea of spending any time in this codebase is just something I'm not willing to accept. There aren't a lot of professional lines I won't cross, but this is one of them. I have no desire to have that kind of conversation with my next potential employer.