What's it all about, Alfie?

Today has been a lot of clean-up and planning for the future today. I am at a brief pause in the development cycle, waiting for the Q/A Team to certify an application of mine so that I can then put it in production and reap the benefits of the fixes and features. While I was waiting for this, and helping them out when needed, I wanted to run through all my emails and make sure that I didn't have anything outstanding that needed to be done.

Turns out that I did - imagine that.

I needed to pull two pages into one by adding a single checkbox and a simple if-then construct to pick the right URL for the request. This would allow us to choose whether to hit the in-memory database of my app, or hit the back-end database. There are a few new webapps on the server, and this addition was really needed in order to be able to maintain the app under adverse conditions. Just makes sense to consolidate these into one.

There were a few other things I needed to do as well... nothing major, just cleaning up emails, sending out status emails, checking up on folks - organizational maintenance stuff. Got all that cleared out.

Then I spent a little bit of time planning for the next new feature for the app in Q/A. I needed to make sure I could see where to update for the new information, how to make sure it'd work regardless of the conditions. In short, it's not too hard. I'll be able to add in the data to the base elements, then the existing code will automatically aggregate across it, but the new code will be able to filter on it and come up with the exact data we need.

Pretty nice.

I did need to do a bit of explaining to the original author as he came up with a pretty decent idea that will make it easier to be sure that the old test cases and code will work with the new data. It was a little strained after I agreed with his idea and he still wanted to understand what it was that I proposed. We probably went around this a few times before he got it, and that's OK - just a little miscommunication.

In the end, it was a very productive day and lots of stuff got done. It just wasn't Wow! work, but necessary nonetheless.

Well... today I decided it wasn't worth it to keep working in Git when it's not really supported in the Shop. The approved version control is Subversion, and that's not really horrible, but it's not as nice as Git or CVS. The requirement that you have to change directories to change to a branch is silly. I understand why they did it, but it was a bit of a cop-out. At the same time, subversion holds the 'ignore' files in metadata, they could have held the branch/tag info.

Anyway... without support I decided to move. The problem is, there are no real tools to move from Git to Subversion. What I ended up doing was to create a new project in Subversion and then import the latest copy of the files. I lost the entire history to date on the two projects. It was a little disappointing, but in the end, we have something that I can easily share with other developers.

I feel a bit sad for not being able to use Git, but in the end, the realities of the work environment dictated that it had to be done.

Today a co-worker wanted to leverage the back-end Google Visualization system I have created for my web app. Given that I've got all the web services with database connections and the Google Visualization data generation, it made sense to look at the Google Visualization Gallery and see if they had something that would fit her needs. If they did, it would be relatively easy to make a page that issued an AJAX call to my server with the request for the database to be returned and placed into a simple x-y graph tool.

The first thing I needed to do was to subclass the GoogleVisualization class and create one that uses the back-end database connection as opposed to the in-memory database connection. This would allow her to execute queries against the back-end database, which was what she wanted. From there, all she had to do was to glue it together with a little HTML.

It took a while, but she's getting the hang of it.

Today was one of those days you can't believe. I was coding to some wonderful music on my iPod and things were just falling into place. I'd write a method, it'd be say a dozen lines of code, and then I'd look back at it and think Wow! That's impressive. It covers all the bases, and is elegant. I did that again, and again, and again.

I had about 3 hours of this today. It was spectacular. Really. I can't remember the last day I had when I was this hot on the keyboard. It's been a while, I can tell you that.

So of course, it had to end badly. Doesn't it always?

It started when I had the basics of this F/X correction into my web app. It was absolutely wonderful. It slipped in cleanly and easily, and it worked like a charm. Sure, it had methods that needed to be database-driven as opposed to the hard-coding, but that was to save time and make sure I had it all under control.

I had a method that would give me the conversion factor for a 'from' and a 'to' currency code, for example. This needs to be driven of the database of F/X rates, but I had the basic conversions coded up as fixed numbers. The method worked, and once I got the database work in place, it would be driven off better data, but the essentials were there, and everything could be seen to be working.

This was all done about a week early, by the way.

So I mentioned to my manager that I had the rough-cut of the F/X conversions in. I tried to explain the things I had yet to do, but he was already off to the races: "Did you get this in yet? How about this? Can I see this?" Like an old boss I'd had. Excited, which is nice, but already assuming that a week early is "normal" for me.

In a very real sense, I condition people to expect greatness from me and then I'm stuck with their expectations. It's only taken three months for this manager. Yet I can't blame him. For these three months I've been working exceptionally hard to get things going, and I've got an impressive string of successes already. That I got this done in a day is not really all that surprising. But it'd be nice if he at least gave it 5 mins before asking if I'd done the next three things on my list.

Just give me a little rest.

So I ended up on a real downer when it should have been fantastic. Bummer.

Today was an interesting day, that's for sure. I've been trying to tie up several loose ends with this project I'm working on with another developer in the Shop, and I wanted today to be the day that we finally finished it off and sent it to QA. Seemed very reasonable when I got into work today.

My tests went perfectly, and that was about all that did. I had a 'drive by' from two developers saying that the code they'd written in the project wasn't returning the correct numbers, and the way in which I was getting it would have to change. Not horrible. In fact, reasonable, and so we spent the hour there getting that fixed up.

Then there were logging problems with the addition of the log4j jar and no configuration file. That set me back a bit as I waited for the other developer to do it. I ended up doing it to try and get this all put to bed today.

During all this, I had added new data sources to another app I'm working on and those weren't configured exactly right. The net result was that I was missing a ton of P/L in the calculations making it look like there was a huge loss. Someone was freaking out and I was in the middle of trying to check in some things for the other project while he's telling me of the problem.

"I'm on it... just give me one minute to check this in." I said.

No more than 10 sec. later he says "Could it be in the configuration?"

"Yes, I'll check it in less than a minute, I promise." as I frantically try to write the subversion check-in comment for the code sample I just finished for the project.

No more than 10 sec later he says "If this is real, they need to know about it."

"I promise, just another few seconds and I'll be on it."

Then he stands up and tries to talk to be and I look at him, put my fingers in my ears and sing "La La La La La" like the kid that doesn't want to hear what the parent has to say. Childish? Sure. But I didn't want to insult him by saying "Do you have a watch? Count off 60 sec. and be quiet!".

In less than 20 sec. I was done, another 60 sec. and I had the problem figured out, and within 5 minutes it was all solved. He then wanted to discuss why this happened, I explained that it was because the way the app was written by this other developer and the instrument lists were set by the XML config files and that's very manually intensive. He wanted to know how soon we could make it database-driven, and I said we could, but it'd drive out other things that he wanted done.

No easy choices, but the best advice is not to make decisions when you're upset about something. In this case, seeing bad P/L from a new data source. It wasn't even in production - just test. Learn that not all problems have the same impact.

Thankfully, after a few minutes of talking he calmed down and realized that the schedule we put together of work was probably the best, and we would get through this just fine. It's not perfect, but I didn't write the thing, and we're just doing the best we can to muddle through until we can give it a complete overhaul.

But I'm glad it's over.

I have been messing with Tomcat 6 and log4J logging today because this web app I've been working with had to add log4j for an external dependency that was added today. In general, I'm a fan of log4j if you go through the effort of using it properly, it's not bad. But adding it turned out to be a pain that I couldn't have imagined. In the end, I should have gone directly to the Tomcat 6 website and looked at the docs there and not questioned it. But I had done this before, and I was convinced that I could whip this up without any grief. (Ha)

The simplest thing was to create a log4j.properties file like:

  log4j.rootLogger=INFO,A1
      log4j.appender.A1=org.apache.log4j.ConsoleAppender
      log4j.appender.A1.layout=org.apache.log4j.PatternLayout
      log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c - %m%n

and place it in the WEB-INF/lib directory of the web app. I also placed the log4j.jar (version 1.2.15) into the same directory, deployed it and it worked. Almost.

Actually, it worked about half the time. The other half the time I got a warning from log4j saying it couldn't find the properties file. Very odd. If I edited the file, it was there and OK, but sometimes it'd work and sometimes not.

I looked into this from a lot of angles, and there was nothing I could do to get it to reliably recognize the log4j.properties file in the WEB-INF/lib directory of the web app. Then I read this from the Tomcat 6 website.

Seems they are saying that you have to place the log4j.properties file in the $CATALINA_HOME/lib directory. No choice. That means it's the same logging for all web apps on that instance. In general, I can see why that's required, but then I'm a little blown away that it wasn't working where it was. But it was... and it wasn't.

So in the end, I put the files (jar and properties file) into the $CATALINA_HOME/lib directory and restarted things and it's working fine. Kind of a pain to tease me with the possible working within the web app, and that would be exceptionally cool, but it's just not going to happen in this release of Tomcat 6.

Today was an interesting day. I was beginning to work on a new feature for this project this morning and then got pulled away from that to look at the seeming instability of one of the greeks for one of the portfolios. It seemed quite erratic, and clearly not driven by the market data or trading activity. It was really odd, to be honest, and it looked to me to be something in the calculations as opposed to the inputs.

So I started digging.

I had a heck of a time reproducing the input data simply because the SQL-styled input to a data source seemed to work when I had strings wrapped in single-quotes, but two columns of returned data were consistently zero. I dug and dug, trying to figure this out, but in the end, it was the fact that I tried double-quotes around the strings that solved it. With double-quotes around the strings, all the columns came back properly. Very odd, to say the least.

But then I had the data, and noticed that the normalization factors were very large - so large that they might really cause problems with the summing that was being done. Also, I checked on the positions and noticed that there were offsetting positions and that, along with the large normalization factors could quite possibly lead to loss of significance due to subtraction problems.

This took me several hours to come to.

But once I had a plan, I started re-working the calculations so that I don't get into these situations. I've done a lot of this in grad school, so it's not totally foreign to me, but the problems associated with what I had to work with made this difficult all the same. I needed to first calculate the net positions and then deal with a single position value per instrument. That's got to be a big winner. After that, I did the positive sums first, and then subtracted off the negative values. This should make these sums as well conditioned as they can possibly be.

I'll have to wait for tomorrow and see what the effects are on the stability of the numbers. The driving data is based out of London, and it's not ticking now, so I need to wait for the morning and then compare it to the production numbers. I should be a lot more stable, but if I'm not, then I'll keep going and get some stability. I have to, it's just too unstable and unreliable as it stands now.

Heck of a day.

[6/3] UPDATE: the stability was amazing. Really. Stunning. It was all I could hope for and then some. As the production app's numbers bounced around, the new development numbers were very stable. They moved, but they moved deliberately. That is a major improvement. Fantastic.

I had an interesting conversation with another developer today. He stopped by to tell me a problem he'd run into with the part of the code he's working on. He knew what he wanted to do, but he wasn't going to do it. He felt that it was better to give the users something as soon as possible, even though it was wrong, than asking them to hold off a while and build the right solution to the problem. It reminded me of the term I'd heard from another co-worker: Software Debt.

I found this article that does a reasonably good job of describing it. It's something that you have to be careful of every single day you write code. It's easy to hack something together and then say "I'll fix it later". The hard part is actually doing it.

This developer I was talking to was convinced that the solution to the problems created by these hacks added for immediate release was simply Eclipse. Yup, since Eclipse can refactor code, it's easy for him to go back and change everything about the code when he goes back into it later.

Now I'm not saying Eclipse isn't a great tool, but it's a refactoring tool, not a re-designing tool. And going back into the code presumes that you're back in the code. What happens if you don't need to go back in because the hack is only used once every six months, and at the time you notice that it's time to fix it, it's too late, and you're once again in the bind that something is expected by the users.

Rather than argue the point, I said "OK", and he's off to do the coding as he sees fit. I'm not his supervisor, he doesn't report to me, so it's a collaboration. In that vein, until it's my problem, it's his problem, and if that's the way he wants to handle his problems, then that's OK with me. In the end, we both know that it's wrong, and the right thing to do is to build something that's close to right (as close as possible), but still delivering close to the same timeframe as the hack. Get close, then come back and work it into the better solution.

But it's nieve in my book to think that it's just a matter of a little refactoring to change one system to another. Maybe it is that simple. More often than not, it's going to be some re-design as well as refactoring. It's not going to be something that's done unless there's a pressing concern to do it. While it's certainly possible that he's going to fix up this hack in the coming weeks, it's far more likely that much of it will stay and then some day we're going to be under so much software debt that we're going to be forced to throw it all away.

It's a shame. He agrees it's not the right solution, but is doing it anyway. Too bad.

While I've been dealing with a lot of jUnit testing issues this past week, for the last two days I've been dealing with the realities of trying to match the actual data from the system to the application I've got and that had turned out to be a lot more difficult than I had hoped.

The problem is classic: they have no comparison tools for the final stage in testing. They rely on eyes to look at one and the other over time to see if things "look right". This, as expected, can be very hit-n-miss, and I was getting a lot more of the latter than the former.

It's pretty easy to see if you're not even close, but if you're off by 10%, is that a mistake, or a sampling interval issue? Hard to say. If there's a lot of volatility in the sampled data, then maybe it's as close as you can expect to get. So I find myself wishing for fewer jUnit tests and better test frames for the end result. The fist are nice, but the second are critical.

In the end, the only way I'm going to get better test frame data is to be a part of it's creation. Unfortunately, I may have to do that.

The folks on the Google Chrome team have a nice download site for the shapshot builds of Chrome for Mac OS X and linux. I can imagine that the linux port will be a tough GUI as the platform isn't really about a consistent user interface, but the Mac OS X port is coming along nicely.

I'm sure it's not even at the alpha stage in the group's mind, but it's up and serving pages. Sure, that's about 10 mins. with Xcode, but the fact is they have the similar GUI to the windows version and it's working. They are making progress and that's nice to see.

In the end, I'm not sure whether Chrome or Safari is going to be the best browser on the Mac. It's nice to have a choice like that, however.

UPDATE: Ah... it's Chromium that has the builds - not Google Chrome. The former is the basis of the latter, but the two are different. Still... you need to have the one to have the other, so progress is still a good thing.

[6/5] UPDATE: Well, that didn't take long. It seems that there is a Google Chrome for Mac OS X out now. Very early release, but it's there and they are working on it. So they only lagged the Chromium release by a few days. Nice to see they are working on it.