What's it all about, Alfie?

Once again I got the wake-up call and needed to deal with a problem on the server. During the night there had been a problem with deadlocks and it stopped the server. What a drag.

Thank goodness for the extra logging I had put into the server. This time it was the reloading of the missing marks after they had been received. The code evaluated the stock families for those with new prices from the loading of the missing marks. The problem was that in some conditions a tick coming in for the instrument that's getting revalued from the loading of the missing marks was locking from the stock down, and the revalue method was only locking the options. That's certainly a source of potential deadlock.

These deadlocks have been very hard to eliminate. I know that if I used the exclusive locks and not try to be cute with the read/write mutexes it'd be a lot easier. But I think it's important to the server that I try very hard to make these read/write mutexes work. I'm close, I can feel it.

I've got my old thesis work that I was looking at yesterday. It's primarily f77 code, with a few of the critical methods converted to C with f2c and hand-tuned by me to get the best performance possible out of the code. It's nothing exceptional, but it's something I'm proud of, and I like to go back to it every so often and test it on new hardware that I get. Well, yesterday I was thinking about testing it on my MacBook Pro because the last time I tested it was on my G5 iMac, and it was very fast there, and I expect it to be even faster on my MacBook. It's got a faster processor, and if the compiler is capable of using multiple cores, then this will be even faster. So I was looking forward to it.

In the past, I've gone to the HPC Mac OS X page and they have had tarballs of g77, g95, gfortran that are built with the Developer Tools so that they work well with Xcode, which I am a big believer in. So I go there and get the latest Intel binaries for gfortran. Problem is, they don't compile the code. There's a problem in newton.F that causes an internal compiler segfault, and while they say "send in a bug report", I looked at the site and I think it's probably safe to assume that someone else has done this. There are a ton of rules to follow to submit the bug. So I missed there.

I tried the g95 compiler on SourceForge as it is supposed to be a different fork on the code, and while it compiled a little test program, it was spewing some errors that didn't look good, and in the end, didn't compile as much of the codebase as gfortran did. Strike two.

While I suppose I could buy a FORTRAN compiler, I really don't want to. I like the GCC tools, and I know that right now the gfortran snapshot (4.3.0) in the HPC download is 'experimental', it'll get better, and when it does, I'll get the next version. Until then, I'll have to wait. Not the worst thing in the world. It would be nice if Apple put it in Xcode, though. Face it, they're using gcc, so how hard could it really be?

UPDATE: I chatted with the maintainer of the HPC Mac OS X site and he said that he'd be updating the gfortran code in a few weeks due to the updates, but that the g77 code should work and it can be installed alongside the gfortran package. So I got it and it worked! The code compiles in less than a minute, and the tests run faster than any other machine I've ever run the code on. Amazing! Great guy.

As the hardware gets better, the times drop. I may have to come up with different tests. The latest tests show the MacBook Pro beats the G5 iMac. Nice.

Today one of the developers wanted to add lines to the VantagePoint bar graphs that I had wrapped with nice little helper classes and applets for BKit. The problem is that a bar graph can't have lines, but a combo graph can. So I added the combo graph as a type to the BKBaseGraph and the BKBaseGraphApplet. The idea is that it's very much like the bar graph - even being subclassed off it, but the way it handles the lines versus bars is to have essentially two types of variables: regular variables that are the bars, and line variables that are the lines.

Once I figured that out, and then fixed up a few methods like getColumnHeadersOnGraph(), it wasn't that hard to get everything working. The subclassing was interesting in that I had to have tests for the combo graph before the bar graph as the instanceof would have been true for the bar graph class when I really wanted it to hold out for the combo graph. Not bad, but interesting.

The final thing was a new applet PARAM tag to indicate what lines to put on the graph. I chose asline:<col> ("as line") so if I had columns one, two, and three as bars, and four as a line, the relevant PARAM tags would be:

  <PARAM NAME="visible:one" VALUE="TRUE" />
  <PARAM NAME="visible:two" VALUE="TRUE" />
  <PARAM NAME="visible:three" VALUE="TRUE" />
  <PARAM NAME="asline:four" VALUE="TRUE" />

This tests out great and the line width, color, etc. all work on the line in the combo graph - as you'd expect. It's a nice little addition to the toolkit.

Today has been a bit of a slow day - the data is good, trades are flowing, things are really humming along - all of which makes for a slow day for me. Not a tragedy - I had time today to mess around a little with the CSS on the server editor. I had picked up the CSS for tabs from WebFX and the 'winclassic' version was as close to Windows as I could have hoped. The problem was, it used the Windows-defined colors for 3D objects, and while that's really nice on IE, it's a mess on Firefox as those aren't globally defined in Firefox like they are in IE. So I set out to clean up a few things and in the same way that the fonts were cleaned up a while back, by specifying the RGB components, I was able to get IE and Firefox to match exactly. Nice.

I'm sure that if I had the time, energy and interest, I could look at the CSS stuff and make something really spiffy. I've seen very nice scrolling tables in CSS, and lots of other stuff, so I know it's possible, it's just that this kind of fiddling is something I really hate to do. It reminds me of the time when coding Windows apps that you had to layout your GUI components in code - moving things over 5 pixels, saving, recompiling, testing... it got old very fast. Someone is going to make this processes with CSS faster, and when they do, it'll catch on a lot better than it has up to now.

Interestingly, Apple's stock took a hit from the Keynote. I know it wasn't earth-shaking stuff, but it's really interesting to see that perception is far more important than reality in the markets. People expected something wild... something very Steve, and what they got was a developer's conference keynote - talking about the upcoming OS features. I guess even The Steve needs to watch out for his own distortion field... if he doesn't have the batteries freshly charged he can get himself into trouble.

One of the developers using some objects in a Java library I wrote and maintain came to me to ask why this one operation was taking as long as it was. It's basically a table objects and views that can be placed on this table (and other views) to "stack up" a "deck" so that the end result is a table that has just what you want, in the order you want, etc. After talking with this guy, I realized that there wasn't a good reason that an aggregation on top of an aggregation was taking longer than the first aggregation. I mean the data set was less, and so it should, in theory, take less time. But it wasn't. And not by a little. So I decided to dig into it.

The first step was to build a test frame for this kind of environment. You see, it wasn't effecting small data sets like it was effecting the larger ones, so I built up a 100,000 x 100 table, and then aggregated it to 10,000 x 100 and then to 100 x 100. What I saw was that the first aggregation took about 13 sec. and the second one took about 8 minutes. OK, this was a good test case, and so I went into profiling mode to find where the time was really getting spent.

The first thing I thought was that the rows and columns were being inefficiently accessed by linear searches of their labels. But after putting that code in (it was in the base table, which is why I thought the first one was faster), it turned out that it didn't really improve the speed a lot. It went down to 5.5 sec and 3+ min. Better, but not nearly good enough.

Then I looked at some of the individual operations and what was there blew me away. In one method, I was returning the column headers as a Vector of Strings. The implementation for the base table was to return the ivar that was the Vector of Strings, and to have in the comments on the method the warning that this is the reference to the storage of the column headers, so mess with it at your own peril. In the aggregate view I had the code making a copy of the Vector and returning that. This was a hold-over from several of the views where the underlying table's columns can change and the system needs to augment the column headers on the fly.

That was the killer.

By creating a new Vector of Strings each time the row was accessed it took so much longer than simply returning the ivar that when I changed it to use an ivar the times went to 5.5 sec and 0.5 sec - they're going in the right direction now! I was amazed at this, but then I started to think about it. Java's allocator is probably doing a lot more than a typical C/C++ copy constructor, and as such it's load on the system is more. Even so, it was not the best idea to have a construction in the tight loop of the aggregator. All is fixed and I'm looking at the last two views that might need changing, but I'm not sure that even they do as they aren't doing the same kind of work that the aggregator was doing. But I'll give them a look and see if I can speed them up as well.

Today I saw a chunk of code that was checked in and had a very hard time understanding exactly what it was doing. The comments didn't really help, and the variable names weren't a lot of help either. I actually had to walk through the code a line at a time to understand what it was doing. Now I'm no paragon of design and coding virtue, but there's a point where you really need to hold yourself - and the folks you work with, to some bare minimum standards in this area. A 20-line method in Java should not take someone to walk through the code to understand what it's doing. I read Kernighan's quote: Debugging is twice as hard as writing the program, so if you write the program as cleverly as you can, by definition, you won't be clever enough to debug it. and agree with it 100%. The extension of that might be: If you have trouble understanding what you just wrote, the other guy, six months from now, will have to re-write it.

So I figured out the code and started cleaning it up. It didn't need the functionality changed, that was fine. It was the variable names and the flow, and more than anything, the lack of comments saying what was going on and why. I know these are things that aren't popular with a lot of developers, but they make the job of maintenance and extension a lot easier. It also won't hurt to spend just 15 or 20 minutes looking at what you need and trying to see if there's a better way than your first cut. Don't spend the entire day, but spend a little bit of time to make sure that you're putting down something that the next guy will be able to pick up easily.

I'm as excitable as the next person. I get whipped up about a lot of things. Development languages are not among them, and I have to wonder at professionals that call themselves developers that do. I was talking to someone the other day and they wanted to add a Java 5-ism to the existing library that was 1.4.2. I said that there are lots of projects that use this, and many are 1.4.2 still and may not change for a long time - if at all. If they work, there's no reason to update them. They might be updated at some point in the future, but this one feature that was being discussed was certainly not a business-justifiable reason.

But that wasn't the end of it. I had to ask this person "Why?" The answer was exactly what I expected: they wanted to type in three lines instead of the existing way of using indexes which might take six lines. So we're talking about saving three lines each time we run through a certain object's elements, and this is the reason for updating working projects? I think not, and I'm be stunned if the users thought so, either. I have met many Java Evangelists, and it's not just that they know, and promote, the newest features of the language, they condemn those that might say they follow the faith, but don't push as hard as they personally do. For instance, if you're developing in Java, you almost have to consistently push to the latest version or you're "outdated", and therefore "don't get it". Still using Enumerators? You just don't get it. Like we're all still coding in COBOL, for instance. I mean it's silly.

If you ask someone coding in C++ they aren't going to ask you the version of the compiler you're using. They may ask you about the features you're using, but not the compiler tricks. Yet that's just what these Java zealots are doing. If you're not on the latest and greatest - and using the latest and greatest, then you might as well be coding in Visual Basic.

What's funny is that many of these people probably haven't spent a lot of time with different languages. They might have learned Java in school - if they took a class in programming there, and they think that's all there is. But I know differently. I watched the industry move from big iron to PCs. And then with PCs to networked apps. Then to Windows. Then to Web. I know Java is not the final language - it's the current language for a lot of people, but it's not the end-all-be-all, any more than COBOL or FORTRAN were. Don't get me wrong, it's a nice language, and it's got some nice features, but not everything is a nail, so you need more than a single hammer.

Yet you'll never be able to tell these people that they are missing the point. That a language is just that - a language. It's a tool for expressing what you want a machine to do for you. Maybe the most expressive way is with Java - maybe not. Maybe the difference in the expressiveness of Java 5 to 1.4.2 is big enough to warrant a move. Maybe not. The fact is there are no universal truths in development. There's a lot of professional development still being done in FORTRAN. That doesn't make it any less useful or usable. It's a set of tools. But I got tired of tilting at windmills and walked away.

I've spent more time than I'd have thought possible getting a page in IE to look like the same page in Firefox. I'm sure this is no surprise to anyone doing web design - HTML is not really going to give you what you want, you've got to use CSS and then you get into the same realm that you were in back in the days of nroff/troff and LaTeX. What's wrong with these people?

When I did my thesis, it was not on a Mac - it was on the VAXen at Purdue. Additionally, there were standards you had to meet with the style and layout of your thesis - all the way down to section naming conventions (7.2.1, etc). I understand why Purdue wanted to do this - so all theses from there had the same look to them. Reasonable. To do this, there were massive macros and included files for troff and then you could run it through the Versatec for "laser-like" printing long before there were laser printers. You then had to cut and tape/paste your illustrations on separate pages so that it looked like it had all been done 'professionally'. This involved knowing the font sizes of everything, equations were a pain, figures needed to be planned out at the end of each chapter, it was a mess. But it was all we had.

Today, you'd use Word, or OpenOffice, or something, and include the images from some drawing program, and then ship it to a simple color laser and you're done.

So why, oh why, are we back to the days of specifying the intricate details of the rendering of text? The web, and specifically the web browsers, need to take a page from the word processing advances of the last 25 years and get with it. There needs to be a way to make all of this a lot simpler. Oh sure, if it's in the browser, then there are those that say Microsoft will make the rendering in IE of Word-built documents better, nicer, faster, etc. But that misses the point of the web these days. Today, it's not about static pages, it's about dynamic pages, and for that, you're not going to be able to use Word. You're still going to need Perl, PHP, Tomcat, WebObjects, etc. to pull in the data, and get it ready to show to the user. What I'm complaining about is the presentation markup that we have to use.

There are those that say it's all a matter of standards - get the right set of CSS files and everything is easily done from that. OK, there's something to that, but that's no different, really, than my grad school days. I was lucky to have the benefit of many other people making these troff templates, but I still had to do more typography than should have been necessary - or than is necessary today.

So we need to move the web forward with the lessons learned from the Word processor wars... get a format that does more than text layout - get page layout and make it a lot easier to use.

I've got an interesting problem facing me now. I now have multiple servers - on different continents, and it would be nice to be able to have them act as one unit. This isn't a requirement of the project, the requirement was simply that we have a BCP (a.k.a. disaster/recovery) site for the server, and that's done. But there's a step that we can take, if we want to, and that's to make the system better because of the existence of the other server - improve response times, share the load, etc. The problem is in how?

The first obvious solution is to do nothing. Not very creative, but it's the lowest-cost solution and has to be considered given that we have limited people and time available to us.

The next might be some kind of master/slave set-up. Something where all the data still comes into the main server, and then after it's done verifying it, processing it, something is sent to the other one to keep it up to date. This has the benefit of being the only solution where we know that the second (and third) server's state is - because we set it. No inputs other than price are allowed to this read-only, slave server. This keeps things under control nicely. However, the downsides are significant:

• Plenty of development needed - we're going to have to come up with multiple protocols that will be used to transfer the data from one to another with handshaking and acknowledgments so that we know what we sent was received successfully, etc.
• Significantly more bandwidth needed - if we're going to be moving the complete state (as it evolves) from one machine to the other, we're going to need a lot more bandwidth between sites to keep the latency on updates low. This might be a big issue and then again, it might not, but if we do this, we certainly need to keep this in mind.
• Still going to need verification - no matter what, we'll have to have something that will verify that the slave is indeed a copy of the master. This might take the form of daily script, or it might be an hourly check, but something or someone has to check them against one another.

The other solution I can think of is some kind of peer-to-peer system where a change entering one gets sent to the other prior to it actually being checked and acted upon. The most obvious extension of this is to have all the changes come into the one, Chicago, server and then have it send the data to the other(s) so that we're pretty sure that we don't have to worry about cross-site changes, which could be a drag. Still, the problems with this solution are significant too:

• Loosely coupled == More differences - no two ways about it, if the systems are loosely coupled, then it's very possible that the data in them will be different. This might also happen in the master/slave, but it's going to happen a lot more in the peer-to-peer.
• Support Costs - because of the first point, this solution almost guarantees that we'll need someone in the right timezone to make sure the differences are kept small. It's a person at this point - not a script.

I've sent out these ideas and trade-offs to a few of the folks here that would be most effected by the decision. I want to hear what they have to say. My initial impression is that I should have the second server running every day, taking ticks, ready to cut over at a moment's notice, but that linking these guys is a lot of work that isn't really needed, and won't necessarily pay off in the long-run.

Tonight I have releases of several components of a large app that I deliver, and since multiple pieces are changing, it's a good idea to make today a slow day so that I don't move the codebase in case something comes up regarding the releases.

I spent time helping track down a few issues... configuring a few boxes... doing my part to increase the Corporate Red Tape... all things that needed to be done, but have been put off as they aren't that important. But today was their day.