What's it all about, Alfie?

I have listened to a lot of people say that multi-threaded programming is dangerous, and can be avoided with careful planning. I can see their argument: a properly created finite-state machine with sufficiently small processing chunks can appear to be processing many things at once on a single thread, but that's an illusion. A bad one at that.

First, the effort to write good multi-threaded code is simpler than making the equivalent finite-state machine code. I know, I've done both. You can do the finite-state machine, and it's great on old hardware, or single-core machines, but it's not necessary. Most machines now are multi-core, and getting more and more with each generation. No. You need to know how to write multi-threaded code in today's high performance systems. No two ways about it.

Second, there are times when the overhead of 'polling' in the finite-state machine construct overwhelms the processing that really needs to get done. There's a reason we went to interrupt-driven I/O over polled I/O - one is much more efficient than the other - but harder to code. Suck it up, Boys! Code like you're worth the money they are paying you! Write the code that should be written - not the easy stuff - the hard stuff.

I've been working with a package that has a central object repository, and caches in each API instance. This is smart in that it allows for the client to grab something, and then hold on to it for many requests so that there's not a continual re-fetching if the data doesn't change. Nice. Problem is - they made the API cache single-threaded. This means that in order for a client to get any updates, it has to ask for them.

Normally, this isn't a horrible problem, but if the asking requires a re-fetching (as opposed to getting just the updates), then the more things there are in the cache, the worse it is on performance. Remember - this is single-threaded, so they aren't going to let you do anything until the re-fetching is done.

What they should have done is to make the cache update in a background thread with a simple "complete replacement" when the update arrives. Keep the bulk of the API single-threaded, but as updates flow into the API's cache, allow them to update a hidden copy of the object, and then lock the API thread, swap out the objects, and unlock the thread. This way, the cache stays consistent with the data in the central object repository.

Why is this important?

Because let's say I get an object and modify it. I've got a new copy, and I save it to the central object repository. Now another user does a similar action. They get the copy, modify it, and now my copy is stale. This is bad, because if I don't ask for updates, I may try to edit it more, and save those changes. Well... that's not going to do because my 'before' copy is not what the central object repository has. This means it's going to throw an exception.

But what about my edits? They have to go. No choice. I have to re-fetch, and then start over again.

Without even the simple flag like "re-fetch me", which could easily be added to the API under the covers, I have no idea that my work will be in vain. But it will.

This is the case for multi-threading that I'm making. There are times that you need to simplify the problem into multiple threads and have one dedicated to the simple task of making sure the cache is consistent. It's simple, surgical, and not hard to do. The results would be dramatic in the simplification of the client code.

So learn how to write multi-threaded code. Many times, it's the simplest way to get what you need done. And in the era of 60-core boxes, you're going to need it to get the most out of any box.

This morning I noticed that Coda 1.6.2 is out, and the release notes indicate that there are a few FTP bugs fixed as well as some cosmetic highlighting and a maintenance issues. Not really earth-shattering, but I'm guessing that with the FTP process being so critical to Coda's functionality they wanted to get this out as soon as possible.

Once again, it's great to have products that are created by indie Mac developers. They see the problem, fix it, and post a release. It's that closeness with the customers that I like.

This morning I was reading the JavaDocs for VantagePoint's TwoDimCrossScatterViewer to see if I was missing anything on the secondary Y axis issue I was having yesterday, and I saw it. In working with the new graph type we have two ways of looking at the data - as pairs and as columns. In the first cut I assumed they wanted the pair number. There, in the docs, was the key I'd been missing. In the docs for the TwoDimCrossScatterViewer was the method:

    public void setSecondary(int colNum, boolean isOnSecondary)

Clear as day... they are saying that in order to put a column on a secondary axis, you have to specify the column. Not the pair. You can put either the X or Y on the primary or secondary axes - making this far more configurable than I had planned. This was very clever of Gordon. Love it.

The result is beauty:

I fired off an email to Gordon to tell him I "got it", and to ask him which question of mine he wanted to answer next. There were three:

• How to format Dates on Time Series Graphs - this is so we can make better line graphs for the developers doing a lot of time-series data.
• How to put Labels on the bars in Bar Graphs - this is in order to make the data identified in the graph and not by the legend.
• How to put Bars in a TimeSeries Graph - another developer wanted to have bars and not lines on the time series - another graph I'd have to add support for.

One of the types of graphs that we have wanted to make available in BKit with VantagePoint is the true x-y line graph. Currently, the line graph assumes that all x-coordinates are shared across all data sets and that they are uniformly increasing. This means that there's no way to really properly display a step function... or anything that might "double back" on itself like a hysteresis loop.

A while back, I asked Gordon at VantagePoint if there weren't some way to do this, and if not, could it be added? He responded that yes, indeed, there was - the TowDimCrossScatterViewer. Basically, it's meant to be able to plot multiple scatter graphs where each set of points is a data set. But if you go in afterwards to the data points and tell them to connect the dots, you can actually get a graph. Like this: (sort of)

The problem here is that I'm trying to display the square wave on the secondary axis, which was Y in the previous graphs, but it now seems to be X, and that's not really what we're going for. So it's close, but I still need to ask Gordon what I need to do to get the secondary axis to be the secondary Y axis. We'll see what he says, but it's getting really close. Exciting stuff.

It's exciting to see that there is clearly the new ability to create non-functional representations. This has been something that I really wished we had had a long time ago. Well... better late than never.

GraphicConverter 6.3 is out, and it's one of the first graphics apps I got for my first G3 MacBook. It's been a valuable asset to me over the years, and while I haven't used it much since getting Acorn, I still want to support the authors, and while this was a free update, I'll get the paid updates as I think this is really good software and is deserving of my support.

For quite a while, we've been trying to make a true x-y line graph with VantagePoint. The problem has always been that the traditional line graphs assume that the points on the x-axis are simple markers and equidistantly spaced on the axis. Not good if you have scientific data.

I heard back from Gordon at VantagePoint and they do having something that I can use to create a true x-y line graph - the TwoDimCrossScatterViewer. It turns out that this graph takes pairs of columns (x,y) as the Variables and then plots them independently on the graph. Not bad, but I need to connect the points as well. Gordon has that covered as well:

  /**
   * This is the way to draw a line between points on a cross scatter graph.
   */
  for (int var = 0; var < itsGraph.getNumCrossVars(); ++var) {
    VariableAttributes   va = itsGraph.getVariableAttributes(var);
    va.setOption(VariableAttributes.voiDrawLine, true);
  }

With this, I can make it look a lot like the traditional line graph, but instead of just giving me columns to plot as lines, the users will have to give me pairs of columns for a single line - just like you'd want.

I'll need to make a new kind of graph, but that's not a problem. It'll be similar to the line, but under the covers it'll be a very different beast. Good job, Gordon!

This morning I saw that the great guys at Panic have released Coda 1.6.1 with quite a few improvements. I've been seeing quite a lot of traffic on the Coda mailing list - a good deal of which was a pretty nasty thread about Git integration. Now I'd love it if they supported CVS and Git, but I'm not going to get out of hand if they don't - face it, the tools don't have to be integrated to be useful.

It's a wonderful took for working on non-Java web sites. Nothing wrong with that. But it would be nice if they had CVS and Git support to the tool.

This morning it's been a hectic morning - lots of things to dig into, fix, write up, explain... it's been a blast. Nothing major, but it's like a wonderful little buffet of things to keep my mind working - and in some cases, exercising my patience as well.

Athens Exchange Acting Up Again

Seems every so often the Athens exchange is acting up from Reuters. Today we didn't get the clearing of the earlyClose.flag and that meant that things were not ticking as the price was based on the earlyClose.price and not the incoming ticks. Once I cleared that up, things started working better. My favorite support guy in the price feed group is going to look into this and try and see why we missed it yet again.

Looking to Filter Ticks by Historical Data

There are problems with the data coming from the exchanges - no doubt about it, but when it comes like it did today, it's a significant pain in the rear, and there's a significant push to try and find a solution for this problem. Today there was an option that was sending a quote (bid/ask) of 0.3/1.2 - creating a price of 0.75. Then a quote of 0/150 would come in making the price 150. One of these is clearly wrong, but is there an objective way to determine that? Not really.

We need to somehow look at the historical data for this instrument and determine if it's correct or not. If the last n ticks have averaged 0.75 with a tight grouping, then the 150 is bad. But it's also possible that the reverse is true. We need to look at the data in a larger sense. However, this isn't easy. The rules you start to make are hard to pin down with any accuracy. You end up making artificial judgements (limits) like 200%, and such. It's not logical, it's experimental.

So I sent out an email asking if now is the time to look at this - or do we want to put this off for later. It's going to take some time as the prices coming from some exchanges are not always clear and complete. Some have CLOSE, for instance, some do not. So we have to make complex rules for all this. And then there's the training process. It's not easy... or quick. It's going to take some time.

So we'll see if this is that time.

Fixed a Few Bugs

Hey... everyone makes mistakes, and this morning I had to fix up a few of mine. This was my gotcha based on the fact that some fields were not getting populated consistently. I had to make special cases for the different ways they got populated, and that fixed the problem. Shucks.

Worked Around Changed Behavior in VantagePoint

In the BKTimeSeriesViewer, the init() method was blowing up with an exception in the hiding of the non-existant points on the graph. It turns out that in the latest cut of VantagePoint, the method getVariables() will return a null of there's nothing on the graph to show. In the past, it returned an empty int[]. This null was caught and we had the problem. So... what was the solution?

Turns out it's pretty simple - allow the null. The method getColumnsOnGraphOnSecondaryY() in the BKBaseGraph was the problem, and if we model it after VantagePoint, then it's simple to say that if there is a null returned from getColumnsOnGraph() (as it calls the VantagePoint method), then it's reasonable to return that from getColumnsOnGraphOnSecondaryY() as well. The rest of the code handles it nicely, so it was a simple change.

Gave this to the developers asking for it and all was well.

Today a developer chatted me saying he was having a problem with the secondary Y axis on a graph he was building. I knew there were the problems with VantagePoint prior to the fixes I made, so I asked him if he had the latest versions of the BKit jar and the VantagePoint jar. Both were important for the fix.

He want away for a minute or two and came back saying that he got the latest BKit jar and it seill wasn't fixed. OK... let's make sure you have the most recent VantagePoint jar too - it's important as well (didn't I already say this? Yes, I did.)

He went away for a minute or two and then said it still wasn't working. OK, I asked him to send me the HTML and I'd look into it. I got the page, stripped out all the crud that wasn't necessary (headers, text, Javascript) - got it just down to the applet and it's tags. When I ran it in appletviewer I got this:

Looks good to me. So I sent him back the HTML after I called him over to see the graph, and agree that it was as he thought it should be. OK, it's got to be on his end.

He comes back a few minutes later saying it's still not working. So I walk over to his desk. The jars are the correct version, and appletviewer on this box shows the right graph, so how about changing the codebase on the applet tag and seeing if appletviewer will show the right graph.

Oh... that worked.

"Well..." I said, "it's got to be IE's cache." - which I had mentioned as a possibility a while back, but he dismissed.

A few minutes later I get a chat from him - all is working. Surprise.

My rant is that this is simple debugging that any developer should be capable of doing. This isn't something special to me, or any advanced or elite status... this is simple Debugging 101. Yet it eludes him so completely. I know it's not going to make any difference, but it's stunning just the same. Some days I wish I had the ability of calling a programming foul and penalizing him 15 yards for illegal use of an education. Something like that. Gimminy.

Over the past several weeks I've been mulling over the MySQL/PostgreSQL and PHP trifecta in my head. Wondering if I needed to just give in and switch to MySQL as it was/is supported by the PHP Apache plugin from Apple. It would certainly make good sense. Then this morning I read this article and I'm glad I didn't.

In the article, MySQL creator Michael Widenius is quoted from his personal blog:

"We still have 20 known and tagged crashing and wrong result bugs in 5.1 [and] 35 more if we add the known crashing bugs from 5.0 that are likely to also be present in 5.1," he wrote. "It's of course impossible to get all issues fixed, but we should at least have tried to ensure that all issues important to a lot of MySQL developers and MySQL users should have been discussed, fixed and/or addressed in a public manner! We should also never have a single serious crashing/wrong data bug in a GA release."

This sounds like a serious quality control issue until you read later that it's systemic to the MySQL group at Sun. Again, from the article:

Despite the alleged friction between prominent MySQL developers and Sun management, Widenius contends that Sun was not responsible for the failures behind the 5.1 release. He blames Mårten Mickos, the senior vice president of Sun's database group and former CEO of MySQL AB. According to Widenius, Mickos no longer treats quality as a priority and unilaterally chose a premature GA release date because "he needs something he can sell."

While I can understand the need to sell a product in Sun's corporate environment, it seems more reasonable to have smaller, more stable releases, as opposed to adding in all the amazing new features in 5.1 just to have something that's feature-rich for the selling. It's certainly not where PostgreSQL is right now, and given this, I think it was the right thing to do, and continues to be the right thing to stick with.

I was hoping that Sun would take MySQL to a new level... but maybe it was a new low. Sigh.