What's it all about, Alfie?

I've been using PHP and PostgreSQL for a long time. I'm talking like in excess of 7 years. I love the database and the language, I like the fact that they work well together, and apart. I've been using them for a very long time and very happy about it. Lately, however, I've gotten a little concerned about the continuation of PHP and PostgreSQL on Mac OS X. Specifically, Marc Liyanage has been providing wonderful packages for PHP and PostgreSQL that work wonderfully well together. But it seems he hasn't released an update in a very long time.

In fact, Apple's PHP version is more up to date, and that's saying something.

Apple's PHP has support for MySQL, but nothing for PostgreSQL, so I can't really use that. If I switch to MySQL I can use Apple's stuff and the packages from MySQL which reportedly are pretty decent, but again, I'm learning a new database and that's not a horrible thing, but it's not something I really want to do, either.

If I wanted to build these myself, then I'd have to build PostgreSQL... use that and the Apache2 on Mac OS X to build PHP, and then I'd be OK. But that puts me in the middle of the build process, and honestly, I'd rather have something that I can update and not have to mess with. Sure, it may sound like I'm lazy, but if I go to the hassle of doing this, I might as well put Marc out of business and start updating the builds myself.

Maintaining a build of PHP and another of PostgreSQL might be OK, but there's a lot of people already doing one of these - it's the combination that I'm looking for. With linux, I get it all for free, but that's because they expect you to want that as an option. It'd be nice to see Apple just throw in the PostgreSQL support into PHP and then point to an installer like they do with MySQL.

But I'm worried this is not going to happen. Maybe I need to just email him and ask him straight away if he's stopped doing the builds. At least then I can make a decision to go to MySQL or try to go the route of the builds. Certainly, if I do the latter, and Marc is out of the game, then I'll have his build scripts to work off of, which isn't horrible.

Lots to think about... unfortunately, nothing really looks all that appealing. Probably the path of least resistance is to get MySQL on my Macs and then use the Apple PHP and be done with it. This way, I don't have to worry about continued support - they seem to have made a choice, and will most likely stick with it.

UPDATE: while I was finishing this up I decided to see if PostgreSQL had a pre-built package for Mac OS X like MySQL does. Turns out, they do. With this, I can be up to date on PostgreSQL without having to depend on Marc. I only need to find a way to get PHP support based on this build. Nice to know.

I ended up sending him messages on his support BB... one for PostgreSQL and another for PHP. We'll see if he replies, or if someone else replies. I can see building PHP like Marc does, but how difficult that might be I do not know.

I noticed that there was a new version of gfortran available from the Mac HPC site, though it has probably been available for a while now. I was looking at the Intel FORTRAN compilers and at $600 each, you have to have something significant to use it on in my book. There's nothing wrong with the gfortran compiler, and while I'm not solving weather problems or simulating nuclear explosions, it's still cranking out answers for me that took machines a few years ago a long time to answer. That's not bad.

I still remember very fondly the days in grad school writing FORTRAN code for the Cyber 205, Gould NP1, and the Sun 3/60... fun days indeed.

I've been wrestling with a VantagePoint bug concerning the placement of Variables on the secondary Y axis, and this morning I got another email from Gordon with an example application that showed the secondary Y axis working perfectly. It took me a little bit to figure out that Gordon had put in the setLicense() call into the code, as I put mine in at another place and his trumped mine and so nothing worked, but that was a "Duh!" moment that was easy enough to figure out.

No, the problem was that on a simple graph, like Gordon's example, where the graph was all set up and ready to go before the first call to unlock() on the graph, didn't show the problem. When I looked at my code, there's all kinds of lock() and unlock() calls, and in general, the construction is far more dynamic - by design.

I was playing around with a lot of things and then came across this little nugget that was the beginning of the end for the problem. When I did the initial setting of the Variables on to the secondary Y axis, the code looked pretty familiar:

  for (int i = 0; i < cols.length; ++i) {
    getGraph().getVariableAttributes(cols[i])
        .setOption(VariableAttributes.voiSecondary, true);
  }

As you'd expect, this runs through the array of int values for the column numbers and sets those that need to be set. Pretty simple. Or so I thought.

When I did this I found that setting cols[2] actually also set cols[0] and if I cleared out cols[0] it also cleared out cols[2]. It's like they were linked somehow. I verified this by logging the status of each getOption() call for each of the columns on the graph after each call to setOption(). I was stunned.

I also was unable to reproduce this in the example Gordon gave me. I wanted to be able to send him back something but it wasn't looking good. Then I got the idea that maybe this was an order of operation issue, like some of the others have been with VantagePoint. Basically, for showing points and labels, it seems there's a necessary order to respect. This isn't required when you build the graph atomically, like Gordon's example, but it is in my code.

So I started messing with the order.

The first thing I did was to move the setting of the secondary Y axis to after the visibility of the points and labels. Immediately, it worked. Wow. That was fast. There wasn't much else to do but to document the heck out of the placement of the code, and then write it up to Gordon. In the end, I'm not sure if he's going to write back with requests for more data. If he does, I'll certainly try to supply it, but it's going to be tough. Simple is not where this guy shows up.

But it's nice to have it working as it's supposed to:

I've been suckered into helping a non-developer pretend to be a developer and write this critical piece of code for trade processing. It's a bad idea from the start, but I knew the probability of him getting it right from the start, without any help, was so near to zero as to be indistinguishable from zero. Some may think me cruel, I think of it as honest assessment. This guy is a good natured, decent guy, but has no formal training in developing at all. Everything he's learned is from hacking around on the job. While that's nice, it's no basis for making a critical component. It just isn't.

So I agreed to help. And help I have. I've been sucked into cleaning up the code many times - each one because he's gone off where we left off and started to add in a new batch of functionality. Lots of copy-n-paste, but not even getting a clear and concise set of requirements from the functional stand-point. What's supposed to happen on these inputs? What about these? Some he knows, and some he doesn't. So I have to help there too.

But today it was a new dimension in this as I got called into his manager's office who clearly didn't know I'd helped this guy to the level that I have. To be honest, the code it 80% mine, and 10% the guy before me, and 10% his. He really doesn't know what's going on, and it's not because he's dumb, it's because he didn't code it and didn't spend any time figuring out what the code is doing.

So we're in his manager's office and he's saying things like "We developed this." Hold on there, Professor... Let's be careful about the use of the pronoun "we" here... I wrote a bunch of it, based on his somewhat dodgey specs. If there's an issue about what it should be doing talk to him - if it's about how it's getting done, then that's me. But this manager is not known for his precise language. He's considered somewhat of a micromanaging developer-wannabe, and as such he's throwing around "we" way too much for my taste.

But he gets mad at me when I say "Watch the 'we', I wasn't in it at this point" to which he gets angry and says "It's our code - that means we built it." Yikes! That's a leap I'm not willing to take at this point in time. So we go back and forth. Finally, we get to the point where he's telling me the specs that I didn't get from the guy I was helping. I put them into the code in all of 15 mins and the tests work wonderfully. Surprise.

It's this getting suckered into this project and then yelled at for not making it perfectly without any specs that makes me a little leery of helping these clowns ever again. After all, it's not my job if this guy can't deliver anything that works. It's his. At some point, if this place doesn't start to turn around, it'll be every man for himself, and at that point I'll be fine, but he's going to be looking at the door.

Late yesterday I was working on enhancing a feature of my fast-tick risk server where I wanted to be able to take research portfolios and load them into the system as if they were real positions - just tagged a little differently so they aren't confused with real positions. As I was doing this, I realized that I needed to parse the option symbol and remove a single component.

In my server, the IBM Dec 2008 85.00 Put is symbolized as IBM:IBM.U:20081220:85.0000:0 where the components are separated by colons (:) and the first is the underlying (many times the symbol for the underlying is not the option symbol), the second is the option symbol, a dot, and the exchange the option is traded on, the third is the expiration, the fourth the strike, and the last is 0/1 for Put/Call. Pretty simple. But for technical reasons of the file formats, I needed to have:

  IBM:20081220:85.0000:0

essentially stripping out the option symbol and exchange. I knew it was possible in Perl, but at the time I was on the train trying to work this out on my way home. Thankfully, OS X has a complete perl reference built-in.

I started assuming that the symbol was given to me. I knew I had it in the script, I just needed to mangle it to the proper form.

  my $symbol = "IBM:IBM.U:20081220:85.0000:0";

and if I did the simple regex on it, I almost got what I wanted:

  my $symbol = "IBM:IBM.U:20081220:85.0000:0";
  print $symbol . "\n";
  $symbol =~ s/(^.*)\:.*\:(.*$)/$1\:$2/;
  print $symbol . "\n";

I got:

  IBM:IBM.U:20081220:85.0000:0
  IBM:IBM.U:20081220:0

and as soon as I saw this, I knew it was because the first wildcard was being 'greedy' in it's matching, and I was deleting the second to the last, not second, component of the symbol. So I looked up the perl docs on my Mac, and there in a wonderful example was the way to make it a stingy match:

  my $symbol = "IBM:IBM.U:20081220:85.0000:0";
  print $symbol . "\n";
  $symbol =~ s/(^.*?)\:.*?\:(.*$)/$1\:$2/;
  print $symbol . "\n";

With this, I was able to match the first part properly and the results were what I wanted:

  IBM:IBM.U:20081220:85.0000:0
  IBM:20081220:85.0000:0

While I knew there was a key to regexs that would make the normally greedy match a stingy match, I'm still amazed at the power of a language like Perl with it's very powerful regex system built in. I put in the code this morning and it worked like a charm. It's really pretty neat that a half-dozen lines of a perl script can add all this functionality. Sweet.

This morning I had a problem with my fast-tick server where it appeared that one of the 'sidekick' threads was not able to successfully process it's run-loop. I noticed this because it started to run the 'purge' of stale greek values, but it never completed the task. When this happened I had to restart the complete app because the code was locked up and yet wasn't crashing. No fun, and it happened twice in one morning!

The relavent portion of the code responsible for this basically did the following:

    /* 
     * Interesting problem... when this guy runs and there is a lot of 
     * work to do, he'll effectively lock out the CalcEngineWorkers and 
     * nothing will get done with the CalcNodes. All because this guy 
     * is too fast and keep locking other processes out. So... I'm going 
     * to put a few 'breaks' into the processing flow. A hundred of them 
     * to be exact, and each time, we're going to see if someone else 
     * needs a turn. Then we'll continue. This is just a cooperative 
     * way to get the job done without locking other threads out. 
     */ 
    int   blockSize = InstrumentManager::numberOfInstruments()/100; 
    // lock up this guy for a read to make sure he doesn't change on us 
    __lockRead(); 
    try { 
      // log what we're going to be doing 
      getLog() << l_status.setErrorId("InstrumentManager") 
               << "taking the time now to purge old data from " 
               << "the instruments" << endl; 
      // next, we need to get an iterator for all the instruments 
      int                   cnt = 0; 
      int                   pass = 0; 
      tIterator< void * >   iter = __allInstruments(); 
      while (iter.hasNext()) { 
        INSTR_BASE      *inst = (INSTR_BASE *) iter.getNext(); 
        if (inst != NULL) { 
          inst->retain(); 
          cnt += inst->purge(); 
          inst->release(); 
        } 
        /* 
         * Check and see if we should yield a bit and see if the system 
         * has something else that needs to get done. With only the read 
         * lock, this isn't a bad place to pause. 
         */ 
        if (++pass % blockSize == 0) { 
          sched_yield(); 
        } 
      } 
      // log what we did 
      getLog() << l_status.setErrorId("InstrumentManager") 
               << cnt << " unnecessary data elements purged from " 
               << "the instruments" << endl; 
    } catch (...) { 
    } 
    __unlock();

the code essentially locks up the list of all instruments for a read, goes through each instrument, telling it to purge any stale data, and then reports on the results and releases the lock. Seems pretty simple. But it's got issues.

First, why maintain the read lock for the entire process? Well... if we don't, then someone can add or remove instruments and we'll not have the first clue about it and the instruments may actually be deleted and that's going to cause us a world of hurt.

Second, why are we waiting to put the retain() on the instrument until we get to it in the iterator on the list? Seems to be a better idea would be to put the retain() on the instrument as soon as possible and then we know it'll be around for us to use when we get around to checking it. Good point, that was one of my concerns about this code when looking at it today.

But the real kicker is the hidden issue: other threads/processes waiting for the lock to be removed so they can modify the list. This may not seem like a lot of work, but with in excess of 400,000 instruments, it takes about 10 seconds (wall clock time) to run through this section of the code. That's a lot. When we are in times of a lot of changes - like the open of the US markets, then this is a real issue. This lock causes us to pause the processing of ticks and greeks and that's no good. So... what can we do to fix this?

Answer: do the obvious: copy the instruments and then process them. More correctly copy the pointers to the instruments and then run through the list processing each.

Why is this a big difference? Well, first off, it means that the lock is only going to be on the main instrument list for the time required to copy about 400,000 pointers. That's essentially nothing. Then the lock is removed and the other threads and processes are free to do what they need.

Second, it allow is to put the retain() on the instrument at the time of the pointer copy, and that means that it's "safe" as soon as possible, and that means it's nearly impossible to have an instrument killed out from under us. Much nicer.

In the end, the code now looks like this:

    /* 
     * We need to make a copy of the pointers to all the active 
     * instruments right now. As we do this, we're going to retain() 
     * each so it doesn't go away. When we're done with each, we'll 
     * release() it to be nice. 
     */ 
    tVector<INSTR_BASE *>   instruments(instrCnt); 
    // lock up this guy for a read to make sure he doesn't change on us 
    __lockRead(); 
    try { 
      tIterator< void * >   iter = __allInstruments(); 
      while (iter.hasNext()) { 
        INSTR_BASE      *inst = (INSTR_BASE *) iter.getNext(); 
        if (inst != NULL) { 
          inst->retain(); 
          instruments.addBack(inst); 
        } 
      } 
    } catch (...) { 
    } 
    __unlock(); 
    // update the instrument count to what we actually have in hand 
    instrCnt = instruments.size();
 
    // next, we need to run through all the instruments 
    int     cnt = 0; 
    for (int i = 0; i < instrCnt; ++i) { 
      INSTR_BASE        *inst = instruments[i]; 
      if (inst != NULL) { 
        cnt += inst->purge(); 
        inst->release(); 
      } 
    } 
    // log what we did 
    getLog() << l_status.setErrorId("InstrumentManager") 
             << cnt << " unnecessary data elements purged from " 
             << "the instruments" << endl;

My initial tests show that the clean-up is being done just as before, but the pauses in the processing are not - which is good, and expected. I'm very pleased with this because I think it's quite likely that this had something to do with the deadlock, and now shouldn't be an issue any longer.

The developer that asked for the 'Zoom Reset' feature on the axes changes was testing this feature today and noticed that when changing the z-axis selection the data was all visible, but the zoom was not reset. Fair enough... any change means any change. So I added in the zoom reset for any change in the z-axis pick list.

Not too bad, and it's what he wanted. Given that I did this for him, it makes sense that he gets what he wanted.

Today I spent some time working on removing the unnecessary calls in the 'simple' BKit graphs that synchronize the graph to the GUI widgets that allow the user to configure the graph. There were two biggies - first off, when adding a column label to the graph, the standard procedure is to sync the GUI widgets to the change at the end of the change. But if we're setting all of the column headers - as we would be at the onset, then we're calling this a lot more than we need to. Fix there is to be clever and set the column headers on the underlying graph - save the last one. Do that last one like the original and have it sync the GUI widgets at the end. This way, (n-1) are done quickly, and the last one is done completely.

The next issue was the updating of the secondary Y pick list based on the addition of columns of data to the secondary Y. Again, this makes sense for the addition of a single column of data, but for the initial set-up, we need to pause the listener handler while adding them, and then resume it when we're done. Simple enough, but to find it took a little bit of time.

I'm not fooling myself into thinking that this is going to make the graphs faster or more responsive - this is a quicker set-up, that's all. But as it was a linear problem, the larger the data set, the more improvement we'll notice. So it's not bad. Plus... don't do something twice if you don't have to. Makes good sense.

This morning I saw that MacTelnet 4.0 was out, and while I'd never heard of it, I wanted to give it a look-see, to see if there has been anything I've missed. I'm always on the look-out for a Terminal.app replacement. I'd settle for getting rid of the scroll bars, but that's not in the cards - at least not today.

Anyway, the MacTelnet 4.0 screen is pretty standard, but there's a border around the screen that is a little annoying. You can't get rid of the scroll bars, and there's no anti-aliasing of the text or manually setting the horizontal and vertical spacing. All these are in Terminal.app and iTerm. Seems that MacTelnet is going for a different audience.

I will say that I like that it's got TEK 4014 graphics. I spent a while in grad school creating a TEK 4014 terminal emulator only to see that VersaTerm on the Mac was far better, and did everything I needed. Still, there appears to be a different audience for this app than the iTerm/Terminal.app crowd. This seems to be for the larger font size, more GUI-based guys. Full screen seems nicely supported, but I'd never use that. Extensibility with Python is nice, but I'd never use that either.

So I'm interested to see where they are taking this, but for now, Terminal.app is still the winner, and iTerm a close second. We'll have to see what MacTelnet evolves into.

Several weeks ago, a developer asked me if it would be possible to expand the functionality of the secondary Y axis on the BKSimpleLineGraphApplet such that when the secondary Y axis was visible, a new pick-list of columns appeared at the top of the widget. This second pick-list would mimic the original but be tied to the secondary Y axis, such that the user could select whatever they wanted to see on the secondary Y axis. This ended up spawning an entire set of issues with VantagePoint and the problems that it's had with the secondary Y axis in version 4.6.6.

While I'm waiting for Gordon to get back to me on the status of that final fix, I wanted to clean things up and check everything in, given that it appears to be working, and the only test is when we get a fixed version of VantagePoint.

So it's built, checked in and ready to go, and should work fine if it's used against version 4.6.4 (but that would require a recompile as the constants have changed).

UPDATE: I recompiled BKit against VantagePoint 4.6.4 to see how it looked and found that the same error exists there! So it doesn't matter if we use 4.6.4 or 4.6.6 - they both have the 'modulo 2' bug and I need to have a fix from Gordon for that. Good to know, sad to see. So I re-built it against 4.6.6 build 209 and that's what the developers can use until we get a fix.

UPDATE: I got an email response from Gordon saying that the next scheduled release of VantagePoint is 11/11/08 - one week from today. He said that this bug would be corrected in this release, and asked if that was soon enough for me. I told him that was fine, I can wait a week for the fix. It's good to know that it's fixed, and when we'll be getting the update.