What's it all about, Alfie?

Yeah... remote access to work is both a blessing and a curse. There are a few times when it's nice to be able to be at home and get a little work done, but in reality, my work is done on several monitors, with (literally) a dozen different Vim sessions open and at least a half dozen shells open to edit, compile, run, monitor and debug the code. It's work. So having a laptop with a 14" screen is not really getting that kind of work done, but in a pinch, it can be done.

No, realistically, it's the support of production systems that remote access is for. That's where it really pays for itself. If you have to get in and fix a database issue, or do a quick fix to a script, then there's nothing like it. Nothing. Getting in a car, driving 45 mins to the office at midnight is just not realistic. You need to be able to do some things - triage the problem, remotely in a few minutes. That's what remote access into work is for.

So while "new guys" look upon it as a liberating, no almost empowering status symbol. "I have remote access now." But then a few years on, they realize that they have no recourse when work needs to be done on off-hours. So it's the thoughtful, or egocentric person that refuses remote access even when it's asked that he have it.

Such was the case today I learned of a co-worker in another land. This co-worker is going to hold up the deployment of a multi-million dollar system because he doesn't want to come in on the weekend and refuses to accept the offered remote access solution so it can be done from home. Interesting idea. Part of me is impressed at this guy's sense of boundaries. There is work, and there is home, and they are not to be confused with one another. Bravo.

But if I were this guy's boss, it'd be a simple business fact of life. One person, no matter how well-intentioned, respected, kind, generous, and giving, can hold up a multi-million dollar project. No one. Period. If he can't do it, then he's got one choice - get it or get out. It's as simple as that. Management is not about being nice. It's about managing a project with risks and rewards. If one person has the ability to hold up that process, then they are a fundamental risk that needs to be addressed. That risk must be eliminated, or at least significantly reduced.

And while I admire his bravado, I am appalled at his arrogance. I don't want to answer problems in the middle of the night. But I do. It's a fact of life in a team where the business runs 24x5. Period. I've taken countless calls - from this guy in the middle of the night, and not complained about it. Yet he feels he's entitled to be the one holdout to this invasion of our personal time.

Good for him. But if I were this guy's manager he'd be toeing the line or walking out the door. It's that simple. This place needs to have a little sacrifice. There aren't enough people to cover things completely all the time. You have to take the lumps now and then. And by making this stand, he's saying there's no reason to do it, and thus empowering the next person to take a similar stand.

If this catches on, there's not going to be anyone for him to call when he needs help. I'm sure he doesn't care, as he'd blame us for not being available.

So I label this person an International Prima Dona. They believe the rules apply to everyone except them. They are the One, after all. In the end, he's nothing more than someone expecting special treatment, and that's likely to get him special treatment by a lot of folks. I hope he's really ready for it.

I've been a user of GIMP for more than a decade, and it's always been a solid graphics tool - invaluable on some platforms that have nothing else, but even a nice to have on the Mac. There are just some things that it does exceptionally well. But of late, the builds of GIMP for Mac OS X for Leopard have required that you update the X11 package to a more recent build of the XQuartz project - the basis of Apple's X11 package. While I've wanted to stay current with GIMP, I did not want to have a non-standard X11 install. So I was stuck.

Until now. Today I saw that GIMP for Mac OS X 2.6.5 was released, and I decided to download it on the off-chance that it would run on a stock Leopard install. It did! Fantastic!

While it's not going to be something I run all the time, like Acorn, it's something I'm really glad I have at my disposal when I need it. This is wonderful news.

I was looking at lockless data structures for multi-threaded programming again this morning, and I came across a link to TransactionKit. This sounds very interesting to me. Basically, the group has implemented a thread-safe NSDictionary and NSMutableDictionary pair that are completely safe for multi-threaded environments, use no locks, and offer the basics of transactional integrity: begin transaction, commit and rollback functionality. And I have to admit I'm very intrigued.

The biggest challenge to me in working with these lockless data structures is that you might not know you're going to need them when you start the project. It's only after you build the project, see it scale for a while, and then hit a fundamental limit that you see it's time to switch over. Problem is, by then, you're stuck. You've got all this code that uses one form of thread-safety, and now what you realize you need is something else all together. But you can't justify the complete re-write.

What you need is lockless alternatives to standard data components that are easily swapped in when the 'going gets tough'. For example, I like TransactionKit's take on the inclusion in Objective-C - make it look like a simple NSDictionary and NSMutableDictionary so that it's a simple search and replace for the vast majority of the retrofitting. This makes things a lot simpler.

But now I'm looking at the CKit stuff. It'd be one thing to have a truly lockless CKFIFOQueue. Then another that uses mutexes for those times when it's not a major issue for speed. Finally, one that uses the lockless atomic calls to make it appear to be lockless, in fact, you can even simply No-Op the lock() and unlock() methods in the subclass, but all the while, the thread-safe behavior is in play.

The problems with general coding are a lot more difficult, however. Take for example, the retain/release code I posted a few days ago. That's not so simple. Previously, I had:

void Instrument::retain()
{
    // first, lock this guy up
    CKStackLocker      lockem(&mRetainReleaseMutex);
    // now up the count
    ++mRetainReleaseCount;
    // see if it's in the pool to be released, if so, remove it
    if (mRetainReleaseCount == 1) {
        CKStackLocker      lockem(&mReleasePoolMutex);
        mReleasePool.remove(this);
    }
}
 
void Instrument::release()
{
    // first, lock this guy up
    CKStackLocker      lockem(&mRetainReleaseMutex);
    // no decrease the count
    --mRetainReleaseCount;
    // see if it's an error
    if (mRetainReleaseCount < 0) {
        getLog() << l_error.setErrorId("Instrument.release")
                 << "the release count for " << mmSym() << " went negative! ("
                 << mRetainReleaseCount << ") This is a serious problem." << endl;
    }
    // see if it's at zero, and then if so, add it to the pool
    if (mRetainReleaseCount <= 0) {
        CKStackLocker      lockem(&mReleasePoolMutex);
        if (!mReleasePool.contains(this)) {
            // log that we're going to dump it in the trash
            getLog() << l_error.setErrorId("Instrument.release")
                     << "adding " << mmSym() << " with retain cnt="
                     << mRetainReleaseCount << " to garbage" << endl;
            // ...and then do it.
            mReleasePool.addBack(this);
        }
    }
}

where the critical points were one (1) in the retain() code and anything less than or equal to zero in the release() code. This was due to the logical assumption that the value of mRetainReleaseCount would be initialized to 1. But what if we have it initialized to zero? Then the critical points are zero for the retain() code and anything negative for the release() code.

#include <asm/atomic.h>
 
void Instrument::retain()
{
    /*
     * Increment the count. If it's zero, then we were ready to flush this guy
     * and now we need to pull him out of the garbage before he gets flushed.
     */
    if (atomic_inc_and_test(&mRetainReleaseCount)) {
        CKStackLocker      lockem(&mReleasePoolMutex);
        mReleasePool.remove(this);
    }
}
 
void Instrument::release()
{
    /*
     * Decrement the count, and if we are negative then it's time to add this
     * guy to the release pool for deletion.
     */
    if (atomic_add_negative(-1, &mRetainReleaseCount)) {
        CKStackLocker      lockem(&mReleasePoolMutex);
        if (!mReleasePool.contains(this)) {
            // log that we're going to dump it in the trash
            getLog() << l_error.setErrorId("Instrument.release")
                     << "adding " << mmSym() << " with retain cnt="
                     << mRetainReleaseCount << " to garbage" << endl;
            // ...and then do it.
            mReleasePool.addBack(this);
        }
    }
}

if I then initialize the value of mRetainReleaseCount with:

    ...
    mRetainReleaseCount(ATOMIC_INIT(0)),
    ...

in the constructor's code, then we'll start with zero, climb positive, and then when we go negative we'll by tossed into the release pool for the next cycle on the garbage collector thread. Since the garbage collector thread also checks the retain/release count before it wipes out the instance, we're safe.

At least I'm pretty sure we are...

Well... here's the problem I see now. Because we have removed the mutex on the retain() and release() methods, we can be doing both at once. The count will be handled properly, but what if the release() hits it's counter operation first, but the retain() method's if() body executes first to get the lock on the second mutex?

Now we're in trouble. The counter will be saying 'leave the instrument out of the pool', but the code will execute in the order indicating that the instance will be placed into the pool. Crud.

Yup, this is not easy. I was ready to put the code into play, but there's no guarantee that the body of the conditional will execute in the same order as the atomic operation. This means that it's possible to have them in the wrong order. This, in turn, tells me that the greatest application of these atomic operations are in the thread-safe lockless data structures. That's about it. When you have two operations that need to be treated atomically, then you're sunk. Darn.

I have users in many different timezones, and I do my best to keep them all straight - most of the time. But there are times - around the switches where I'm not exactly sure what the time is in London - or Hong Kong. So a few years ago, I got VelaClock for my Mac.

It's a wonderful little Dashboard widget that allows me to keep track of each timezone in a nice, little, compact widget. It's much better than the World Clock widget from Apple, and it's got all kinds of nice little capabilities that I just don't use.

This morning, I noticed that they released version 2.1.11. It's too bad the auto-updater in the widget doesn't work, but that's no big deal. I can go to the site and download it from there. Nice to have for people dealing with a lot of different timezones.

Found out today that iTerm 0.9.6.20090209 has been released with a few bug fixes on the rendering of bold text and a permissions issue. Nothing major, and while it's nice with it's ability to add the window border on the bottom (or not) and the ability to remove the scroll bar, I'm thinking it's still not ready to replace Terminal.app because of the window sessions.

Still... maybe it's something to suggest to the developers? Might be worth a try.

I was listening to an interesting podcast today about lockless state management with atomic operations and got to thinking that there's a lot this can be used with if you spend a lot of time to get things into simple word-sized blocks. But there's a ton that can be done just as-is.

What I'm thinking about right now is the simple retain/release counters that I have on my instrument objects in the fast-tick server. Face it... they are about as simplistic as you can get. This article by IBM talks about the atomic operations in linux, and looking at a few include files, I can see that it's supported in the version we're using simply by including asm/atomic.h. Good news.

What I'll need to do is read up more on this and see if I can fit these easily into the retain/release code. If I can, then it's probably worth trying. The number of times I hit that code is non-trivial, and that's all kernel-space work. Better to be in user-space and not have the overhead. The code I have now looks a lot like this:

void Instrument::retain()
{
    // first, lock this guy up
    CKStackLocker      lockem(&mRetainReleaseMutex);
    // no up the count
    ++mRetainReleaseCount;
    // see if it's in the pool to be released, if so, remove it
    if (mRetainReleaseCount == 1) {
        CKStackLocker      lockem(&mReleasePoolMutex);
        mReleasePool.remove(this);
    }
}
 
void Instrument::release()
{
    // first, lock this guy up
    CKStackLocker      lockem(&mRetainReleaseMutex);
    // no decrease the count
    --mRetainReleaseCount;
    // see if it's an error
    if (mRetainReleaseCount < 0) {
        getLog() << l_error.setErrorId("Instrument.release")
                 << "the release count for " << mmSym() << " went negative! ("
                 << mRetainReleaseCount << ") This is a serious problem." << endl;
    }
    // see if it's at zero, and then if so, add it to the pool
    if (mRetainReleaseCount <= 0) {
        CKStackLocker      lockem(&mReleasePoolMutex);
        if (!mReleasePool.contains(this)) {
            // log that we're going to dump it in the trash
            getLog() << l_error.setErrorId("Instrument.release")
                     << "adding " << mmSym() << " with retain cnt="
                     << mRetainReleaseCount << " to garbage" << endl;
            // ...and then do it.
            mReleasePool.addBack(this);
        }
    }
}

The trick will be to do the entire method atomically. I can't just change the count. The checks are also critical, but maybe if I do the change, and keep the flag, that will work. Problem might be that another operation gets in the and wants to undo my work, but I come later... it's not an easy problem to solve with a single atomic operation on an int.

Certainly something to think about.

This morning I noticed that Apple had released two updates on Software Updates: Apple Java for Mac OS X 10.5 Update 3, and Security Update 2009-001. The Security Update is pretty obvious - there's a few underlying libraries that have exploits in them and these are the patches. I've read that it includes the Safari RSS security problem written up a while ago. Good enough. The more interesting one to me is the update of Java.

It's supposed to update the Java WebStart which is interesting as I've been using it for quite a while now, and I'm convinced that it's a valuable way to deploy a Java app to remote sites with auto-updating. Very slick. Also, they are supposed to have worked on the applet functionality. Again, good news.

So it was a reboot, and that's a pain, but maybe someday they'll have a "restore state" for the entire login and then it won't be so bad. Yeah... that would be very nice...

While it's not a major update, there are a lot of bugs in the version I was running, and so Daniel J. released an updated version of MarsEdit that addresses the biggest offending bugs he's come across in the past few months.

I think I'm going to like a few of these features on the preview pane, but I'll certainly be looking forward to the next big release with lots of cool, new features.

OK... well, this has been an interesting exercise. I've been working on a Python script to grab data out of a system here, and generate an XML file for a vendor upload. It's a big file, and there's a lot of things that have to go into it, but I wanted to be able to do as much as possible from within the Python code itself. If I had to post-process it and add tags later, then that's OK, but I didn't want to start with that plan. I wanted to make the utmost of python's capabilities for this task.

For the most part, I haven't been disappointed. Python has a pretty nice XML DOM system (xml.dom.minidom) built-in, and I was able to create the document very easily. The problem came in dumping it to a file. There were two basic ways I could dump it:

<exchangeTraded><id type="SEDOL">B10RB15</id>
<positionName>1605=JP</positionName><amount>-214.0</amount></exchangeTraded>

where it all appeared on one 'line' and no spaces were to be found, for the 'simple' XML output, and:

<exchangeTraded>
    <id type="SEDOL">
        B10RB15
    </id>
    <name>
        1605=JP
    </name>
    <amount>
        -214.0
    </amount>
</exchangeTraded>

for the 'Pretty Print' output. Neither was really good, but as the lesser of two evils, the second one was far more readable for the size of the file I'd be sending. But I knew that I needed to get back to this as soon as reasonably possible and fix this up with a decent output processor.

Today has been that day. I was quite lucky that I got a good head-start with Google, and was able to put together the following replacement for writexml():

def new_writexml(self, writer, indent="", addindent="", newl=""):
    # indent = current indentation
    # addindent = indentation to add to higher levels
    # newl = newline string
    writer.write(indent + "<" + self.tagName)
 
    attrs = self._get_attributes()
    a_names = attrs.keys()
    a_names.sort()
    # lay down the attributes on the tag
    for a_name in a_names:
        writer.write(" %s=\"" % a_name)
        xml.dom.minidom._write_data(writer, attrs[a_name].value)
        writer.write("\"")
    # now lay down the child nodes
    if self.childNodes:
        if len(self.childNodes) == 1 and \
                self.childNodes[0].nodeType == xml.dom.minidom.Node.TEXT_NODE:
            writer.write(">%s</%s>%s" % (self.childNodes[0].data, self.tagName, \
                        newl))
            return
        writer.write(">%s" % (newl))
        for node in self.childNodes:
            node.writexml(writer, indent + addindent, addindent, newl)
        writer.write("%s</%s>%s" % (indent, self.tagName, newl))
    else:
        writer.write("/>%s" % (newl))

where the arguments are exactly the same as the original version, we've just cleaned up the printing of the child nodes when there's only one and that one is a text field. To install it into the proper place in the runtime, I simply need to:

#
# This is the main working section of the script
#
def main(argv):
    # hook in our new XML DOM writing code
    old_writexml = xml.dom.minidom.Element.writexml
    xml.dom.minidom.Element.writexml = new_writexml

and it's good to go with the same old code.

The results are perfect:

<exchangeTraded>
    <id type="SEDOL">B10RB15</id>
    <name>1605=JP</name>
    <amount>-214.0</amount>
</exchangeTraded>

I'm sure I'll be using this again very soon, and I wanted to document it so I didn't forget it. Really quite simple, but the effects are quite dramatic on a large file with thousands of nodes.

I've been writing an XML file generator in Python hooking into a few data sources (systems) here at the Shop. I'm really quite pleasantly surprised at how easy it is to do all the little things you need to do. Formatting, logging, database, it's all there. With a decent access to the data, and a nice object model, it's downright pleasant.

I have to admit I'm surprised. I expected a more 'stripped down' language than this has shown itself to be. Yes, it's supposed to be all wonderful and that, but until you really need to dig into it, refactor it, and do all the little things that you're not going to find in a simple 'Hello, World!' app, you can have your doubts, as I did mine.

The object model is nice, if not a little confusing. The proliferation of self is a trifle annoying - can't this be inferred from scope? Maybe not. It's something that I can get over, but it is a bit odd at first.

The indenting at least makes things look nice. The speed is first-rate, I'll give you that. There isn't anything I can complain about there.

If I spent a lot of time in it for a few weeks, I'm sure I'd pick up a lot of habits that would make it very nice to write in. Libraries, or finding those libraries like the XML DOM, are real time savers, and yet it takes time to look them up, learn how to use them, etc. Time worth spending, but time you need to spend.

I have to say... it's not PHP, but it's really quite nice.