Archive for the ‘Cube Life’ Category

Dropbox – Amazing, Seamless, and Just what I needed

Wednesday, November 26th, 2008

Dropbox.jpg

This morning I was reading the feeds, like I always do, and came across a reference to a piece of code done right. The author was talking about how the best software gets out of your way and does everything that's needed without you having to think about it. In this case, it was shared, remote filespace. The app/service was Dropbox.

What made this so useful to me was the fact that I had gone through the headache of setting up a WebDAV server at home for just this reason - a place to throw things that I could get at form other machines. The problem is, I was still limited by my home bandwidth, I wasn't being intelligent about sending diffs, and the clients for other operating systems were web browsers or FTP-like clients. It worked, but it wasn't what I wanted.

Dropbox is exactly what I wanted. It's a smart, version controlled, remote file repository. It's got clients for Windows, Linux and Mac OS X. Once you register for it, you simply have a directory that "hot syncs" to the repository in "the cloud", and sends the updates to your other registered machines.

I have a laptop and two desktops. I register them all. I see on each machine an identical set of files. Make a change in one, and it'll be mirrored to the others in a very efficient manner. Drag a new file into the folder on my laptop, and it'll automatically be sent to my two desktops. Simple.

I still have the file on my laptop - which makes this a ton better than WebDAV, and the syncing is done automatically, but clearly visible so that I know when something's happening. There are default folders for public content, photos, an easy way to make URLs for accessing the files... the list goes on and on.

Certainly one of the nicest features of this is the fact that it's really a file repository so you can go to the Dropbox web site and see the files you've deleted. And get them back! You can't beat that. It's like Time Machine built-in.

The first 2GB of storage is free, and for $9.99/mo or $99/yr you get 50GB. I'm sold. The version on the app is 0.6.402, and I'm guessing they do an automatic update, but we'll see. I feel so much better with the idea of installing this on my machines and then being able to really share files on the 'net. Incredible app.

Working Side-by-Side with Users Testing Features

Tuesday, November 25th, 2008

SwissJupiter.jpg

Today I got the chance to work side-by-side with a user that was testing changes to this project I had been making over the last few days. Normally, they'd be testing and writing up a ton of trouble tickets and then I'd have to go through all the effort of finding the duplicates, thinning them down, filling out the paperwork... it's all a lot easier if they just ask me to sit with them and we can work though things so that they understand what's going on under the covers.

Thankfully, today I was able to do just that. A user asked me if I wanted to sit with him or have him file a bunch of tickets. I picked the former, in a heartbeat.

It was amazingly simple. Walk this user through a few screens, explaining the details of what's happening under the covers to him and all of a sudden, things became crystal clear. It's amazing what a little bit of good, honest communication will do to help a situation. In this project, it's been seriously lacking, and the project has suffered for it. I'd like to think that maybe things are changing - at least for me.

Sitting with users and explaining the same thing a dozen times is not wonderful but it sure beats the alternative - a user gets frustrated because they're convinced that the code isn't working right, and then you try to explain it to them, and they have a bias now to see the error as an error as opposed to just a misunderstanding.

I hope this continues... I'm going to do my best to make sure it continues. I think it's the only way to really save this project.

Getting Back Into a Rhythm

Tuesday, November 25th, 2008

cubeLifeView.gif

It's been a few days now that I've been back from my back problems that kept me home for a week. It's going to take a few more to get feeling like I'm really back in my old rhythm, but even now it's nice to be able to feel a little more like my old self again. I tell you, being laid up, unable to walk, unable to really do all the things I was doing only a few days ago... it really makes you glad for the little things - like taking a shower and being able to go to work.

I'm guessing after the Thanksgiving Day break I'll be feeling even better and more rested. Then it'll be back into the rhythm for the rest of the year. Nice to be getting back at it.

Enlightenment is Never a Gradual Thing

Friday, November 14th, 2008

PHB.gif

I have never considered myself a real people person. Sure, I like people. I like talking to them, I like listening to a good story, and even like listening to them when they don't necessarily have a good story. But I'm happier with fewer people around me. I don't like crowds. I find great comfort in silence. So I would not consider myself to be someone that's got a great read on people.

Because of how I was raised, and what life experiences I've had, I, like many people, do not really see myself as others see me. I don't see when someone is threatened by me if I am not threatened by them. I tend to filter what I see, by what I'm experiencing at the time. Consequently, I do not see deception very well. I'm easily fooled.

In one sense, I like this. I love magic. I know it's not real. I have a great friend that does magic, and he's shown me a lot of the tricks. The manuals, books, tricks, and the very elaborate and expensive machines that really "do" the magic. It's an illusion, but one that I gladly succumb to.

I have held my current position for over seven years. As an hourly consultant. That's a long time in this business. Not as long as some, but a lot longer than others. Why? I believe that it's because I work very hard, I'm smarter than the average bear... in short - I Produce. No two ways about it. I make ideas a reality in timescales that simply don't apply to the other people I work with.

I've heard the CTO call it "Bob Time".

Fair enough. I think it's a compliment, and I take it as such.

But some people close to me (Liza and friends) have been saying for years that many people I work with are threatened by me. In the past, I've agreed - as if to say "Sure, I believe you", when I really didn't. I mean really - who'd be threatened by me? I'm about the most un-threatening person I've ever met. (OK, here I understand that this is very subjective and I'm probably way off base on this, but you get the point.)

Yesterday I was asked to a meeting with several managers in the Shop with the topic of how to move forward in the coming year(s) with the service that my application provides. It was carefully phrased that it wasn't about my application (suite, really) - per se... it was about the service that my application provides. Fair enough, I can't imagine what they are planning on doing - seeing as how there's no application to take it's place (really) in the near future, but I'll play along.

So I get to this meeting and on the manager's whiteboard I see designs about what "could be" where they have gutted my fast-tick server and replaced it with another risk system. It's possible, in theory, to make this replacement, where the theory fails is in the timely delivery of the amount of data necessary to keep the end-users seeing what they want to see when they want to see it.

I am very familiar with the replacement engine. It's a package from a vendor that I cannot name (already been threatened with lawsuits) and a product that I cannot name (ditto). But that doesn't make it any better a product. It's a mess. It's a multi-million dollar mess. And in this meeting I find out that we have contractually agreed to buy 80 seats of this app which they now realize is a horrible waste of money. Not being good business people, they don't understand the concept of sunk cost, and are determined to leverage this expenditure to ruin other things as well.

At the current time, my app does everything it needs to, faster than anyone expects, and at a cost that is so low it's hard to imagine anything purchased beating it in cost/benefit analysis. But they want to leverage this other (bad) product. So they want to merge the two and make the "best of both" solution.

That's the start of the meeting.

I was the only technical person there.

They never asked my opinion. Not once.

They were far more interested in telling me their ideas then listening to why it was a good idea, or not, and what really should be done to try and make use of this horrible multi-million dollar white elephant. I never understood why until today.

They really are threatened by me.

Not for their jobs... I'm the lowest of the low on the totem pole. There's no one I threaten for their job. But their reputation is something else. Also, as they consider themselves to be "smart people", they want to be able to look back at what's been built and say "There - see that? I told him how to do it." Ego.

At the end of the meeting another issue was raised, and I was told that I'd do what one of these managers asked of me - and they knew I didn't like it, and didn't care.

So to add to the insult, they show that they have power over me, and even on top of that, they point out that they don't care - effectively putting me in a 'servant' role. How low did they plan on placing me? Slave?

I was upset, and left early. I didn't go in today, and had the opportunity to watch a little of the show House on DVR with my 14 yr old son. I'm glad I did... it was the seed of The Enlightenment.

One of the characters in this episode asked House (a genius doctor with horrible people skills) why he got into medicine when it was very clear that he disliked people so much. "Research, where you could make your own world, would seem a better fit". House responded with a little story about his childhood in Japan.

He and a hurt friend went into the wrong door of a hospital and saw this janitor scrubbing floors. They found their way to the Emergency Room and as his friend was getting fixed up, House walked around a bit. There was a patient that the doctors were stumped on. Totally baffled about the disease. So they finally decided to call in the specialist. Through the door walks the janitor. He was one of the 'untouchables', but was so smart, and always had the right answer, that they knew when they were completely beaten, they could go to him and he'd know the answer.

House went on to day "It was then I knew it wasn't about fitting in. It was about being right." It hit me like a ton of bricks. That is very similar to how I'm treated by my co-workers at the Shop. I'm the person they call when they have no idea what the problem is, but know in their bones that I'm going to have the solution to their problems.

Case in point: Yesterday one guy couldn't log into his linux box. He'd been able to for weeks, so it wasn't something configured wrong. It was new. Also, someone else was able to get into his box and all was OK. He and his neighbor worked on this for at least two hours. No luck. Then they came to me. I had ti solved in 5 mins. Time. The clock on the box wasn't being synced to the time servers so kerberos didn't like the time shift and disallowed the login. Simple fix - set up the step-tickers file and start ntpd. Easy.

But in the face of the one guy, he was angry. Very angry. He's relatively new - only a few weeks here, but is an old friend of the management group. Good Old Boy. He's never said 'Hi' to me after I've said 'Hi' to him in the halls. Many would call his actions downright rude. So for me to have fixed the problem he and his co-worker couldn't was a blow to his ego.

And the same is true for the meeting I was in.

No one wanted my opinion. They wanted to shape my opinion. When they saw that it wasn't happening, or they felt they'd said what they wanted to say, then they stopped and the meeting was over. That I hadn't agreed with them made the last part easier, but it would have happened anyway. Maying sure to put me in 'my place' was essential to their sense of order. I was (and am) capable of doing great things in record time. This is undeniable. Talent like that usually commands respect. Or fear. They were simply showing themselves to me to be the kind of people that fear me.

It's sad, but it became as clear as day with that TV show. I'm not someone to be treated as an equal. In that sense, I'm 'untouchable'. Yet they know I can solve these things like no one else can, and in so doing, make them look good for having made the choices they have. When you make good choices you don't need a miracle worker to make them look good - so almost by definition, I'm only called in on the very bad decisions.

When I do what they ask, and make something wonderful out of a mess, they hate me for being able to do it so effortlessly. Anger, fear, power... it's not a great combination in a group of people. It leads to bad things.

It's clear to me that they will never treat me kindly. It's not what they are ever going to do. I'll finish my time here and get a new job, because there's no place for me long-term where so many of the management team fear and hate me.

In a Good Organization, Communication is the Key

Monday, November 10th, 2008

cubeLifeView.gif

This morning I walked in to find that over the weekend about a dozen of my machines around the globe had been rebooted. Normally, this is OK because the Unix Engineers tell me this and I get online on Sunday evening and fix things up so that everything is ready to go in the morning. But because they didn't tell me, I had no idea, and was playing catch-up for a few hours this morning to get things up and going.

Now I realize that this isn't the end of the world. It impacted a few users for an hour tops, but the idea is that the preventative medicine here was a 10-sec email. That's it. When the effort to make things "good" is that little, I'd think it's worth doing. Keep the lines of communication open. Make sure people understand what's happening. That's all I'm asking for.

Well... this time it was more than they did. I was very disappointed, but things got going. It's just that I don't feel the need to keep them up to date of my changes, if they aren't going to keep up their end. It's degenerating into an every man for himself environment and that never works out well.

The Problem With Agreeing to Help Another Project

Friday, November 7th, 2008

Detective.jpg

I've been suckered into helping a non-developer pretend to be a developer and write this critical piece of code for trade processing. It's a bad idea from the start, but I knew the probability of him getting it right from the start, without any help, was so near to zero as to be indistinguishable from zero. Some may think me cruel, I think of it as honest assessment. This guy is a good natured, decent guy, but has no formal training in developing at all. Everything he's learned is from hacking around on the job. While that's nice, it's no basis for making a critical component. It just isn't.

So I agreed to help. And help I have. I've been sucked into cleaning up the code many times - each one because he's gone off where we left off and started to add in a new batch of functionality. Lots of copy-n-paste, but not even getting a clear and concise set of requirements from the functional stand-point. What's supposed to happen on these inputs? What about these? Some he knows, and some he doesn't. So I have to help there too.

But today it was a new dimension in this as I got called into his manager's office who clearly didn't know I'd helped this guy to the level that I have. To be honest, the code it 80% mine, and 10% the guy before me, and 10% his. He really doesn't know what's going on, and it's not because he's dumb, it's because he didn't code it and didn't spend any time figuring out what the code is doing.

So we're in his manager's office and he's saying things like "We developed this." Hold on there, Professor... Let's be careful about the use of the pronoun "we" here... I wrote a bunch of it, based on his somewhat dodgey specs. If there's an issue about what it should be doing talk to him - if it's about how it's getting done, then that's me. But this manager is not known for his precise language. He's considered somewhat of a micromanaging developer-wannabe, and as such he's throwing around "we" way too much for my taste.

But he gets mad at me when I say "Watch the 'we', I wasn't in it at this point" to which he gets angry and says "It's our code - that means we built it." Yikes! That's a leap I'm not willing to take at this point in time. So we go back and forth. Finally, we get to the point where he's telling me the specs that I didn't get from the guy I was helping. I put them into the code in all of 15 mins and the tests work wonderfully. Surprise.

It's this getting suckered into this project and then yelled at for not making it perfectly without any specs that makes me a little leery of helping these clowns ever again. After all, it's not my job if this guy can't deliver anything that works. It's his. At some point, if this place doesn't start to turn around, it'll be every man for himself, and at that point I'll be fine, but he's going to be looking at the door.

Perl and Regular Expressions are Pretty Amazing

Friday, November 7th, 2008

perl.jpg

Late yesterday I was working on enhancing a feature of my fast-tick risk server where I wanted to be able to take research portfolios and load them into the system as if they were real positions - just tagged a little differently so they aren't confused with real positions. As I was doing this, I realized that I needed to parse the option symbol and remove a single component.

In my server, the IBM Dec 2008 85.00 Put is symbolized as IBM:IBM.U:20081220:85.0000:0 where the components are separated by colons (:) and the first is the underlying (many times the symbol for the underlying is not the option symbol), the second is the option symbol, a dot, and the exchange the option is traded on, the third is the expiration, the fourth the strike, and the last is 0/1 for Put/Call. Pretty simple. But for technical reasons of the file formats, I needed to have:

  IBM:20081220:85.0000:0

essentially stripping out the option symbol and exchange. I knew it was possible in Perl, but at the time I was on the train trying to work this out on my way home. Thankfully, OS X has a complete perl reference built-in.

I started assuming that the symbol was given to me. I knew I had it in the script, I just needed to mangle it to the proper form.

  my $symbol = "IBM:IBM.U:20081220:85.0000:0";

and if I did the simple regex on it, I almost got what I wanted:

  my $symbol = "IBM:IBM.U:20081220:85.0000:0";
  print $symbol . "\n";
  $symbol =~ s/(^.*)\:.*\:(.*$)/$1\:$2/;
  print $symbol . "\n";

I got:

  IBM:IBM.U:20081220:85.0000:0
  IBM:IBM.U:20081220:0

and as soon as I saw this, I knew it was because the first wildcard was being 'greedy' in it's matching, and I was deleting the second to the last, not second, component of the symbol. So I looked up the perl docs on my Mac, and there in a wonderful example was the way to make it a stingy match:

  my $symbol = "IBM:IBM.U:20081220:85.0000:0";
  print $symbol . "\n";
  $symbol =~ s/(^.*?)\:.*?\:(.*$)/$1\:$2/;
  print $symbol . "\n";

With this, I was able to match the first part properly and the results were what I wanted:

  IBM:IBM.U:20081220:85.0000:0
  IBM:20081220:85.0000:0

While I knew there was a key to regexs that would make the normally greedy match a stingy match, I'm still amazed at the power of a language like Perl with it's very powerful regex system built in. I put in the code this morning and it worked like a charm. It's really pretty neat that a half-dozen lines of a perl script can add all this functionality. Sweet.

Pushing More Ticks Through the Fast-Tick Server Safely

Thursday, November 6th, 2008

servers.jpg

This morning I had a problem with my fast-tick server where it appeared that one of the 'sidekick' threads was not able to successfully process it's run-loop. I noticed this because it started to run the 'purge' of stale greek values, but it never completed the task. When this happened I had to restart the complete app because the code was locked up and yet wasn't crashing. No fun, and it happened twice in one morning!

The relavent portion of the code responsible for this basically did the following:

    /* 
     * Interesting problem... when this guy runs and there is a lot of 
     * work to do, he'll effectively lock out the CalcEngineWorkers and 
     * nothing will get done with the CalcNodes. All because this guy 
     * is too fast and keep locking other processes out. So... I'm going 
     * to put a few 'breaks' into the processing flow. A hundred of them 
     * to be exact, and each time, we're going to see if someone else 
     * needs a turn. Then we'll continue. This is just a cooperative 
     * way to get the job done without locking other threads out. 
     */ 
    int   blockSize = InstrumentManager::numberOfInstruments()/100; 
    // lock up this guy for a read to make sure he doesn't change on us 
    __lockRead(); 
    try { 
      // log what we're going to be doing 
      getLog() << l_status.setErrorId("InstrumentManager") 
               << "taking the time now to purge old data from " 
               << "the instruments" << endl; 
      // next, we need to get an iterator for all the instruments 
      int                   cnt = 0; 
      int                   pass = 0; 
      tIterator< void * >   iter = __allInstruments(); 
      while (iter.hasNext()) { 
        INSTR_BASE      *inst = (INSTR_BASE *) iter.getNext(); 
        if (inst != NULL) { 
          inst->retain(); 
          cnt += inst->purge(); 
          inst->release(); 
        } 
        /* 
         * Check and see if we should yield a bit and see if the system 
         * has something else that needs to get done. With only the read 
         * lock, this isn't a bad place to pause. 
         */ 
        if (++pass % blockSize == 0) { 
          sched_yield(); 
        } 
      } 
      // log what we did 
      getLog() << l_status.setErrorId("InstrumentManager") 
               << cnt << " unnecessary data elements purged from " 
               << "the instruments" << endl; 
    } catch (...) { 
    } 
    __unlock();

the code essentially locks up the list of all instruments for a read, goes through each instrument, telling it to purge any stale data, and then reports on the results and releases the lock. Seems pretty simple. But it's got issues.

First, why maintain the read lock for the entire process? Well... if we don't, then someone can add or remove instruments and we'll not have the first clue about it and the instruments may actually be deleted and that's going to cause us a world of hurt.

Second, why are we waiting to put the retain() on the instrument until we get to it in the iterator on the list? Seems to be a better idea would be to put the retain() on the instrument as soon as possible and then we know it'll be around for us to use when we get around to checking it. Good point, that was one of my concerns about this code when looking at it today.

But the real kicker is the hidden issue: other threads/processes waiting for the lock to be removed so they can modify the list. This may not seem like a lot of work, but with in excess of 400,000 instruments, it takes about 10 seconds (wall clock time) to run through this section of the code. That's a lot. When we are in times of a lot of changes - like the open of the US markets, then this is a real issue. This lock causes us to pause the processing of ticks and greeks and that's no good. So... what can we do to fix this?

Answer: do the obvious: copy the instruments and then process them. More correctly copy the pointers to the instruments and then run through the list processing each.

Why is this a big difference? Well, first off, it means that the lock is only going to be on the main instrument list for the time required to copy about 400,000 pointers. That's essentially nothing. Then the lock is removed and the other threads and processes are free to do what they need.

Second, it allow is to put the retain() on the instrument at the time of the pointer copy, and that means that it's "safe" as soon as possible, and that means it's nearly impossible to have an instrument killed out from under us. Much nicer.

In the end, the code now looks like this:

    /* 
     * We need to make a copy of the pointers to all the active 
     * instruments right now. As we do this, we're going to retain() 
     * each so it doesn't go away. When we're done with each, we'll 
     * release() it to be nice. 
     */ 
    tVector<INSTR_BASE *>   instruments(instrCnt); 
    // lock up this guy for a read to make sure he doesn't change on us 
    __lockRead(); 
    try { 
      tIterator< void * >   iter = __allInstruments(); 
      while (iter.hasNext()) { 
        INSTR_BASE      *inst = (INSTR_BASE *) iter.getNext(); 
        if (inst != NULL) { 
          inst->retain(); 
          instruments.addBack(inst); 
        } 
      } 
    } catch (...) { 
    } 
    __unlock(); 
    // update the instrument count to what we actually have in hand 
    instrCnt = instruments.size();
 
    // next, we need to run through all the instruments 
    int     cnt = 0; 
    for (int i = 0; i < instrCnt; ++i) { 
      INSTR_BASE        *inst = instruments[i]; 
      if (inst != NULL) { 
        cnt += inst->purge(); 
        inst->release(); 
      } 
    } 
    // log what we did 
    getLog() << l_status.setErrorId("InstrumentManager") 
             << cnt << " unnecessary data elements purged from " 
             << "the instruments" << endl;

My initial tests show that the clean-up is being done just as before, but the pauses in the processing are not - which is good, and expected. I'm very pleased with this because I think it's quite likely that this had something to do with the deadlock, and now shouldn't be an issue any longer.

Trying MacTelnet 4.0 as Terminal Replacement

Tuesday, November 4th, 2008

MacTelnet.jpg

This morning I saw that MacTelnet 4.0 was out, and while I'd never heard of it, I wanted to give it a look-see, to see if there has been anything I've missed. I'm always on the look-out for a Terminal.app replacement. I'd settle for getting rid of the scroll bars, but that's not in the cards - at least not today.

Anyway, the MacTelnet 4.0 screen is pretty standard, but there's a border around the screen that is a little annoying. You can't get rid of the scroll bars, and there's no anti-aliasing of the text or manually setting the horizontal and vertical spacing. All these are in Terminal.app and iTerm. Seems that MacTelnet is going for a different audience.

I will say that I like that it's got TEK 4014 graphics. I spent a while in grad school creating a TEK 4014 terminal emulator only to see that VersaTerm on the Mac was far better, and did everything I needed. Still, there appears to be a different audience for this app than the iTerm/Terminal.app crowd. This seems to be for the larger font size, more GUI-based guys. Full screen seems nicely supported, but I'd never use that. Extensibility with Python is nice, but I'd never use that either.

So I'm interested to see where they are taking this, but for now, Terminal.app is still the winner, and iTerm a close second. We'll have to see what MacTelnet evolves into.

Added Simple Arithmetic Methods to the BKTable

Thursday, October 30th, 2008

BKit.jpg

Today a developer using BKit chatted me asking if it would be possible to add some simple arithmetic operations to the BKTable. Basically, adding, subtracting, multiplying and dividing the values in the table by simple external values without having to pull out the Object, get it's double value, do the math, create a new Double, and place it back in the table.

It seemed like a very reasonable suggestion. After all, because of the JEP parser, we had the ability to add complete tables, it only makes sense that we do the same for the contents of the individual cells in the table. Problem was, I didn't want to reproduce the code for the arithmetic operations themselves as I'd already written that once. I wanted to leverage that without making the code overly complex and include a JEP parser for each operation.

Thankfully, that wasn't necessary. The way JEP is structured, the operations are classes. I have sub-classed their Add, Subtract, Multiply, and Divide classes to make BKAdd, BKSubtract, BKMultiply, and BKDivide already to make the complex operations in the JEP parser work. Again, luck was on my side in that each of these had simple methods to really do the heavy lifing.

BKAdd used add(), BKSubtract used sub(), and so on. This meant that as long as the BKTable had an instance of BKAdd, it could add. So I make transient ivar of a BKAdd, BKSubtract, and so on to the class so they wouldn't be shipped around the universe, and then sent to work making the methods.

It was somewhat tedious work because I wanted to have forms that would take either integer or String indexing into the table - that's standard for the BKTable, and also have a version that took the generic Object, but also one that took a double for those times when it's just a bunch of numbers, and you don't need the added complexity.

Because it's possible to have nulls in the table before the operations happen, I took the logical stance that a null was the same as a zero for these operations. So, adding a number to a null gives you that something. Multiplying something by a null gives you the null - simple, but important, I think, if you're doing this to minimize the hassles of dealing with Objects when you want primitive values.

Took a while, but it was really nice to see it work. The ability to modify the individual cells in a BKTable now is really nice. It is going to make some of the work done with BKTables a lot cleaner.