What's it all about, Alfie?

Just got out of a meeting with my manager. Seems a vendor I've been dealing with here at the Shop is unhappy about the fact that I have posted my experiences dealing with them and their products. Given that they are a multi-million dollar company and I'm a developer, the decision to take all references to them, their people, and their software down was easy. I just did it.

But it's gotten me to thinking... why threaten legal action to someone who's unhappy with your product? It's not like there are real trade secrets that I revealed, at least none that I recognized as such. Sure, there were discussions about architecture and the pros and cons of the design, but I have a feeling that had I said nothing but glowing things about the company and product I might not have been asked to remove them. But maybe so. Maybe they just don't want people talking about them.

Either way... I took it down. But I can't help but wonder who was watching and why. Maybe they have a legal team just scanning the net 24x7 looking for things like this. It's certainly possible. It's just amazing to me to think that my comments generated this level of excitement.

Today I had a good morning... lots of things to check on, fix up, make sure they don't happen again... you know - work. But then I've gotten so far ahead of the rest of the people I'm working with that I was in for a long afternoon of thumb-twiddling. Not exactly what I wanted to do, but there really wasn't much I could do about it. All the projects I'm on are in a state waiting for someone else to do something and there's nothing that's sitting out there waiting to be done.

Things are on order, or the people need to decide what they want, or the people creating this API aren't quite done with it, or the database group haven't made the databases for us... it's all just stuff, but it's all just stuff that I have no control over.

So I work hard at trying to stay busy. Not the easiest thing to do. Hopefully Monday will get some things moving. If not, it might be time to break out Snoopy vs. the Red Baron on my PSP.

Again, I try to stay out of the mergers and acquisitions business realm, but sometimes I have to say something. Microsoft's offer of $44bil for Yahoo is one such time. There was a time that Microsoft was the best thing for the computer industry. Prior to Windows and it's abstraction of the peripherals like displays and printers, every application needed to have people on staff to work out the graphical interface and write drivers for each printer they wanted to support. WordStar was a classic example. Gobs of driver disks and you have to make sure that what you needed to be supported was, in fact, supported by the code.

Major hassles. Then came Windows with it's printer driver model and graphics support. Application support teams no longer had to support each and every printer and display. Way easier for someone to make an app now. This was really the 'gift' of Windows and Microsoft. And it was good.

Similarly, in the beginning, there were a few web crawlers, like lycos - ones that you could use, but you almost always had to have several, and there were even applications on some platforms that would hit several search engines to try and get you something useful with a single query. Then came Yahoo. Again, it was an early pioneer, and it made a big difference in the search engine landscape. It also was one of the first to have 'busy' home pages - portals, many called them, with news, groups, etc. all on the front page.

But with absolute power comes corruption. Unfortunately, Microsoft, like IBM had to battle a horrible reputation that it had taken years to build. Yahoo sat on it's lead and didn't work to keep itself on top of the heap. Now Microsoft finds itself in the position of playing 'catch-up' to Google in the search engine space. Yahoo has long since fallen to Google as well. It's not surprising that Microsoft (with tons of cash) tries to close the gap on Google with the acquisition of Yahoo, but you can't really catch up to #1 by buying #2. The best you can do is to close the gap a bit.

In the end, what needs to happen at Microsoft is they need to go back to the days when they were innovative and not just making money. Like IBM, they need to repair their image with the developers who ultimately make them look good. Take the time to embrace the non-commercial developer. Make amends, and over the long haul, they'll get back into the coveted spot they once held. But it's going to take more than buying Yahoo to do it.

This morning I saw that MarsEdit had gone to v2.1 with a bunch of improvements in the release notes. Honestly, the performance is nice, but there has been one little annoying thing that I've mentioned to the author and he's yet to address - probably because he thinks it's not necessary. That is the minimal size of the posting window.

Window sizes, I've come to learn, are a very personal thing. I've had some people look at my 80x24 terminal screens and think I was insane. But it's what I learned on, and the idea of having a big, wide, terminal is just something I may never use. I'll always deal with 80 column editor windows, and "taller" terminals are OK, and I use them often enough, but for a lot of the work I do, smaller is better.

So when I started using MarsEdit, I liked the fact that it could be personalized to the weblog I had (WordPress), and that just about everything about it was configurable. Very nice. But then I tried to make the posting window narrower - like an editor window, and it stopped short of what I wanted. Now, I'm guessing that I can go into the nib file and change this minimum size, but I sent the author an email and his response was that he needed to make sure there was room for the 'sidebar' in the window. Well... I don't use the sidebar, and do I've got more space than most.

He said he'd look at it, but I'm guessing that it's not going to amount to anything. Maybe I'll ask him again today.

Anyway, the update is nice, and the tool is, in general, a wonderful way to post to just about all the weblogs out there.

UPDATE: basically, he said 'Take IB and change the nib'. Not what I hoped, especially with updates, but it's always been a possibility.

While I know it's not possible in a lot of cases, it really is nice to be able to go back and revisit code you've written several months before. You can see a lot of things that seemed obvious at the time - and now they aren't so obvious, which means you might need to recode them to make them easier to understand, or at least add docs in the code to make it easier to understand why the code looks the way it looks.

Case in point: today I found a minor bug in the pricing logic of the server. When a quote and a trade come in with the exact same timestamp, the logic should have been that the trade 'wins'. What was happening was the quote was winning. Seems like a trivial change - somewhere a 'greater-than' needed to be changed to a 'greater-than-or-equal-to' and that should have been it. Oh... if only it were that easy.

The pricing logic in the server is really, unfortunately, quite complex. We need to look not only to see if the quote is good - meaning does it have the requisite parts, but also does it look right? Meaning, is the bid/ask spread too far to be considered a good quote? We have tons of rules like this in the pricing engine part of the server. So it's not going to be one place. But it was certainly more complicated than it needed to be. So I dug into it.

I was able to simplify it a lot by taking the price and the instrument the price was for as two different logical entities. First, see if the price itself had a 'good trade', and a 'good quote' - regardless of the instrument it might be applied to. Then look at the instrument and see if it was supposed to be quotable, and if so, was the quote more recent that the trade? If not, then don't use the quote.

If I got no quote, then see if I got a decent trade for this instrument. The logic sounds easy, and is... now... but the original code combined too many things at once - trying to see if the price/instrument combo was good. This meant that a lot of checks were duplicated for different instrument types. That's just bad.

So even though it looked logically fine to me 4 months ago, it really needed to be re-written to make it even easier to understand. Now, when we run into issues it'll be a lot easier to follow the logic of the trade/quote part of the pricing section. Almost everything can be improved upon.

Since last Friday I've been really out of it. Bad, bad sickness. That's the primary reason that I haven't posted anything. I haven't written much code, either. It's been a hunker down siege mentality - just get through the days. Then, when I started to feel about 75% recovered yesterday, I decided to come in to work and see what's what at the Shop.

Several meetings - unusual for me, but it's all part of a new 'interest' in the goings on of the development staff. It's gotten big enough that the business (traders) don't know all the faces and therefore have no idea what each one is doing. This shouldn't apply to me, and when I asked, I was told only to include a few broad strokes as they knew what I was doing, but even so, it's a level of inspection that shouldn't apply to someone who deals with the traders on a daily basis. I know my Boss isn't filling out what he's been doing.

Anyway, that kind of micromanaging might be perfectly normal in a lot of places, but here, it's not normal at all. I fill out a journal with everything I do on a daily basis for this exact reason, but they don't want to read that - they want the executive summary. I don't like doing paperwork for the sake of paperwork. They know what I'm doing as I'm doing it in front of them, but with the new org-chart, I've been put in a 'box' that requires that we fill this out. It's more than annoying to me.

If my 6+ years here haven't proven themselves as trustworthy and doing the "right things", then let me know and I'll move on. After a time, asking me about what I'm doing it really quite insulting. Enter the 'disgust'.

More meetings... and then one with a data vendor. As a result I'm supposed to change the utility of one of the market data server's data providers and change it to conform to their new requirements for reporting. It comes down to this: I have to know who's asking for what and ensure that that's the only person looking at it. Well... in an of itself, that's not a bad request, but in comparison to what's out there now, it's a major step backwards. For example, all automated jobs getting data are now dead. Period. I have to know who it's going to, and that person needs to be logged in right them. That's the part that kills the jobs. What if the job takes an hour to get all the data? Today, we have that hour-long job start at 3:00 am so the data is sitting there waiting for them when they arrive.

Tough luck, they say... they have to be logged in and then they can start the hour long request.

It's unreasonable, and rather than allow me to point these things out, they say "Sure, OK" and then it'll be up to me to make it work. Which means I'm going to have to find some silly way to circumvent the literal rules of the license just because they didn't say "Hey, we love this data... but it takes too long to get it how you want us to get it. Make it take 30 sec and not an hour and then we're OK." But no, they know nothing, and yet they are the ones making these decisions.

Classic Dilbert.

I know it happens everywhere, it's just that I've been sick and have no stomach for this today. Gotta get some rest and let it go. Problem is I care, and that's a key reason I'm as good as my job as I am. Who wants to have someone that's not committed to the job? Not me. No way to live.

OK, maybe birth is more than this warrants, but I've been working on the local price injector for a new application and I've finally today gotten it to the point that it's talking to me (via chat)... that it's getting prices from the price feed, checking to see if they need to be sent along the way, and cycling through all it needs to do.

The backups are looking good... heck, I've even managed to have nice sub-second times for the log messages. It's getting close. It's to the point that the major issues remaining are the eternal questions: Where are we getting the symbol set? and the maintenance issues like: Are we going to need web access to this data?... that kind of stuff.

When we get the group together next week it'll be easy to hammer out these issues and then fold this into the code base. But it's a great feeling to get past the first phase of development and see a 'heartbeat' from the app. Sweet.

Today I needed to change our CVS pserver from an old Sun workstation to a linux box because some goobers at work decided to change the network taps on the Sun box and hook it to a non-responsive network port. Thanks, guys. Anyway, I took this opportunity to move the pserver from this old Sun workstation to a linux Blade in the server room. The steps are pretty simple.

First, make sure that /etc/services has the following line in it:

    cvspserver      2401/tcp

Then in the directory /etc/xinetd.d create a file called cvspserver and in it place:

    # default: off
    # description: A CVS pserver
    service cvspserver
    {
        disable = no
        socket_type = stream
        protocol = tcp
        user = root
        wait = no
        server = /usr/bin/cvs
        server_args = -f --allow-root=/path/to/your/cvsroot pserver
        log_on_success += USERID
        log_on_failure += USERID
    }

and then restart xinetd with:

    service xinetd restart

That's it. It works just as well as a pserver on Solaris because it's the same bloody thing. Now I don't have to worry about the goobers and network taps.

Possibly the most useful thing I learned while getting my Ph.D. was totally unrelated to the work. It was simply this: Often the difference between 'decent' and 'great' is a matter of endurance. The movie Galaxy Quest said it more compactly: Never give up. Never surrender.

Today is one of those days.

Several hours ago I realized that I had no motivation for the work I'm doing. I just didn't care. But it's at these times I remember the final year of my Ph.D. where I was simply wishing for the degree or death - I didn't much care some days. But as I got within the last months I saw that there really was a difference between those that kept giving it everything they had even when they didn't believe they had it.

You see, it really is just an impression... a feeling of desperation. Not that I was any really less capable of writing code this morning. Or that code on one project was inherently harder to write than another. Nope, it's all in the emotions of the situation. Take them out of it - even for a little and you'll be surprised that you can really keep going when you think you really can't.

So I kept pushing today. Take a walk to get a drink ever now and then, and get back at it. Never give it a break. Don't stop even when you want to. After a bit you start to see real progress. This may not make things that much better, but it's going to be reassuring tomorrow when you might feel the same, and it'll be nice to know that you did it today. If you did it then, you can do it again.

After a while things will turn around. They always do. If you can keep pushing through the hardest of times you'll feel better about the good times. Just don't give up.

Late last week, Rock Star - the fallen one came by my place and asked me why the two different source of historical prices were taking different amounts of time to return from my market data server. I explained about the caching of the data, and he mentioned that even on the second and subsequent hits, the one source within the Bank was considerably slower than the source outside the Bank. Of course, I tended not to believe him, or put his musings down as experimental error.

But as he showed me the data he was getting, I began to think that he might be onto something. So I wrote up a test case of my own and checked the two sources for the same data for a long enough date range to make the timings significant of the work and not of the overhead involved.

What I found was that the one was faster than the other. And immediately I thought I knew why. The historical data from any source has to be understood within the confines of the requested date range. Say I ask a source for 10 years of data for IBM. It's not going to return anything for holidays and weekends, so it's very possible that the first available day of data is not eh first day requested. If you have the logic of "check the cache for data, fill as necessary, and then respond" you end up asking for the same little bits over and over again. But they never amount to anything.

So I had to modify the general historical cache code that I had written for the second source, and add in the knowledge of the requested first and last dates - in addition to the actual first and last data dates. This made the code hit the cache completely which is a big win, but there was still a little difference in the cache return speed.

What I had done was to get the list of dates between a range of dates, and then using those dates, get the data from the time series for those dates. Over the weekend, I realized that the way I was getting the dates was just inefficient - scanning a std::map is not as good as using lower_bound() and then using that iterator to move through the map until we get to a point where the date is outside our range.

Likewise, it made more sense to subclass CKTimeSeries and provide a 'response filling' method so that we could use the same idea to directly fill the response from the cached data. Something like this:

  void MDTimeSeries::addToResponse( CKTimeTable *aResponse,
                                    const CKString & aSymbol,
                                    const CKString & aFieldName,
                                    int aStartDate, int anEndDate )
  {
    if (aResponse != NULL) {
      CKStackLocker          lockem(getTimeseriesMutex());
 
      std::map<double, double>::iterator    i;
      for (i = getTimeseries()->lower_bound(aStartDate);
           i != getTimeseries()->end(); ++i) {
        // see if we've gone past the limit of the data we want
        if (i->first > anEndDate) {
          break;
        }
        if (!isnan(i->second)) {
          // copy in the data as it's relavent
          aResponse->setDoubleValue((long)i->first, aSymbol,
                                    aFieldName, i->second);
        }
      }
    }
  }

This turned out to be a big win for the speed and it's now faster than the older implementation, which means I can get rid of it at sometime in the future if I want. Not bad.