What's it all about, Alfie?

Today I've spent a good deal of time working with a trader and a little app I wrote for their desk to pull some data from a service in a format that they can read into their applications. It's the kind of thing that I'll put a day or so into and they'll use it for a long time without modification simply because it's a data access component to them. No problem - in theory.

Yesterday, I worked on the socket communications for the clients of my market data server to change the way packets of data were read back from the server into the client's space. Rather than read a 2kB packet and then process it, I changed the code to read everything that was available at the socket, if anything was available. This meant that if we were receiving a 50kB message, we didn't read it in 25 chunks, we read it in one large chunk and then processed that. This made the processing of the data much faster because we didn't re-scan the first 2kB 25 times - we scanned it once. Now a 59kB packet is one thing, but some packets will be 1MB or more. Now we're talking significant savings in time.

Well... today I had to deal with someone that was not convinced that this was faster. In fact, they were convinced that it was slower. Given that they don't know the code, and only see that something has changed, I can understand their need for some kind of assurance that things have changed for the better. So what I did was to run two sets of trials: old versus new, five runs each, same data set to see what we'd get. Ideally, this will be a large enough sample set to be able to factor out the small (or large) variations in the access speed, network traffic, and other variables that you run into on large computer networked applications.

What I found out was that the time to gather the data from the source was somewhat variable, but the time it took to process the data once I had it was pretty controllable. The old way had times for this experiment from 43.2 to 43.7 sec - a pretty nice grouping, and the new way had times from 6.5 to 7.0 sec - again, a nice grouping. While the access times to get the data were much more variable, I made sure to include the access time as well as the processing time so that we could easily see where the time was spent.

Having spent more than a little time with experimental evidence myself, this was a nice sample size, and the breakout of the data made it clear where the variation was, and wasn't. Unfortunately, for this person, the data didn't say the same thing. They didn't see anything like that in the data. They saw the variation in the total time and said "See, the new is slower than the old here, so your changes hurt the system." When I tried to explain that if you looked at the breakout of the times, it was clear that the difference was the time spent in getting the data from the source (not in my control) and the processing time was nearly constant. But there was no convincing them.

For several hours I fought through this - more trials, asking what would convince them, etc. All the while, I'm thinking that these folks are extremely arrogant. It was only after I stepped back for a bit and looked at what they were saying that it hit me - they aren't arrogant, they're just horrible scientists.

Since their background is advanced degrees in science, I made the poor assumption that they actually were decent at reading experimental data and figuring out what the data is saying to them. After all, the job they have deals with experimental data every single day - it's called prices. Stochastic processes abound in this field, and it should have been second nature to these folks to read experimental data, but it's not. So me sense of frustration with their arrogance quickly turned into sadness about their lack of fundamental skills in this area of their work. Sad but true, I can't imagine trusting them with a dime of my money if they can't read experimental data like this.

As is so often the case, we carry in expectations to relationships that are sometimes far below the mark, and sometimes far above. It's all about really getting to know where the other folks are coming from and where their skills and weaknesses are. Now that I've properly calibrated myself to these folks, it'll be much easier to deal with them in the future.

There's no denying that the financial industry has it's share of personalities. And I'm being kind here, as you will soon see. For the most part, I take it as part of the job. They are millionaires and are making (and losing) millions a day, so they are (somewhat) allowed to be high-strung, and a bit temperamental. But today was an experience that I get only once in a great while and it's worth writing about.

One of the users of my market data provider chatted me saying he was seeing vastly different response times for a 300 symbol request of some historical data. I had helped this person just yesterday get another data request under control, so I was wondering what it could be that we didn't get covered yesterday. Well... I got the request and tried it on two machines - the one that he was running it on, and one of my machines. I've learned enough in this business to know that there are all kinds of unseen issues on some boxes - wild processes, memory hogs, etc. all need to be factored out to make sure you're looking at your problem and not someone else's.

What I found was that he was right - there was a significant difference in the response times, and more importantly, the data provider was getting the data to the server reasonably quickly, it was getting the data from there to the client that seemed to be the problem. I told the user what I thought the problem was, and my attack plan. I always like to be as transparent as possible with the users as it lets them know what I'm doing even if they don't understand it all.

He didn't agree with my attack plan. He wanted me to determine why the variance existed in the first place. I tried to tell him that I believed that the gathering/decoding logic in the client API was incredibly inefficient and the resulting CPU-bound process was wildly varying because the load on the box was wildly varying. He wouldn't listen to it.

So I bit my tongue and went on to prove to him that even-though hat he thought was faster wasn't really any faster, it was a sample size of one with a large variation in the load. He tried to tell me that I was all wrong. That he had done the tests properly and the sample size didn't matter. I tried to say what I thought was the problem and what my plan of attack was going to be. He only got madder.

We'll move along through this part of the day quickly because it's really not going to help to cover it in any detail. Suffice it to say that I was treated very unprofessionally, but finally was told to keep him appraised of any updates.

I then took my time and put in tests to see where the real time was being spent. I was surprised to see that the vast majority of the time was in reading data off the socket. In fact, the protocol between the client and the server has all datasets ending in a CRLF combination, and so the data that's read in needs to be checked for this combination.

What I realized was that in reading from the socket, I had limited the data read in from the socket to about 2kB a 'chunk'. Each chunk was then added to the result set and checked for the terminal data condition. But imagine if the data was going to be 2MB? That's 1000 chunks and the first chunk is going to be checked 100 times for the terminal data. There's the problem. So I changed the socket reader to read in everything that's available into the buffer as opposed to a small chunk. What happened was not surprising: all the data was read in at one time, the check was done once, and the result was that the reception of the data was far far faster than before.

When I tested this out on his requests I saw 10 to 40 min requests fall to 2.5 min and stay there. Very little variability now because the process isn't as CPU-bound and the transfer takes priority. Very nicely done, in my mind.

When I gave him the results, he wanted to test it, naturally. When he found out how fast it was, he said "Why haven't we done this before now? We've wasted a lot of time. This should have been done LONG ago!" How nice.

I pointed out that when we put this system together it was an order of magnitude faster than the system it replaced. In fact, it was just fine in the performance department for everyone - even him. That I didn't investigate every possibility for a performance improvement is because I had other things to do as well, and they were all very happy with the speed as it was. He admitted that was true. How nice.

So in the end, it's much faster thanks to a little thought that he didn't want to have me consider, and then was mad that I hadn't fixed the problem that didn't exist until today. Amazing person. Truly one of a kind.

I was out on Friday feeling pretty crummy. Today I'm feeling a little less crummy only because I'm a little more used to it. When I got back, I saw that my analytics engine had been spitting out a bunch of NaNs for this one analytic. So I started to dig into the problem. What I found amazed me - it should have been broken much worse than this much earlier than this.

The engine is written in C++ and as such, those pesky NaNs are something you have to consider, and if you're clever, use them to indicate illegal values, etc. This engine has been running for years doing the same kinds of things day in and day out many times a day - seemingly without problem. But when I dug into it today I realized that a few months ago I changed the datasource for instrument prices and in doing that set the stage for a bug cascade that ended up biting me Friday while I was out.

The problem starts with the data source of prices - Reuters. When they are about 2 hours from a market open, they will zero out all the data in the records they send as an indicator that the instrument is about to enter the active market portion of the day. Normally, this is OK, but the problem comes in when you realize that you need to make a "price" from a bunch of zeros, and you realize that those zeros are telling you to use the historical mark for the instrument and not use the data from Reuters. I thought I had the code in the application to do that, but it seems I was more than a little mistaken.

No... I wasn't ignoring the zero prices, I was converting them to NaNs and putting them into the time series data. I know I was thinking that this would signal later in the code to skip this data point, but even that was unnecessary as I simply should not have overwritten good data with bad no matter what I was thinking I was going to do with it later.

After fixing that bug, I realized that even with the one data point a NaN, there was no reason for all the historical data points to also return NaN. As I looked into the problem more, I realized that I had made the historical calculations biased by the value for today. That way, once I've calculated the historical numbers, I subtract out the value for today and then when I call it again, I can simply compute the value for today, add it to the rest of the values and everything is up to date.

But again, since I was putting in a NaN for a zero price, I was messing up all the values by having the value for today so messed up. Amazing. Fixing it was not too bad - only took a little time, but it was the data change that started the whole problem. It's just amazing to me that I need to keep up with debugging when you change things like the data sources - based on the assumptions that come with the data.

Today I overheard a disagreement between one of the developers here and Rock Star, whom I've discussed before. I have to say that I tried not to listen, and did a reasonably effective job - given that it went on for over 3 hours. Then there was an hour in their manager's office. This disagreement took up the entire morning of these two guys day. Amazing.

The one guy came and told me what it was all about afterward, and I have to say I'm not in the least bit surprised. It was turf wars - plain and simple. The Rock Star wanted to be involved in all decisions relating to a project he's on - but is not the owner of. That, I have always found, is the key distinction. If you own it, then you are responsible for it, and then you're the one expected to make it work. But if you're just on a project, then the guy who owns it, is the one to make decisions like who is involved in what decisions.

In this case, while Rock Star wanted to be the owner, he wasn't, and the manager of the two developers pointed that out. This did not please the Rock Star, but then that's the reason the disagreement between the two took all morning. What seems to have happened is the real fall from grace for the Rock Star. He's now just Veruca - the kid from 'Willy Wonka and the Chocolate Factory' - "I want it NOW!". He's exceptionally focused on getting what he wants, and getting people to give him what he wants, even if they have no intention of doing what he wants them to do.

So the concerns I had about his work are history. There's no need to worry any longer. His manager has a good understanding of how he works, what his requests are, and hopefully, how to handle him. I still think he can make good contributions to the Shop, I think he needs to understand that not everything is going to be given to him how he wants it, when he wants it. There are other concerns at play, limitations in the hardware, production issues, that he's simply not looking at - but needs to. Hopefully, he'll stick around long enough to season into a good developer.

It's been an interesting few weeks seeing how the Rock Star interacts with the other people at the Shop. Interesting in that he's dealing with them in a very similar manner to how he was dealing with me. This observation from a distance has been really educational for me.

First, it shows me that I'm not that different than the other people here in the Shop. We're all trying to meet deadlines with less than perfect tools and support, but we have to get the job done. This means that we all need to recognize that there is a time to ask for assistance from a third-party vendor, and there's a time to take what they have given you and run with it. Nothing is ever going to be perfect, and you have to take what you can get most times.

Second, it shows me that I didn't handle Rock Star all that poorly. I never insulted him. I never treated him like a non-person. While I didn't go out of my way to talk to him, he (and I) had plenty to do and it was just very easy to not talk to him. I certainly could have handled our interactions a lot worse.

Finally, it shows me that in the end, it really is a delicate combination of skills and personality that comes together to make a great team. You can't have people that all get along - without skills they can't really get anything done. You can't ignore the 'human element' either, because you'll end up with people that can do the work individually, but don't really interact well.

I've learned a lot about how I deal with people, and how others do as well. I'm happy to say that I'm pretty pleased with the results. I'm not perfect, no one is, but I'm really happy that I've had this chance to see that I'm not as bad as some folks might like to paint me out to be.

I decided today that I needed to update the technical interview questions I give to new folks wanting a job at the Shop. They are a mixture of database, Java, C, C++ questions that are really meant for the widest possible audience so it's not a big deal if each candidate doesn't know the answers to a few of the questions. C++ STL is, after all, pretty narrowly focused, but for those coming in with C++ STL experience on their resume, it's nice to have a little something to ask them.

Today's additions are primarily from Troy who I sought out for his little "gotchas" in Java as he's been working in Java for a while and I was guessing that he'd have a bunch of little 'pearls' that would make good test questions. I wasn't disappointed.

It's fun to come up with both the simple, quick questions like Why was Thread.stop() depricated? and the more complex ones like What happens in the JVM when a class is instantiated? The additions just give me more material to talk over with the candidate. I don't like to put too much weight on any one topic or language, so it's nice to have more, fun and interesting questions to ask.

Thanks goes out to Troy for his additions.

I was looking a few things up today and I could not help but hear a nearby coder getting angry at the code he was writing. Actually thinking that the code was disobeying him. I'm the first to admit that I've gotten mad while looking at code - most often at the guy who wrote the mess I was forced to fix, but to actually get mad at the code is something I find very funny.

I've always liked coding because it's about the only place in the world where what I asked for was done - without argument, exception, grief, or hassle. I mean really - people are far far more difficult to deal with than a machine. So in terms of frustration, coding is about as stress-free as things come. Oh sure, deadlines are a drag... trying to do more with less... there's a ton of ways to make any job a pain in the neck. But those are reasons to get irked regardless of the job.

It's like a carpenter getting mad at a hammer. It's a hammer - if you don't like what it's doing, hit something else. Don't get mad at the hammer. But I have worked around this particular coder for quite a while, and he's got talent, but he misses the boat when it comes to drive, commitment, and really enjoying the job.

I think he'd rather be somewhere else, but he took this job because it paid significantly better than his last job. I can't blame a guy for making a buck, but if that's the only reason you took the job, it's not going to be long before you realize you aren't really happy and you'll find another job.

One of the things I needed to get going today was the regular mirroring of my Chicago home directory to my new London home directory. In the past, there has been only one, but with the disaster/recovery tests requiring that we take the NFS servers offline, if I want to log into my account in London, I needed to have a local home directory there so that it didn't choke trying to get back to the Chicago file servers.

Once I got word that things were set up properly in London, I checked to make sure that I knew the destination directory - accessible from Chicago, and then I whipped up a little script that would make it very easy to keep things in sync:

#
# Specify the source and the destination
#
srcDir="/home"
target="beatyro"
destDir="/final/London/fileserver/directory/"
# make sure we have the source directory or bail
if [ ! -d "${srcDir}/${target}" ]; then
  echo "You don't have the source directry!"
  exit 0
fi
# ...make sure we have the destination too
if [ ! -d "${destDir}" ]; then
  echo "You don't have the destination!"
  exit 0
fi

# run the command to update the home directory
cd ${srcDir}
rsync -aREpogtxz --delete --exclude='.snapshot*' \
      --exclude='.nfs*' --exclude='.wastebasket' \
      --exclude='.Trash' "${target}" "${destDir}"

I've used something like this guy for a while now, mirroring my MacBook Pro's Users directory to an external drive. The idea is simple, exclude what you don't want, and everything else will automatically get mirrored. By adding in the --delete I made sure that if I removed something from Chicago, it was removed from London too. No need to have stragglers in London.

I like the fact that rsync is capable of doing a lot more than simply working between machines in a client/server mode. In several cases, there's no need to have two machines - you just want to make this directory mirrored on this base path. Simple. Clean.

Add this into a simple, nightly crontab job and every day I'll be sure to keep things in sync. Nice.

As an update to the RockStar programmer that I've mentioned previously, I've been hearing other folks complaining about him and his desire for change in order to show his worth at the shop. And to an extent I can understand his dilemma - if you're a highly paid developer and you're not changing something then you might begin to wonder if they think they need you - as opposed to a junior developer. And that's the big mistake. It's not about your perception of your value - it's how the business values you. Much of the time they can be out of skew with the business valuing you less than you think, but it's just as bad if the reverse is true.

Bad plays are made by desperate people. If you're feverishly trying to show how valuable you are by instituting change so that you can point to your list of accomplishments at review time then you may actually be making things worse for yourself than doing nothing. Take the guy that's changing code simply because it's not Java 1.5-like. This guy might very well miss a few of the subtleties of the existing code and then the converted code may look like new Java code, but it's not working in the same way as the previous version.

This has happened to our RockStar. He's also gotten a few folks cross with him because he doesn't like the fact that testing is not using JUnit for testing. But that's just a tool, and the real issue is not if you're using that tool, but is proper testing being done. Let's face it - long before Java came along, good testing was being done. So was bad testing and no testing and everything in between. But it seems that RockStar, like so many Java fanboys, believe that if you're not using the standard tools then there's something wrong and it needs to be fixed.

It's sad, and I want to tell him to look at what he's doing, but I know that he's not interested in listening to me. Maybe someday when he gets enough experience in this environment he might see that it's really different than he thought, and maybe he'll ask. But if he does he'll be the exception to the rule as most of the Java fanboys I've known simply write this place off as wrong and get a job where more fanboys work and they get to do it the way they want.

Well... as an interesting follow-up to the Victimization by the Majority I was talking to a few folks, including my boss, about this today. What I came away with was that the way I work - with high expectations for myself, and lesser, though still better-than-average expectations of those around me, was the real issue. It wasn't that I'm a mean person. I don't try to make people feel bad, but by being as intense, and demanding of myself, I create a 'gulf' between what others want to provide and what I think they should.

Most importantly, to me, is the fact that the reason that I expect a lot from myself is that this industry pays exceptionally well. And if a developer is getting six-figures, when their same skills in any other similar industry would be worth 30-50% less, then I think they should realize that the premium they are getting paid is not because of the generosity of the employer, it's because they expect more - and that more is effort.

There are literally hundreds of developers in this market that can do this job. It's not rocket science, after all. The premium we are getting paid is because we are expected to respond at all hours, go the 'extra mile' when needed, and we are expected to make fewer mistakes, write better code, faster, and with more of an eye for the future. They aren't paying more just because they want to... they are really expecting more. I agree with their assessment, and am surprised when I've talked to new developers coming into the shop surprised and slightly offended by the mere idea that they should be available any time of day. What did they think they were getting more money for? Their good looks? Please.

So some times, it's not the thought that counts. It's just the actions. My actions are off-putting and aggressive to many and they don't like it. So be it. There are as many that have come up to me in recent days and said that what I'm doing is the right thing and the others need to grow up and realize that they are in a business, and that business is in the business of making money, and for better or worse, I help them make more money than the whiners. So I guess I'm just going to have to try harder to keep my mouth shut, my opinions to myself, and ask people "Are you really sure you want to know what I think?" before answering their questions. If they say "Yes", then I at least gave them an out, and they can't say I didn't warn them.