What's it all about, Alfie?

I was chatting with a friend today and he was telling me the story of a guy he worked with. This guy was holding up the project they were on because he refused to see that he really didn't have the SQL skills to complete this one part of the project, and didn't hand it off to someone with far better skills. He insisted that it would be better for the team if he learned how to do this, and in a sense, he's right. It could be nice for him to be able to do more for the Team, but he shouldn't get that knowledge at the expense of the project. Pick it up on something that has a nice, long timeline, and not the project they are on that's already behind and threatening to harm the business.

It's about understanding one's limits and staying within them. Oh, there are time to stretch your comfort envelope, but when other people are counting on you, and you have the option of handing something off or keeping it for yourself and trying to get it done - you need to think of the Team a bit. Otherwise, people aren't going to want to work with you and you'll end up getting only those things that other people know you can do.

I don't know how this particular story is going to end. I'll check back with my friend in a few days, but it's something I really feel sorry for the guy about. I mean, he probably doesn't even recognize how he is perceived. He probably thinks this is normal behavior. But it's going to get him into a load of hot water when management asks him why it's not done and he's still saying "...I'll have it done tonight."

I realize it's hard to be a vendor to a lot of big companies. Some want stability and verification of easy little point release, and others want to see new features and fixes turned around as quickly as possible. Heck, I run into that every day here. Some groups want stability - which implies very few changes, and others want to see new things they've asked for as soon as possible - even if it means a little loss of stability. I get it... you can't please everyone.

But this can't be an excuse to do nothing. And by nothing I mean you can't have people on site helping to work on your multi-million-dollar product and have them (along with the users) identify bugs and needed features, and then just sit on them. That's not going to be 'OK' with anybody.

But it happens. Can't be helped. If a vendor is unresponsive, and you have to still make it work, you have to exhaust all possible options and avenues before throwing up your hands and saying "Well... we tried everything and still it's no-go. We have to wait for them to fix this."

I've had experience in both cases, and one is not more comforting than the other. You can't stop at the first sign of a vendor problem - you have to inform them, to be sure, but you can't stop there. You have to dig and dig, because you can't be sure that they are even going to look at the problem with the same intensity as you will. Plus, it's another pair of eyes - look and maybe you'll see something that they don't because they're too close to the problem.

At the same time, when you hit the end of the road, you have to accept that it is the end of the road and not try to change facts just because you want them to be different. You have to try and accept the facts. But you can't give up too quickly.

I've run into guys that have simply given up too quickly. Maybe that's all they had... maybe they couldn't see that there was another way to approach the problem. In any case, giving up is not going to make things work. You have to be more determined than that. If the answer can't be found, that's one thing, but you have to be sure that it's not that you're giving up too soon.

After all, you might be dealing with a vendor that's not going to be as responsive as you like, and you may only have a few weeks to get things working or you loose the management backing to continue with the project. You have to take things into your own hands a lot of the time. There's simply no other way to get things working.

It's interesting working with developers and users when it comes to market data. They are often convinced that they need real-time data for prices and calculations driven off prices and they don't really stop to think Why? I do that a lot, and have come to the conclusion that most, and by 'most' I mean anything involving a human being in the loop does not need real-time data. Period. Here's why.

First, there's the calculations. Most applications aren't simple tickers - those are the trading apps, and they need prices that are as close to the market as possible, but then again, if you're trying to watch a few hundred symbols, the odds that you have a powerful enough desktop machine to actually keep up with the ticks from a data source like Reuters is iffy at best. You need it to be as close to the market as possible while not crushing your machine and making it virtually a single-use terminal for data.

So if you have calculations, like exposure, or running P/L, and it's aggregated in any way, then there's very little chance that you have an efficient enough system to actually handle all the ticks that a real-time price feed can dish out. Getting backed-up isn't the answer because then you're behind the market and still have to play catch-up. Nope... you need to be intelligent about what you do.

Secondly, even if you could keep up with the flow, the human watching isn't going to be able to respond to all the ticks individually - heck, it takes us 0.7 sec to hit the brake in an accident situation, there's no way someone if going to respond to a tick for a very liquid stock twice (or more) a second. No way. Automated trading systems are a different beast, but they don't have a human in the loop.

So... the reality of the situation is that for price data feeds you really need a good set of prices. Something that's very close to the market, say less than 3 sec, but not real-time because that's too much. Problem is developers want to have real-time systems because they sound neat. Yeah... I can see that, but it's not reasonable, and when you try and tell them this they aren't at all interested. Rather, they tell you that they can make it happen... and they've thought it all through and have all it takes to do it. This is must likely when the buzzwords come out with newfangled messaging systems to boot.

So you have to back up and explain the realities of these feeds to them. It takes about 30 to 45 mins to get through to most decent developers, and then they start to see the real scale of the problem. Statements like But I'm only registering for 400 symbols turns into "Yes, but you're registering for the most liquid 400 symbols that is going to be a significant real time load." Oh... I didn't think of that. Yes... I know.

Once it's explained, you end up with the standard market data 'bet': Let's try it my way, OK? It's already built, debugged, and ready to go. If this isn't good enough for you and your customers, then we'll do it the other way, OK? In all my experience I've never had to come up with the 'other' way of doing things.

But today I thought 'Why even get into it? Make something that appears to be streaming, even if it's not?' and so I did. I started by creating a nifty set of coalescing queues (FIFO and LIFO) where the push() method takes a key and a value. The key is the primary identifier and the property on which the coalescing will take place. For prices, this is the name of the ticker, but for other things it could be the address, or primary key from a database. The idea is that the order of the queue will be preserved but if you push() a value onto the queue that's already there (as defined by the key) then the value will be replaced, but the order in the queue will be maintained. This means that if you're using this for market data, the prices will keep updating even when you're not servicing the queue, so that when you do service the queue, the order is maintained but the data is the most recent data possible.

I put this into the client code for my price server and then made it possible for users to subscribe for prices and then 'turn on' the delivery of updates and simply "watch the queue" for updates. The queue has all the thread-safety and conditional code in it to make it very easy to simply ask for something from the queue and then as soon as something is ready, it's returned and you can process it and start at the top of the loop again. It's easy to put this in a simple service thread that does nothing but pick things off this queue as they arrive.

It's the illusion of real-time without the headaches. I let the users think they are getting real-time prices and not polling when they really don't know the mechanism that's getting those prices into the queue in the first place. Additionally, they aren't having to deal with identical prices and filtering them out - I do that before I put the prices on the queue in the first place. What it does is really short-cut the argument a bit by saying "try this, and let me know" - and then not hearing from them ever again.

Beautiful solution. I love it.

I've been involved with computers since before the days of widespread Chat, ftp and telnet. It was initially modems with teletypes or (if you were lucky) a glass tty (terminal). There was nothing to download to - save the box you were logged into, and then it was pretty much your entire world. If you wanted to look at another system you hung up and dialed it. Period. It was all we had, and in relation to what came before (keycards and terminal rooms) it was heaven on earth. But things changed.

Ethernet changed a lot, we got networks of machines in college - not yet PCs, and the old hobby machines of the 70's didn't have ethernet cards in them. If you wanted to have them talk to one another you did a special serial cable with Tx/Rx crossed and then you were good to go. But by the time you had widespread adoption of ethernet, you had networks, and when you had networks it didn't take long to have telnet, ftp and chat.

I can remember first using ftp to get at things across the globe while in grad school. There were newsgroups that might publish the site and directory of something useful, and then with anonymous ftp, you could go get it. It wasn't fast, and there were no Google-like search engines in place... you had to log in, search the directory tree for the file you were looking for... download it and hope it made it down before you lost your connection. Still, this was big computer to big computer, it wasn't until the PCs came out that you really saw the growth of the Kermits, XModem, YModem, ZModem file transfer protocols. FTP was almost forgotten.

But IRC Chat staged a massive growth, and telnet was the way to get from one machine to another. So not everything was forgotten. But fast-forward to today. It's amazing to me to see that ftp, chat, and telnet (ssh) are as strong now as they were in the early days - not because something better hasn't come along, but because the work put into the documentation way back then.

Look at the RFC for FTP or Telnet someday... it's amazing the detail they went into. I've implemented both, and was immediately impressed that the RFC was right on point with the intended audience - people wanting to understand and implement the protocol. The docs are very well written, complete and detailed, but not overly verbose and wordy. These things were written by people wanting to pass on this knowledge to others in the industry and make sure that there were no lingering questions and problems.

FTP is, to this day, a great way to get files around, and is in every web browser around. Telnet (SSH) is virtually unchanged - save for the additional security, but is just as useful. It's Chat that seems to have taken on almost mythical proportions.

Look at all the Twittes, IMs, IRC Chat clients. It's all the same, basic premise - I type and you see it, you type and I see it. They first version I used was even called type on the Unix BSD4.x systems at Purdue. But look how many ways you can now communicate through this little concept. You've got the store-n-forward of SMS or Twitter... you've got the IRC Chat and a ton of different IMs. It's as if this - communication between people - was the real killer app of the network. Sharing a computer meant files and resources of one were available to another. Putting the machines on different desks took that away - only to be brought back by the network.

I use Chat (Colloquy), IM (Adium), and Twitter (Twitterrific) all day every day. It's fun to think about where things started and how they have evolved over the years. I'm sure the video chat will be bigger when the bandwidth is there, but right now, it's just not. Give it time, though, and it'll be just like all the Sci-Fi movies you've seen. It's a great time to be alive.

I got an email this morning from the UnixEng crew - a good lot of guys and gals, to be sure, and they were saying that one of the machines we have here in the development group was having a ton of network errors. The switch was set to 100/Full, and typically, that's what we need because we've learned that the auto/auto negotiation seems to have more problems than it's worth.

So the network guy switched the port to auto/auto, and that helped a little, but I wanted to set the linux box to 100/Full because I know that's better than leaving it full auto. Typically, we do this when we build the box, but a lot of times it gets skipped (forgotten) because things start out working, only to fail at a later date.

Anyway, the nice tools to remember here are ethtool and ifconfig. If you run ifconfig and look at the data for the ethernet port (eth0 in this case), you can see if there's a likely problem in the port.

Typically, you might see:

   eth0   Link encap:Ethernet  HWaddr 00:17:A4:99:07:EF
          inet addr:146.180.7.94  Bcast:146.180.7.127  Mask:255.255.255.128
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:123456 errors:0 dropped:0 overruns:0 frame:0
          TX packets:123456 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelength:1000

but if you see large numbers for the errors or the collisions then you're probably looking at a duplex mismatch. The way to see what your card is set at right now is to run ethtool on the port:

   ethtool eth0

and a lot of nice, useful information about eth0 is going to spill out.

If the network card needs to be forced into a mode, then the easiest way to do this is to edit /etc/sysconfig/network-scripts/ifcfg-eth0 and add the line:

   ETHTOOL_OPTS="speed 100 duplex full autoneg off"

and a quick reboot later and things should be fine. That is they should be. Today it was something different.

When I made the changes to the /etc/sysconfig/network-scripts/ifcfg-eth0 file and rebooted, the services Avahi and HAL failed to start. When HAL failed to start, the box hung. Thankfully, I could get it to boot into single-user mode and start to look at things.

Google pointed to SELinux, but that was very clearly disabled on this box - as it should be. We looked at a lot of things - X11 drivers, etc. In the end, I was faced with the fact that it was the only thing that has changed on the box, and so I removed that one line.

Bingo! That was it. Now what's odd is that that line is on another box with the same hardware spec - so we're baffled as to why for this machine there's a problem. It could be the version of the kernel or packages - a yum update would help, but it's working now, so we may put that off. It's also possible that it's the switch ont he other end... but that's going to be very hard to pin down.

In the end, we're going to keep an eye on it, but what a mess trying to fix something like this.

OK, this is something I never thought I'd have to do, but this evening I had to password protect my blog. I was getting more grief about it from the management where I work - specifically saying that I could no longer discuss anything related to people or events that took place at the office. In one sense, I can understand my manager's issues - he's told by his boss that this shouldn't have happened, and it shouldn't have, and he responds by trying to make himself look less of the bad guy and more of a sympathetic representative of The Man.

Interestingly enough, you can't do this password protection in the WordPress configuration. At least not unless it's hosted at their site. I don't know why this is, but I found a plugin that allows me to put a password on anything related to the weblog. I'm not sure it's 100% crack-proof, but it's certainly well past a good-faith effort to keep people out of this blog.

UPDATE: If I turn this on, then I can't use MarsEdit to write my posts. That's not something I can live with. What I can do is to post all the messages as drafts so that they aren't visible to the average viewer until I specifically publish them. They are still held on the server, just in a draft state. This way, I can make it appear as though I'm not posting anything else while I am - just not making it visible to anyone but me.

[2/7/08 3:31am] UPDATE: I'm trying to get the password plugin to work with MarsEdit. Hopefully, this will work. Excellent! By excluding the XML-RPC URL from the default password protection, I can allow WordPress' password protection on that to work and the plug-in's password protection to work on everything else. It seems that I have a very workable solution now. No one can look into these pages without the password.

[2/11/08 4:00pm] UPDATE: I've taken it offline simply because I've received more talking to about this weblog. I was meant to help me feel better, enjoy what I've done, and act as a record of the problems I've solved so I don't have to solve them again. But it's become a boat anchor and I've got to shut it down.

Today I was working on loading up my new price injector with the instrument data from the vendor's system and was originally thinking that if they just made their database layer available to me via their supplied shared library, I'd be able to send the query, read the answer and be done. Alas, they don't allow that.

But what they do have is a stand-alone app that takes a file with a request in it and can generate a file with the output of the command. I was a little concerned about how it'd all glue together, but I shouldn't have been. C/C++ really does have just about everything you could ever ask for in a language, and interfacing with stand-alone apps is no exception.

I was able to build the command, execute it to a temp file that I then read after the command was done. It wasn't as clean as it could have been, but it really wasn't that bad either. Thank goodness that all I had to do was get a list of instrument data.

So in the end, my concerns about the clunky nature of this interaction was really much ado about nothing. C++ made it much smoother than I was afraid it was going to be. Then again, when we fill up disk space and the output file can't be written, I might be singing another tune. But that's for another day.

Just got out of a meeting with my manager. Seems a vendor I've been dealing with here at the Shop is unhappy about the fact that I have posted my experiences dealing with them and their products. Given that they are a multi-million dollar company and I'm a developer, the decision to take all references to them, their people, and their software down was easy. I just did it.

But it's gotten me to thinking... why threaten legal action to someone who's unhappy with your product? It's not like there are real trade secrets that I revealed, at least none that I recognized as such. Sure, there were discussions about architecture and the pros and cons of the design, but I have a feeling that had I said nothing but glowing things about the company and product I might not have been asked to remove them. But maybe so. Maybe they just don't want people talking about them.

Either way... I took it down. But I can't help but wonder who was watching and why. Maybe they have a legal team just scanning the net 24x7 looking for things like this. It's certainly possible. It's just amazing to me to think that my comments generated this level of excitement.

Today I had a good morning... lots of things to check on, fix up, make sure they don't happen again... you know - work. But then I've gotten so far ahead of the rest of the people I'm working with that I was in for a long afternoon of thumb-twiddling. Not exactly what I wanted to do, but there really wasn't much I could do about it. All the projects I'm on are in a state waiting for someone else to do something and there's nothing that's sitting out there waiting to be done.

Things are on order, or the people need to decide what they want, or the people creating this API aren't quite done with it, or the database group haven't made the databases for us... it's all just stuff, but it's all just stuff that I have no control over.

So I work hard at trying to stay busy. Not the easiest thing to do. Hopefully Monday will get some things moving. If not, it might be time to break out Snoopy vs. the Red Baron on my PSP.

While I know it's not possible in a lot of cases, it really is nice to be able to go back and revisit code you've written several months before. You can see a lot of things that seemed obvious at the time - and now they aren't so obvious, which means you might need to recode them to make them easier to understand, or at least add docs in the code to make it easier to understand why the code looks the way it looks.

Case in point: today I found a minor bug in the pricing logic of the server. When a quote and a trade come in with the exact same timestamp, the logic should have been that the trade 'wins'. What was happening was the quote was winning. Seems like a trivial change - somewhere a 'greater-than' needed to be changed to a 'greater-than-or-equal-to' and that should have been it. Oh... if only it were that easy.

The pricing logic in the server is really, unfortunately, quite complex. We need to look not only to see if the quote is good - meaning does it have the requisite parts, but also does it look right? Meaning, is the bid/ask spread too far to be considered a good quote? We have tons of rules like this in the pricing engine part of the server. So it's not going to be one place. But it was certainly more complicated than it needed to be. So I dug into it.

I was able to simplify it a lot by taking the price and the instrument the price was for as two different logical entities. First, see if the price itself had a 'good trade', and a 'good quote' - regardless of the instrument it might be applied to. Then look at the instrument and see if it was supposed to be quotable, and if so, was the quote more recent that the trade? If not, then don't use the quote.

If I got no quote, then see if I got a decent trade for this instrument. The logic sounds easy, and is... now... but the original code combined too many things at once - trying to see if the price/instrument combo was good. This meant that a lot of checks were duplicated for different instrument types. That's just bad.

So even though it looked logically fine to me 4 months ago, it really needed to be re-written to make it even easier to understand. Now, when we run into issues it'll be a lot easier to follow the logic of the trade/quote part of the pricing section. Almost everything can be improved upon.