What's it all about, Alfie?

Today I spent most of my day dealing with a horrible system that has been folded, spindled, and mutilated far more than anything I've ever seen. This code has methods like:

    void klugeToGetAroundAnotherFatAssBug( ... );
    void klugeToGetAroundAnotherFatAssBug_node( ... );

and I'm not kidding, either. The old coders of this mess didn't think enough of the even older coders of this mess to write decent methods and just fix things. No, they had to make it personal. This is what I'm walking into. I really hate it.

Anyway, today I had to try and track down a data problem that was effecting just one instrument in the application. I looked in the tables in one of the databases it uses, and found the table the data should be in. Nope... it's empty. Nice try. But then why keep the table there? Just to mess with people's heads?

Then it hit me - it's The Curse. Good, decent programmers have to work on this code and it is such an incredible burden to them that they go insane and go work in flower shops, or pump gas for a living. It's painful, but it's real.

I want to get this solved, and then document a few things about this mess and then move on. It's not going to be easy, and it's certainly no fun, but it's what I do.

There's a reason I like working where I do - it's not the hours, that's for sure. It's that for the most part decisions are made by the appropriate people. That means that if there's a need for the entire organization to be involved, then the decision is handled at a very high level. However, if it's the decision to update from an unsupported product to a supported version of the same thing, then, again, for the most part the decision is handled at a low level as there's not really a lot that we're going to do about it. We're going to have to update, and what we update to is really the only question.

Today I've had a couple of things I'm working on go from being properly handled to being very improperly handled. A while back it was the ordering of hardware and the request to get the vendor's technical sales reps involved. True to my prediction, they knew nothing. They took more than two weeks to come back and tell us that they knew nothing. Total waste of time.

A few weeks ago, we wanted to get a nice messaging bus in - Tibco EMS. We knew it was going to be a ton of money, but there were a ton of reasons to spend it now and convert the Shop to Tibco. But since it wasn't going to solve a pressing problem, it was into the "project" category... complete with timelines, Visio diagrams, step-by-step proposals. The Works!

I read a wonderful quote the other day attributed to Mark Twain: The best time to fix a hole in the roof is when the sun is shining. The reality here is much more like: They only time to fix the hole in the roof is during a monsoon. I appreciate that this is not pocket change that we're talking about, but the problems facing us aren't likely to be solved in an afternoon, either. Serious problems demand serious people and serious tools. This is not something to slap together with perl and Visual Basic.

But that battle is lost. Today I was trying to get a new version of the modeling libraries into production. I try to keep a low profile - there's no real option. We have to do it, so the fewer people that know about this the better. The fewer people involved the better. More visibility almost always translates to slower decisions, more arguments, more grief, and in the end, since there's really no other option, simply a slower roll-out of the needed functionality.

I tried not to get flustered with this as it was a friend that made the critical mistake of involving too many people early on. His opinion is (was) that this would be 'no big deal', and could not have been more wrong. I appreciate what he was trying to do. He just wants to keep people up to date and he thinks it'll be a slam dunk. But when someone usually thinks that I have the nervous feeling that if they are wrong, they'll be wrong in a very big way.

So it got involved with four more people that have very little time to look at things and have very little idea of the real alternatives open to us. This means that they'll look at a difference of 0.0005 and say "That's too much - match it exactly!". But Dude! That's the old numbers... we match the new ones exactly. Let's not get bogged down about the difference on what was - let's look at what is.

But I know that's not how it's going to go. It's going to be a few days of people discussing it and trying this permutation on the data, and that input set, and this and that until they get frustrated and finally see that it's the old values that are off and give up. In the end, it'll be just as if they never got involved. They'll add nothing, because there's nothing to add. But it's something to argue over.

This the is most Dilbert-like part of my job. I really don't like it.

Today was a little mixed bag of coding issues. I had some old code to clean out in the server for option strike formatting that was causing me grief, and then after yesterday's problems with my Sun development box for the MareketData server, I decided to make sure that I could build it on another box - the other development Sun box that my co-worker has.

For the most part, it was building pretty nicely. I had to fix up a few things and move a few directories to a new partition to make space for them, but other than that it was humming along pretty nicely. Then I started noticing some stalls in the network traffic and did a little netstat -i to see that there were some in-bound errors. The network crew replaced the cables from the box to the switch and I'm hoping that will help, but I really think it might be a duplex issue on these auto-negotiation NICs and switches. We'll see.

There was also a mount point issue, and after I pointed out the problem they updated the YP maps so that after the nightly push, I'll be able to reboot the box and then get the right maps for the box. With this, I should be able to build the code on the other box. That will certainly give me a nice buffer... some breathing room, so that if my development box goes on the fritz again, we have some place to build the code for a production problem like we had yesterday.

I also added in some interesting little code to the MarketData server to allow all the symbol mappers for each of the data providers (each provider has it's own symbology, and these components map from our symbology to the native symbology of each provider's API) to change their mapping parameters. For the most part, these are database connections where I run stored procedures or SQL queries to get the native symbology components from our symbology components.

What this is going to allow me to do is to change the database connections on the fly through the chat interface of the server. I can also query the server and see what parameters are being using by each provider. This would have made the need for a rebuild unnecessary yesterday, and it makes switching as easy as typing in a few lines to the chat bot for the app. Not bad.

We also gave a presentation about the existing infrastructure and why we want to use Tibco EMS as opposed to the home-grown message brokers that we have. It was basically showing the other folks - including management, how much we depend on custom communication protocols and why it'd be a good idea to standardize on something like Tibco and get the non-standard stuff out of the shop. It's going to be a hard-sell, unfortunately, as it still represents a ton of money to do the switch. When it's important to them, they'll find the money. Until then, I can make what we have work.

Today was the first day back from the long weekend where we went to Philadelphia for Liza's marathon. It was an amazingly crazy day that had more than it's share of issues and problems that I had to overcome. Not a fun day - not by a long shot.

First, I had more of the 'NFS not found' errors on my Sun box. This was supposed to have been cleared up with the new networking and cables, but it wasn't. This was made infinitely worse today as I had problems with the production MarketData server and without my Sun box to develop/debug, I was in the dark. They came and replaced the network card in the box and reconfigured it to use that new NIC as opposed to the one on the motherboard. This took about an hour from the time I started 'yelling' about the problem and it's impact to production, so that's not too bad, but that it was out after having told them to replace the NIC weeks ago is a little troubling. But I hope this is it for this problem.

Next, there was the problem that I needed that box to solve. That was, the database that I get symbol mappings from was not returning data as it should have. If I went into SQSH, the data looked to be there, but when I went in from the program and the SQLAPI++ libraries, it wasn't. After a bit I gave up and pointed my code to the primary database (as opposed to the read-only replicant) and it worked perfectly. I had seen a few database issues this morning early, but thought they were all fixed up. I was wrong.

Later this afternoon, I saw that the read-only database was working again, and I'll switch production back at the end of the day. I try to be a good corporate citizen as much as possible and sing the read-only database is something they want me to do. I'll comply as long as it's working, but when it fails, it's back to the primary as the users have to come first.

After I got that fixed up, I had to fix up problems with my server and the EQSVal libraries. It turns out that with the most recent version of the libraries (v7.3.24b) you have to set things that were once defaults in previous versions. This isn't horrible, and the fix was simple - but figuring out what needed to be done was the problem. Thankfully, the EQSVal guys were helpful and once I generated them a test datafile they were able to tell me the issue and what to do.

After that, I had to make several changes to the web-based editor for my server. Nothing major, but there were some edge conditions that were showing up as records in lists that should not have been there. The perl processing and filtering was about as good as I could do in a single pass, so I had to add a second pass on the data to further clean it up. This will take a little bit more on the execution, but it'll keep the records cleaner and therefore make it less confusing to the users. Good things to do.

Finally, I got the most recent version of a driver from the internal price source folks at the Bank. I needed this because their mapping rules from Reuters to what we needed weren't capable of being exchange-specific enough to really hit the points we needed hit. With this update, my contact Todd is able to get the rules as specific as necessary and that's great. I'm hoping that these work out in test over the next few days so we can roll them out to production as soon as possible. These are tickers that the users have been wanting more smarts in, but haven't been able to deliver due to the limited rules on Todd's side of things.

That's about it. It's been a monster of a day. I'm looking forward to calming down and taking it easy for a bit.

Today I finally figured out the problem I have been slugging out for a few days. Actually, there were a few - none of them of my creation, but legacy issues that I have to deal with in order to get the server making a little forward progress.

The first was the fact that the Bank's valuation library, EQSVal, deals in dates that are ints and I had assumed that the original author of the conversion code had checked to make sure that the conversion was done properly. After all, the data looked right. In C/C++ the standard for converting dates to ints is to compute the number of days since 1/1/1970 - the epoch for all Unix systems. And that's what the server's conversion code was doing - converting the dates based on that reference date. Interestingly enough, the EQSVal library authors have decided to reference the same date as Microsoft Excel. Why? I can't possibly tell you because it makes no sense to me, but that's not the point, I suppose. The point is that they reference the first day of the last century - 1/1/1900. This made all my dates off by 70 years - some 25,567 days.

I saw this problem and realized that the easiest thing to do was to make a new method on MMDate called daysSince1900() so as not to confuse the other uses of the method toInt() that got the date with respect to the Unix epoch. This fixed the first of the two problems. I'm convinced that when I was trying the latest production version of the EQSVal code I was getting what they classified as operating system errors most likely from doing the reverse of what I was doing - subtracting 25,567 from the '1900' date and then using the Unix tools to then trying to convert it to a Unix time struct. Getting negative numbers might really be a mess, and could easily have been the issue.

So... one down, one to go.

The second problem I found was the quote date for the stock 'products' going into the model. The original code had the driver date - which is the quote date offset by the settle offset for options. This was a typo in the code and was leading to problems in the model. EQSVal checks the volatility curves (volatility, skew and kurtosis) to see that the first point in each curve is not before the quote date for the evaluation. Because the stock product was using the driver date, and the first point in each curve was only a business day away, it was advancing over the start of the curve and it was flagging an error. Most of the time this would not be an issue, but near the expirations this is going to happen as the volatility curves are calculated on these expirations.

With both of these problems solved, the values were calculating properly and no flags were being raised. Good enough. Not easy to find, but I'm glad I stuck with it before I left for the weekend.

UPDATE: it turns out that the EQSVal libraries allow for me to specify the date of 1/1/1970 which they use for offset. In general, then, the difference should not have mattered, had we set the offset right. For now, we'll leave it, but it's nice to know that there's a place we can control the offset itself.

There are those days that make you love the Job. There are those days that make you hate it. But for the most part, it's just that - a Job, and there's nothing particularly exciting about it and nothing particularly aggravating about it either. Today is one of those days.

I'd rather be doing something other than what I'm doing - which is debugging other group's libraries in my code without their source or support. It has to be done, because there isn't a real alternative, and it's not horrible because I have decent ways around most of the limitations I've found in their code, but it's still just plain old grunt work. Nothing fancy or glamorous, but the bread-n-butter of the developing job.

It's not fast because a significant change requires a restart which takes about 20 mins, but I try to make those significant, and therefore informative. It's going to get solved, it has to get solved. But it's not really something I'm going to put on my resume. It's just the job.

Today, actually yesterday evening, a trader had a problem with some of the data my app was showing. Now I don't generate this data - I just display it, but as is often the case, if it's connected with your app, you get at least partial blame for it not being right. I understand this and it's something that, to a large degree, I agree with. The problem comes in where the guys feeding me the data won't (or can't) fix the data coming to me in a timely manner. Then I have to get into two modes: Defensive Systems Development and someone else's debugger.

Today was one of those days.

After coming up with a plan to avoid the problem in the short-term, I set out on the task of making my system capable of filtering out the bad data from the upstream source. It wasn't something I was planning on doing today, but it wasn't bad and allowed me to add another level of flexibility to my flagship middle-tier. Not bad really. But in the middle of trying to get al these changes in for stopping and starting the flow of updates from my main collector/distributor, there were a flurry of emails about the upstream system - a database, and trying to find the problem there so that we stop it at the source.

I'm all for fixing things at the source, but I soon realized that they weren't seeing the problem. So I had to just into the fray and try and figure it out myself. I'll admit it was a tricky little problem in a stored procedure. Basically, a table which originally presumed to have only one day's worth of data in it was having, for a few hours, two days worth of data. This wasn't bad because when they changed to making it have multiple days of data, they changed the stored procedures to look at just one day's data in that table. Or so we thought.

In actuality, it was not looking at just one day's data because the where clause on the defining cursor was not limiting the primary table's data to just one day. This meant that for a time, two day's data was being used in the cursor. This would have been fine, if it not for another bug in the storied procedure that didn't limit an update statement to a single day. Now the bug is evident - two days were being looked at in the cursor, both days were being updated at the same time in the update statement, and one of those days was not really complete.

That meant that the incomplete day was getting used and updating the complete day and that was the bug. Once I figured that out, I told the manager of the project, and when I had convinced him, told him what he had to do to fix it. He then had one of his guys fix the stored procedure and we should be fine.

I still finished my enhancements to the filtering on the data sources because I think it's a good thing to have. But it's probably no longer necessary. We have it fixed at the source and that's going to be all we need.

Today we had a meeting with a group of technical sales consultants from a Vendor for a very expensive package that we have purchased and I've been asked to integrate and get working. This meeting was about price feeds into this system, and how the current product's abilities are weak to say the least. They rely on Reuters exclusively, but with a Reuters infrastructure, you (the client of Reuters) can push data back into the RMDS feed, if you so choose. You just become another 'exchange' to Reuters. But that's not a realistic alternative for us.

What we need to do is to get at the Vendor's price feed service and inject prices from my CacheStation based on our rules. These prices are what's necessary - as opposed to what's possible from Reuters, and so I'm not really in favor of simply faking Reuters RMDS with our source - I want a new source that can co-exist with the Reuters feed and then we can configure the application to take certain prices from one feed and other prices from another. Sounds very reasonable.

What's amazing to me is that in this meeting we had close to a dozen people - of which two were really necessary - the lead Vendor representative and myself. Everyone else was at least redundant, and often (as it turned out), painfully in the way. I've worked with the feeds, I know what needs to be done, I understand the basics of the Vendor's price service, and all I need to do is to get them to do a few things to meet me half-way on this and we'll all be fine.

But as it turned out, people that had no idea what was needed or what was happening had to be in this meeting. Why? Politics. What a waste. This is something I hate - don't waste my time if you're going to have people unrelated to the issue blathering on about this or that... get to the point, or confess that you have no idea what is going on and get on with it.

In the end, you'll come back to me - or you won't. If you don't, then you got it figured out without me - Good for you! If you come back to me (as you have time and again), then it's because you have no clue about what to do, and you want me to Just do it.

And it seems this insanity - I can think of no better term for it, is going to continue before there is any hope that it will get better. Tomorrow we're having a meeting about the first hardware purchase to run a version that they've never deployed and don't even have finished. How can they possibly know? It's a guess, so let's guess and move on. Chances are, we're not going to underspend - this is a massive system and it's going to need more than we're throwing at the problem now. But they have to have these meetings with people that have no real clue about what they are doing. It's amazing.

Someday this will end... it'll end the day that they realize that this has to work, or it's a waste of millions of dollars. When that day comes, they have traditionally come to me and said 'Make it work!', and at that point, the meetings stop. Unfortunately, that's typically ridiculously close to the drop-dead date for implementing this thing, so I'm under a lot of pressure to get it working in a very short time.

But that's the day it'll end. I just wish I could sit all this out until that day. It's the kind of thing that gives me a headache - and reminds me why I don't sit in the manager/owner chair any more. It's just no fun.

I was sitting here today working on the bulk push() and pop() for the BKit queues and a fellow developer stopped by to inform me that I wasn't using the best performing locking schemes in my implementations of the queues. While I accept that there are possibly ways to squeeze a few percent out of mine in certain conditions, these aren't custom-designed queues, they are general purpose utility classes that can't make assumptions about their use-cases in their implementations.

But that wasn't at all clear to this developer that stopped by my cube.

And that wasn't the only thing he didn't get.

Good Java coders are first and foremost good developers. Period. That means they understand how the code is executed in the machine. Where things might seem to be unimportant, or for that matter, important, and where they really are.

Case in point about high performance queues. If you're in a process where you have many threads processing data off a single queue, and the processing time is small, then you are guaranteed to have locking contention on the queue - by definition. There's no way around it. The solution I've seen time and again is to remove things from the queue in bulk and then the processing time goes up and the locking contention goes down because the number of threads hitting the queue at the same time goes way down.

But to this developer, the answer was Java 5's optimistic locking. Yes, folks, assume you'll get the lock and then fail and retry if you don't. Well, that's a wonderful idea in a situation that's already got locking contention problems. What was he thinking? The answer was, he simply wasn't. He was thinking that Java's way of doing things was so easy that under the covers it really didn't have to do any low-level locking for the push() or the pop().

So I had to walk him through it. It finally dawned on him that his statements were crud, and when he did, I could see him deflate right in front of my eyes. But for those 15 minutes when he was sure I was missing the point I wanted to yell at him Are you a total idiot? Don't you know any language other than Java? And folks, unfortunately, the answer to that is No. If you don't have any more experience in developing than Java, it's hard to see that it's not the end-all-be-all.

Today I ran into a problem that's somewhat sensitive to me. Because I'm sitting in this cube, and could be doing a lot of other things, I realize that there are certain trade-offs I've made to be here. I'm not in a managerial role, but had I wanted to stay in one, teaching or owning my own company would have afforded me all the opportunity I wanted for that. No... I didn't want that. I like doing, not deciding for a living.

But every now and then there comes along a situation that makes me rather upset with the people I work for. I'm capable of doing the management, and as such, require very little management from above. My management appreciates this, and when it suits them, take full advantage of it.

Today I wanted to get licenses for Tibco for several projects we're working on. This is a replacement of either free or home-grown messaging systems, so there's no obvious feature that this will bring that we don't have now. Yes, it'll be soft factors - industry standard, higher performance, scalability, etc. but today - right now, there's no problem that this addresses. Yet it's a $100,000 expense. Too much for casual spending.

So rather than trust me that it's needed, and will be useful, they want me to make the drawings and slides to show how it'll fit into the next phase of our architecture. If we had run into a problem, they'd never want to see this... they'd want to know what fixes it. But because there's no immediate problem to solve, it's not as important as other things.

Understood... then let's just drop it. I'll wait until it's critical again and then I'll remind you that I mentioned this and you'll say "Well... why didn't you... Oooo!" and we'll do it then. But now, I have to do this for a group of people that either really don't care, or really don't need to care. It's infrastructure stuff, folks. Why spell it out to the web developers how the trade processing will change with a new messaging system? They might care, but when they need to know, they'll ask.

I guess it's the fact that if they thought it was too expensive, just say so and we'll limp along until it's critical and then we'll say "Here's the solution", and get it then. Doing it this way is just part of the reason that I don't own my own company any more - I don't like doing this stuff. And for the vast majority of the time, they don't want me doing this kind of thing - they want me fixing problems or enhancing the products I'm responsible for.

But I have to go through the whole dog and pony show, and let them take it to the Governing Council and then have project charts made, etc. It's turned from a simple little replacement project that'd take a few days to something that will drag out for months. Yucch!