What's it all about, Alfie?

This morning I got a visit from Ammo and he's guarding my desk now when I take a break. It's nice to know that I've got an angry amoeba with an automatic weapon looking out for me.

Today another developer stopped by with a problem I've seen several times before - how to handle user validation of percentages. Specifically, let's say a user needs to put in how to subdivide something - 10% to this, 20% to that, and 70% to the other. Breakdowns like this are easy - until they get into factional percentages and numbers that are, by their very nature, impossible to represent exactly in a computer.

Take 0.12 - it seems easy enough to look at, but try to add it to 0.74 and 0.14 and you're going to find that a double is not exactly 1.0. That's because the numbers are not exactly what we typed in and there are representational errors in the doubles that make it something like 1.00000001. So how to fix this?

One way, which I've done for many cases, is to have an ε something like 10^-6 and check to see if the value you want is some ε from the target value by:

    if (Math.abs(sum - target) <= 1.0e-6) {
        // close enough to be considered equal
    } else {
        // not equal
    }

which can be put into a method or function that makes it easy to call. But the idea is that if you get close enough, you're equal. The problem with this is that the ε that you should use is greatly dependent on the values you're summing, in this example. Say you have values like 10, 20, and 70 - in this case, ε should be 0.1 because there's no need to have it any smaller as all the numbers are integers. But if you had numbers like 10, 20.22, and 69.78 then you may need to have ε set to 0.001. The optimal value of ε seems to be directly related to the maximum precision of the data. Also, there's always the possibility that some oddball case will be just large enough a difference to not be 'equal' and then you're flagging a false positive. But there is an interesting way that this won't happen.

Look at the problem of the user input as if you were a third-grader. Break the problem down into the whole number and fractional components. This makes everything an integer, and adding integers is very easy indeed - with no rounding problems. What I mean to say is that if you had the set of numbers: 10, 20.22, and 69.78 then break the summation down into 10 + 20 + 69 + (00 + 22 + 78)/100.0. This means that you only have to look at the string representation of the numbers - separate out the parts to the left and right of the decimal, find the maximum precision to the right of the decimal (for scaling), and then go to town.

After we talked about this approach for a little bit, the developer went back and coded it up in no time. It's really very simple. But given that users can type in arbitrary precision in the numbers, it's important to sum them as exactly as possible, and using integers for the whole and fractional parts is really pretty sweet. There's no rounding (until you get to the final double) and by then, you're comparing it to the test value, and if it's another integer (in the case of percentages), then it'll be easy.

I giggled a bit thinking about this. It's not often that the best way of doing something is the way a third-grader would do it. But that's the point - not everything needs to be so overly complex and over-designed.

Today I had to spend a lot of time on a application that got dumped on my lap after the last developer working on it couldn't get a new price feed hooked into it and was asked to leave. Since I had done a lot of work on this price feed, it seemed natural that I get the app hooked up to the new feed. It took me a few days and worked great. Then there came the need to add in permissioning for fee-liable exchange data, so again I was the logical choice because I had done similar work for other apps. Ahh... that slippery slope.

It didn't take long for me to be the maintainer of this app. That typically meant that features that haven't worked in years suddenly working because I found mistakes in the code and/or database and then they magically started working again. Some I was specifically asked to fix, others were resurrected completely by accident when I was fixing something else.

Now I've been working on this code for about 9 months. Not full-time, more on an as-needed basis, so I don't mind too much. But I need to say that this code is a real mess. I mean the worst code I've ever seen.

It's primarily C++ on Windows using Visual C++ - but the build systems is GNU make on Cygwin. It uses JNI, so the JDK is involved and that's a mess. There is virtually no documentation, there are dozens of sub-projects. The database access is through a separate process on another machine that has it's own configuration, it's asynchronous and not easily followed. It's over-designed and poorly coded. In short, it's a mess, and it'd be easier to re-write it but the logic is so buried in the code that it'd be easier to sit down and talk to the users to see what it's supposed to do as opposed to how it does it.

One of the features of this app is the ability to create new views (browsers) on the data. This controls the columns shown in the table, their order, how the view is updated, etc. Yesterday one of the big users of the app asked me if it would be possible to make it so that you could create new browsers. The app looks like it's supposed to work, but it doesn't. So I said I'd look into it and see what I could find out. Yikes! what a mess.

The code was a mess and had been that way since 1999 - I found comments where they had hard-coded the name of the new browser to 'x' - literally an 'x', and it was commented as happening in 1999. What were they thinking? I have no idea. I tore that out, figured out what was being passed into the method by way of these odd notification objects, and realized that it wasn't that far from working - on the client side. It took less than 50 lines of code changed, but finding those lines was the challenge. The real problems were on the database layer configuration, and stored procedures.

When moving to Sybase 12.5 the IDENTITY column had to change from an 'int' to a 'number', and in doing that, most all of their stored procedures were broken and never fixed. The access layer expected ints, and they weren't being returned that way - so basically, all the database work was getting dropped in the bit bucket. It was similar to what I'd seen before in this app, but what a striking example of poor updates and testing.

Then there's the bad UI, but that's just because it was created by so many developers over so may years there's no way it was going to have a consistent vision and style. Horrible. Which brings me to the point of this post - the code is ancient. It's been worked on by so many developers that had no idea what the point and architecture of the code was originally about. It's been slapped, hacked, poorly tested and given unworkable hard-coded hacks. It's amazing it still compiles. But this is what some people call developing. I can't imagine putting my name on this if I'd done it. I'm happy with the fixes I've made, but those are only in light of how bad it was to start with.

It's just amazing it works at all.

Yesterday afternoon one of the developers came by to ask me about finding historical prices for delisted symbols. "That's hard", I said - because it is. Bloomberg has historical data back some 20 years, but you have to know their symbology to get it out of Bloomberg, and one of the things my MarketData server does is to allow us to use our own symbology and it converts this to the symbology of the data provider (in this case, Bloomberg) and then requests the data. This is great so long as you can do the symbology mappings. The problem comes in with delisted symbols - they aren't in our mappings table so I can't convert from our symbology to Bloomberg's. This means that eventhough you might be able to get at the data, you don't know what to ask for.

I told him he'd have to investigate another data provider and get the data from them. This other provider needs to have the historical data and some sense of the re-use of tickers so that given a date and a ticker it knows that there was one - and only one instrument for that ticker on that date. This kind of historical data is not common, but it's available. You just have to know where to look.

So we started looking.

There's a project in the Bank that handles a ton of historical data from a lot of sources - one is Bloomberg. So we did a few tests to see if this source - which was in my MarketData server as a provider, had the data for a delisted company. Turns out it did. But at the same time, I found that there were problems with how I was calling and using this data source. So I dug into fixing them.

The first was pretty easy - when I requested data of this source, if I received nothing, then I assumed it was an error. In reality, if there's no data, they returned nothing. So if you asked for a delisted symbol after it was delisted, you'd receive nothing. I fixed those tests up to allow for nothing to be returned. Easy enough.

The second one was more interesting. The data fields you request should stay the way you request them - case-wise, but I wasn't preserving the case of some requests and mismatching the case of others. So I added several nice little methods to CKString to allow the user to copy a string and uppercase it (the only existing method uppercased the string in-place) and then I added equalsIgnoreCase() like Java's String to allow the code to not have to worry as much about case in the tests. When I put these in and realized that I needed an uppercase copy of the fields for getting data from the source, caching it, and then using the original field list to pull it out of the cache, things worked beautifully.

In the end, it was a lot of fun to add these things and see how they simplified the code I was trying to write to fix the problem. Lesson learned: add power to the underlying libraries and that will make any higher-level fixes much easier.

I was talking to a vendor consultant that is here at the Shop helping us integrate their product into our systems. Happens all the time, we tell them what we have, they know what they need and we work together to hook up all the hoses and pipes so that their product fits as smoothly as possible into our systems, and in the end, becomes yet another one of our systems. So I was taking to this consultant letting him know what I did, as his product had to fit into my systems in a very smooth way.

We talked for about 2.5 hours... him asking me what my products were, how they worked, where the data flows were - nothing unusual. But then he asked me how many people were working on all this. I told him that there was a Team of seven on this part, and a few guys on this part, and there's a Team in the other part of the Bank for this part, but the rest of the whiteboard was me. He was shocked.

Which lead him into the question of what makes a really high-performance Team. Seeing as how I'm here to make this integration project succeed, I humored him and let him lead the conversation. His background, it seems, is in leading large-ish group efforts within the very large vendor's suite of applications. He said he was constantly trying to figure out what made Teams really special. Why did some groups seem to just walk in the light, and others were stuck constantly trying to stay off the Dilbert comics.

His take was experience and clarity of focus. He believed that if you had really experienced guys that knew the problem domain, who could look at an issue and easily distill it down to it's elemental key components, then that Team was going to be a winner. And the problem was having to deal with all the customer support and that meant that lesser talented developers had to deal with those issues. That lead to the problem of morale, and that lead to the fact that in any vendor group, 20% or less to the work and 80% are always the focus of the manager's efforts to make them productive.

I told him I thought he was all wrong. I also told him that if he had 80% of his developers that weren't contributing like the other 20%, he should fire them. And I went on to tell him why.

Maxim #1: People Rise to the Level of Your Expectations - as an educator both in one-on-one sessions and in small and large classes - both in the lab and the classroom settings, I believe this with all my heart. If you set your expectations high, people will achieve them. You have to do your part - you can't make them seem unattainable, you have to assist those willing to learn to get the knowledge, and you have to be fair, honest, and communicate these expectations, but if you do that, then people will rise to that level. I've seen it happen so many times, to me it's a universal truth.

So, for the manager that is trying to get the best out of the 80% - he needs only to expect the same from them as the others, and if those expectations aren't met, then those people have to leave the Team. Maybe they don't need to leave the company, but they need to know that they will not being doing development (cool work) with this group because it simply has expectations of each member that are beyond what this person is willing to do. In the end, you will loose a few, but the number won't be large, and even if it is, the Team will perform better because of it.

Maxium #2: Seek those with Commitment, and Everything is Possible - while many may disagree that this is true I'll put it another, less controversial way: If you find people that can acquire your commitment, they will be able to acquire anything else they need to see those tasks through to their ends. Education, experience, assistance - all these are available to the person with commitment. The consultant I was talking to spoke of experience and the ability to have a clear vision of the task at hand. Those are two qualities of a person with commitment, but those two qualities are not by themselves a complete indicator of commitment.

You can have people who are brilliant and do nothing. You can have people who see, and do not do. True, it's most often the case that people who can do these things are doing these things not because they are naturally good at them, but because they have worked to be good at them - natural talent or not. So while I understand what the consultant was saying, I think he was looking in the wrong place for it. Look for the traits that create and inspire the effects, as opposed to looking for the effects.

And this brings me to the core of the issue: Character. If you have it, you'll go far, and a Team with a critical mass of it will infect the others that might be a little low, and the result will be a Great Team. I've been on Teams where I'm the weakest person, and I worked very hard to not be the weakest person. I strove to be the person that could be relied upon to carry the day, if needed, but most times it was just a friendly competition to see who could do better - today. Tomorrow, the race starts again.

OK, this weekend we have our disaster/recovery testing and while I've been ready for a while, several people are using today - yes, the last 24 hours before the test, to get things "set up" for the test. Now to be fair, there are some applications that don't need a lot of preparation, or there may be very limited expectations for some groups. But when those folks that decide to "check on things" at the last minute start to hammer me with requests for my systems to make their disaster/recovery testing go more smoothly, I don't really like it.

I have to agree that a lot of these are good things - mostly edge conditions related to what might happen in a disaster. And I have to say I'm happy to get these changes in no matter when, it's just that it would have been a lot nicer not to have to worry that I can get them fixed this afternoon, and had - oh... I don't know... maybe just another 24 hours... to get them in, tested and to production.

Alas, there are people that like to do things at the last minute. I, however, am not one of them.

Well... as I thought it might, the read-only copy of the instrument master database failed on me last night and while the primary is working fine, I feel it's necessary to be able to find a test case, or condition, where the replicated database fails so that I can give this to the team working on that project and they, in turn, can fix the underlying issue(s). I'm sure the local database admins will be be involved, as they have to be as we don't have that level of control over the servers and the machines. So, mauled by the sharks (from my previous post) I go back into the water trying to find the test case that will highlight the problem.

Last evening, the server was restarted at 5:49 pm, and the symbol set was divided into four groups of 889 underlyings and all four were sent out to the database proxy for loading. Typically, all four will finish within a few minutes of each other, but last night the first one finished at 17:55:12 and the second finished at 17:55:50 - but the third and fourth never finished. When I reconfigured the server to point to the primary, the four finished within 4 mins of each other - as they should. Clearly, there was something with the replicated database that was causing two of the loading threads to sit there waiting for data to come back. The question is, how to reproduce this?

It gets more of a quandary when you take into account that my development server started at 7:00 pm local time and it was fine using the read-only database - all four of it's loading threads finishing within a few minutes of each other. So there's something that's happening to the replicated copy between 5:50 and 7:00 pm that caused this problem, but it was gone by 7:00 pm.

I have a simple web page on the server's editor that allows me to look at the database operations that are being done in the code to see what the data is in the database and what's being retrieved. This has really helped a lot in the diagnosis of database issues like bad prices and missing key values. Yesterday, when we were having problems with the replication and the prices, I did have a few times when this page would not return all the data. Because it's a Perl script, it'd return what it had processed, but it would still act as if there were more to read (because there was), and yet nothing would come back. I'd love to be able to reproduce that for the guys.

Unfortunately, I haven't been able to. I have no tools at my disposal other than the requests I make. I'll keep hitting it throughout the day, but I don't hold out a lot of hope that this is going to point to anything conclusive. This leads me back to the same spot I was at yesterday - do I trust it? Today, however, the answer is different: No. I'll trust it when I have to trust it and not before. Since no one is really as concerned about this as I am, I'll stick to using the primary and see where the chips fall. There's no reason to risk production outages when all I've got to diagnose the problem is a few data loading scripts.

Interesting thing happened in the last 24 hrs. at work. We have a nice instrument master database - it holds all the instrument data as well as marks and such. Very nice. It's so nice that it's replicated to a read-only copy here in Chicago as well as a replicant in London for disaster/recovery purposes. It's great - except when the replication fails.

That's what happened last night. Normally, there's an explanation sent out about the "why" of why the replication failed. Changed tables, duplicate records - something caused the replication that normally works just fine to fail. But today I didn't hear what the cause of the break was. It appears to be fixed, but if we don't know why it failed, why can we assume that if it looks OK that it'll stay that way?

No reason to think so in my mind.

So when I ask the guys whose responsibility it is to maintain the database(s) "Is the read-only copy OK to use tomorrow?" and get a simple answer like "It should be good for tomorrow" I get a little nervous. What was the problem? Are we sure it's OK? I'm feeling like I'm on the beach and they're saying:

"Sure, Bob... go back in the water... pay no attention to the fins circling..."

"Oh, are we going in? No, not right now, just ate, don't want to get cramps... but you go right ahead... and splash a lot... here, hold this pork chop."

But since they are the ones responsible for the database(s), I'll listen to them and if I walk out of the surf a bloody, dismembered ghost of myself I'll say "Hey! You said it was OK!", and if there's fallout it'll be up to them, and not me, to answer those questions.

UPDATE: Well... as if I might have seen it coming, the read-only database was not ready for prime-time and I got the call at home after my server stalled on the restart due to waiting on the database. I switched it to the primary database and all was fine. Guess that means I'll have to figure out why.

This morning I got an email from one of the Hong Kong users about problems they were having with the theoretical values on OTC options in the server. While I didn't really get the proper picture from his email, the follow-up from a user in London provided the proper illumination to see what the problem was. Basically, when changing the volatility and/or dividends curve(s) for an instrument, the 'Open' greeks would be calculated properly and the 'Last' would not.

I dug into the code and saw that the 'Open' greeks are calculated no matter what, but the 'Last' are calculated only if they aren't in the cached data for that instrument already. This was the key - that the cached data was all wrong because it was generated based on the old curves, and with the new curves, all the cached data needed to be invalidated and recalculated.

Once I knew what I needed to do, it was just a matter of putting the code in-place to allow me to clear out the cache at the instrument level, then the underlying which would clear out all the derivatives' caches, etc. Then I had to put in the code to detect the change in the curves as read in from the editor, and putting it all together was pretty easy.

Once I had the code in-place, it was very easy to test and see that we are indeed getting the right greeks for changes in the curves. Also, I fixed a few issues in the editor so that it would not send the curves back to the server on an edit unless they had been edited by the user. This is going to help in simple efficiency as well as not making the server think things have changed when they haven't.

I had a new developer come up to me today and ask me if it were possible to remove a group of rows from a BKTable, and after a little bit of a sync on the terminology, it was clear that he wanted to delete a group of rows from the table based on some criteria. I was thinking about this and it seemed like a good idea even-though there's no way to do it now. So while he had to stick with iterating through the table and removing each one after testing it for it's applicability in the final results, I decided it was something I wanted to ask Jeff if he'd use it enough to add it to the class. Turns out, it's something that he'd like to see too. So I spent a little time today putting that into the BKTable.

I went with the idea that you'd provide this a JEP expression and it would either remove the rows that matched this expression or remove every row but those that matched this expression. Since I could use a lot of the same components that are in use in the filter table view this wasn't too hard. In fact, the effect is very similar to the filter table view, but in this case the removal is permanent and you can easily choose either set to remove.

After I got this in and tested I talked to Jeff and let him know that it'd be OK to have a list of a few things they wanted in BKit. He mentioned that his guys are suggested to come talk to me, with suggestions/problems so I guess there just aren't a lot of problems. OK.