What's it all about, Alfie?

If someone asked me what I felt was the most important thing to have to be a successful developer, I'd say it was determination. I was beating my head against these production configuration problems today and when I finally got them figured out, a new problem of a JavaScript menu bar popped up to bang at until I got it working.

In the end, the JavaScript menu bar I was looking at didn't really work as nicely as I'd have hoped. It worked, but the style was just too different from what we'd been using. So I re-organized the menus and that gave me the room I needed to make the page render nicely.

I'm pooped. There's not doubt about it. But as with all the other things I've been tackling lately, it's about not giving up. Working and working at it from different angles, from different ideas, to come up with some little crack that you can exploit to get the system to do what you want. It's just plain effort.

I've been digging into this problem of 'noisy' data coming out of one of the systems I've inherited here, and at first I thought it was a problem of loss of significance due to subtraction, but that turned out to be part, but not all of the problem.

Today I was digging again, trying to find out what was possibly causing this. I was looking at trade flow... and greeks being sent in... nothing seemed to explain it. I talked to the original developer of the system for a while and together we didn't some up with anything.

I left thinking that the best I was going to do was to shotgun the problem and hope I hit something that might be close enough to the problem to point out the solution.

I sat down and started to look at the code again, and then it hit me. At least I think it hit me. I have a very plausible reason in mind, but in order to know for certain, I'm going to have to reconfigure some machines and run them tomorrow.

Basically, I think we're getting into a situation where the multitude of portfolios are feeding data into a single collector for P/L calculations. The problem might be that because several of these portfolios are normalized to the same contract, we're get an increased feed rate on that normalizing contract, and that is upsetting the calculations.

My limited understanding of the messaging of data might be off. That's why I need the isolation test. If the data from my test is smooth, and the data in Test and Production is noisy, then we'll know that we have a solution to the problem. The wrinkle here is that this solution is going to mean that we need to have a lot more hardware to run these portfolios than originally thought.

We'll have to wait and see tomorrow.

Today has been spent primarily on getting the PRIVMSG capability working on this custom-developed chat system. Because of the nature of the system, that is not allowing private chats, it was a lot more difficult than it needed to be. While I might have done things differently in the implementation, I wasn't involved and this system is in place and there's very little interest in really looking at this critically. It's "done", it "works", let it go.

I had to work with what I had.

That included the sum total of the code. It was a combination of C#, Python and assorted scripts and was virtually undocumented. It was a mess. But as far as messes go, it was something that I could dig into and get a few hints.

My primary debugging tool was the log of the socket data coming from the IRC server. It was literally invaluable. In the end, this is the interchange I discovered I needed to use in order to get person-to-person chatting, and therefore code-to-person chatting, working.

First, my code had to issue a special command to a particular "Overlord" bot that controlled the users on the IRC server. This in and of itself was interesting. They chose not to implement the rules in the server, but rather in a Bot that existed alongside the server.

When I send the appropriate command to this bot, I need to include the nick of the user I want to talk to. The Overlord then creates a channel just for the two of us with a special prefix, and the two user's names separated by a special character. Both users then get INVITEs to this new channel.

I have to wait for the INVITE, parse the channel invitation into the two user names, see which one is me, and which one is the "other guy", and then cache this data so that when I want to chat to the particular user, I can look up the special channel for this guy, and chat there, instead.

There were a ton of wrinkles with this scheme. First, because the process is asynchronous, I have to buffer my chats to this person if I need to create this channel. That is a pain, but doable with a little thought. Once I get the INVITE, I simply see if I have any buffered messages for this guy, and if I do, then I send them in order to the (newly JOINed) channel and we're back up to date.

Another one was the reconnection scheme - I simply took the point of view that on reconnection we'll do the minimum and each operation will ensure that everything is set up for that operation. It works, but it might be doing a little more "lazy" set-up than a different approach that would cache the channels, etc.

I have to say that everyone I've talked to in this place about these technical details is amazed that it has been done this way. The simplest way would have been to log everything in the server. Period. Database or flat files, or MySQL (combination of the two) would have worked. Then add in a simple authentication method on the server and you're done. Leave private chats as-is, but log them. Done.

It wasn't done this way, and I'm not sure what's going to happen in the long run. I've got this working, and that's the most important thing.

I've spent the day working on getting a decent chat interface into a custom-developed chat server that a vendor wrote for us in response to the compliance regulations. I certainly am no stranger to this, having worked with MindAlign in a previous position. These are all chat systems where the logging and authentication is such that you can't spoof being someone you're not, and all conversations are logged. It makes sense for a place moving money.

Problem is, the system we're working with here (which can remain nameless) is really pretty horrible. First, there's no documentation on the protocol at all. None. Zippo. There was nothing in the code either. I could not find a thing to help me.

Thankfully, someone else had been working on this and realized that there was an XML file for the client that had what was needed. Basically, a bot would send a challenge PRIVMSG and you had to respond with the proper response PRIVMSG to this bot. If not, the bot would kick you off the server. Effective, if a little trivial. There's no server-level authorization so that's a hole, and there's no changing of these challenge/response phrases so that's not very secure either.

But after this is done, you can send messages to IRC channels without any modification of an IRC client. Pretty easy. Sending PRIVMSGs is another thing entirely, and I didn't get this solved today. Hopefully tomorrow.

But Boy! would some documentation have really helped.

There seems to have been no one at the time asking for this. I can't imagine what they were thinking.

This afternoon I had to deploy an inherited app again, and this time it was a little smoother, but the fact that database changes made in one site aren't also done to another really bit me in the behind.

I had asked one of the DBAs to build this stored proc for me, and he did a great job. What I failed to realize is that he stopped right there. He didn't put it on the London copy of the same database, assuming, I guess, that if I'd wanted that, I'd have asked for it.

Silly me.

Well... it was a little harried and hurried to get things in place, but in the end I put the stored procs where they needed to be. Lesson learned, Bob... make sure to do it yourself.

Today has been a challenge for me. I haven't had a really good day, but I have to admit, in the grand scheme of things, it hasn't been the worst either. It's been a day where I've been fighting my instincts to do "more" - a kind term for stepping on people's toes. More VM problems - I've just had it with VMs for web servers - I've asked for dedicated machines for the web servers (UAT and Production).

I've also run into a most 'unique' situation with chat. Seems we had a custom chat system created - based on IRC, but extended to allow for all the compliance issues that might arise in a trading shop. No surprise there.

No, the surprise is that we have no documentation on the authentication system at all. It "just works", but if it stopped working someday, there's not a scrap of documentation to say what it should be doing as a place to start to see what's going wrong.

This lack of design docs bothered me - I just wanted to be able to code it up so that my web app with alerts could chat them to the users. It's a simple thing, really. But you would not believe the kind of run-around I seemed to get.

Some folks just said "don't use it". Well... it's a little late for that - it's in production, not going away, and I need to chat to something. Saying it's horrible is a judgement call, but I still have to work with it. I just want to know what it's doing!

Several hours have passed and I've just asked over and over to talk to someone at the shop that built this. I just need to know what it's passing around and I can take it from there. It's not magic, it's code. But rather than try to guess looking at the code, I wanted a simple design doc - or even just a protocol explanation.

Hours later and nothing. I'm guessing this won't be over soon. I have no idea why.

It's just very draining. Very. I'm tired and I want to go home.

Today was a day where I was coding all day long. In the middle of the addition to the code so there wasn't a lot of design, and not really near the end where there were a lot of tests. It was a day where I was just heads-down coding.

There were a few tests in the middle - when I got to the point that I was able to send emails from my alert system. But those weren't nearly as interesting as the conditions I could check for were pretty simplistic and I knew there needed to be a lot more complex ways of looking at the data before I was done.

Not a lot to say... just coding.

Today I needed to get really busy on the next phase of my web server/services - intraday alerts. Basically, we need to have a system whereby it's relatively easy to configure multiple alerts based on the data we collect about the company during the day. The first test is a simple one - if the P/L of any portfolio jumps more than a $1 mil in one tick - either way, then it's something I need to know about because it's likely bad data.

You can think of a lot more business-useful cases where it's not the data that's wrong, as in this first case, but that is going to bring into focus the moving average code I wrote as well as a much more complicated set of rules than simple edge-triggering. So what I wanted to do was to make a system a lot like we had with InfoShop in my last job: basic tools all wired together, but not configured until they are put in place in a configuration file.

I like the Java Properties file format for this, but primarily in it's ability to dovetail nicely with the System Properties and overlay multiple files, one on top of the other. The idea is to have the property files specified in the web.xml file as a list where order matters, and the list is loaded and layered on top of the System Properties so that the complete configuration of any component can be parsed from this map.

Then, I'll have as one named property a list of names. These will then be added to the 'baseProperty' name - something used to keep the Property namespace clean, and these will then be parsed and created by the system. It's a pretty easy way to have a config file do all the building and running you need while not having it be as overly complex as say, Spring.

My initial file looks like:

  #
  # List all the named alerts to create
  #
  FC.Alerts=CrazyJump
 
  #
  # CrazyJump - send email on big jump
  #
  FC.Alert.CrazyJump.ClassName=com.chicagotrading.firecache.trigger.BasicAlert
  FC.Alert.CrazyJump.Portfolio=[*]
  FC.Alert.CrazyJump.AlertExpr=abs((pnlSumNet - lastPnlSumNet)*fxToUSD) > 1000000
  FC.Alert.CrazyJump.MailTo=robert.beaty@chicagotrading.com
  FC.Alert.CrazyJump.Message="Portfolio " + portfolio + " jumped"

And from this, it's pretty easy to see where the name is used, and that there are a few mandatory properties for each named alert - the ClassName for instance. But there can be a lot of optional ones as well - dictated by the class that is functioning as the alert.

This is where the BasicAlert comes into play - we needed to build it with all the capabilities a general alert would need. Like the ability to parse expressions... send emails... create messages. All these things will be needed by the subclasses that will be the far more interesting alerts to write with far more complex business logic.

Still, it'll be relatively easy. We have a simple init() method that will be called, and then an onUpdate() for each updated row to check. It's a pretty simple pattern that's been used a lot in Java, which is why I used it. There's almost no Java coder that won't recognize this when he sees it. Consequently, it'll be easy for just about anyone to pick up and make an alert.

It took me the majority of the day to get this to the point that I think it's ready to use. Lots of set-up for the uses I'm trying to see will need to be covered. Thankfully, I've done something like this a couple of times before and have a really good idea about what needs to be done for the vast majority of the cases, and a few little things that will make even the odd-ball case possible.

I'm tired, and that's the way I like it. Leave it all on the track. Hold nothing back.

Today I was faced with a nasty little problem. I've started to add the annotations to the Google Visualization AnnotatedTimeLine widget and when switching them on and off I ran into a problem with the way I had implemented double-buffering: the unused area where the annotations are in the Flash component has a transparent background, so given that the two widgets have the same size in pixels, the "hidden" graph that doesn't have the annotations active is going to "bleed" through the one that does, with it's transparent background.

Yuck!

How to fix it?

Well, the simplest idea I had was to place a curtain between the two divs in zIndex space. Basically, in the HTML I added another div with nothing on it and set it's zIndex value to midway between the other two: 25.

This wasn't working right, so I changed the CSS on the curtain div to have a background-color of #fff. This looked a lot better, but again, it didn't seem to work all the time.

I pushed and prodded on this... adding a non-breaking space to the div... changing the color to try and see what was happening... then it hit me: Maybe the zIndex was being changed?

I added the one line to reset the zIndex of the curtain to 25 every time I flipped the other two, and Bingo! That was it. It seems that at least in Firefox, the JavaScript/CSS was "moving" the "static" div in zIndex space when I moved the others. Odd, but at least I have it worked out. Sweet.

My web app caches the data it sends to the clients so that if another client asks for the same data while nothing's changed, I send them the same data and it's pretty fast. The problem there is that it's timed data, and that means that all the clients around the world have been seeing the data in Chicago-time. Not really what I want for a good user experience in New York and London.

So I needed to see if there was a way I could bring client localized time to the app without losing all the advantages of the cached datasets. If I passed in the location (region/TimeZone) of the client to the servlet, then I'd have a different cached response for each request in a region. Not ideal. Much better to be able to do it client-side. But that begs the question of how to do this?

One way would be to have the server generate GMT times for everything and then each client would have to deal with adjusting the data to the local time, and I almost went with that, but I decided to stick with the Chicago-based time and then add the GMT Offset in the package from the server.

If the client has the GMT Offset of the server, then it can see if it's GMT Offset is different from the server's, and if so, it can calculate the difference and apply it to all the times in the result set. Nice... for the Chicago users (the vast majority) there would be no change as the changes would be skipped, but for New York and London, I could update the data.

OK, how to do this?

Well... the first part is to generate the GMT Offset in the servlet. Since the offsets in Java and JavaScript have an opposite sign, the right thing to do is to standardize on one and then make the other 'fit' it. I chose to stick with the JavaScript version, so my Java servlet code looks like this:

  public UserInfo() {
    // don't forget to call the super's constructor - no matter what
    super();
    // start by getting the GMT offset in mins for this guy
    GregorianCalendar   now = new GregorianCalendar();
    if (now != null) {
      _gmtOffsetMins = (int)(-1)*(now.get(now.ZONE_OFFSET)
                                  + now.get(now.DST_OFFSET))/60000;
    }
  }

where my UserInfo is passed back to the client AJAX with some information about the validated user. Now it's got the GMT Offset in the same sign as JavaScript expects. Good. Now all I needed to do was convert the data that needed converting.

I created a JavaScript function:

  function convertTimesToLocalTime() {
    // get the time right now for the correct offset
    var now = new Date();
    var localOffset = now.getTimezoneOffset();
    if (localOffset != userInfo.gmtOffsetMins) {
      // get the soze of the table as we're going to need it
      var rowCnt = graphData.getNumberOfRows();
      // now let's run through the first column anc update the times
      for (var r = 0; r < rowCnt; ++r) {
        var obj = graphData.getValue(r, 0);
        if (obj instanceof Date) {
          // get the time from the server into UTC
          var utc = obj.getTime() + userInfo.gmtOffsetMins * 60000;
          // make this UTC time into a local time
          graphData.setValue(r, 0, new Date(utc - localOffset * 60000));
        }
      }
    }
  }

With this, I don't do anything unless I need to, and when I have to, I get the time data in the local timezone - regardless of where that is! I'm really happy with this as it adds a little overhead to the client, but it allows my server to not mess with localizations and keeps the cache intact. Nice.