What's it all about, Alfie?

Today I've been doing more coding on the SOD position file editor. I talked about it a lot in the post yesterday, but it's nice to be moving forward - even if it's slowly. The Java GUI tools are powerful, I'll grant you that, but they aren't something that you can use once every year and expect to be really proficient at much of anything. You have to dedicate a lot of time to getting into each class and how it fits into the collective whole to be able to code a good looking GUI without having to hit Google for every other feature.

Also, today is my anniversary - 22 years. Amazing. Going to try and have a little fun this weekend.

I've been in the Groove on adding the start-of-day (SOD) position file editing to the editor web for my server. Java GUIs aren't my favorite thing to code up because of the mastery of the object library to really get it done right. Thankfully, Google is there to help.

UPDATE: One of the problems was getting the columns sized correctly. And when I say 'correctly' I mean sized to fit the data in the column headers and column data. So that if you give it enough horizontal space, you'll be able to see everything without having to resize the columns each time you do something like sort the table by a column. The code turned out to turn off the auto-resizing and then go through each column, getting the renderer and asking it what it's preferred size was. Add those up, tack on a bit of margin and you have it. The code is in BKSimpleTable, if you want to have a look.

The next thing was getting the changes back to the server from the applet. I decided to make a cgi script that simply took the old and the new, and replaced one with the other in the SOD position file. It turns out that it's not that bad - but not really efficient either, as you have to scan the file for the matching line. But that's not horrible as we're not doing hundreds of these a day.

The next hurdle was getting the applet to auto-activate in IE. Turns out, IE requires that you call Javascript in another file in order to get it to work. Basically, make a Javascript function myPrintln(s) and in that simple call document.writeln(s); and as long as this is in another file, it'll work. Then, write out each line with this method. Seems like a waste, and it is, but that's IE compatibility. I do with they'd get on the ball.

In the end, things are looking good for a 1.0 release of the page. I've shown it to the Data Team and they will play with it and see if they can find any problems or additions that they need. I'm hoping that this makes the corporate action processing so easy that it's not a major issue any more.

Well... I spent a lot of time Wednesday and Thursday working on these improvements to the Server, but the more I got into it, the more I was convinced that it was a horrible idea - and it was my idea. The basic idea of the enhancement was to have the instruments in the server contain a business date, and then according to a configurable schedule, we'd roll the business date by region (Asia, etc.) and that way the instruments would pick up new SOD positions and 'look' like they were ready for the next business day.

But the problem was really that all the changes required for this to work would have made the server far less stable in the short-term, and that might be a public relations problem that would take months to recover from. Additionally, I needed to know the exact requirements from the traders in Hong Kong, as they are the ones pushing this. Do they need to see the positions for Europe and the Americas when they get in? Do they need to see the marks? These are things that are going to make a huge difference to the code.

In the end, I decided that it would be far, far better to make the server capable of reloading it's complete position set - as well as marks (which it already does), so that we can make a trial SOD position file and start a new server off that - ideally, London. Then, when the final SOD position file is ready, we copy that over to London, and reload all the positions into the London server. This means that the Asia positions and marks will be rock-solid, and the Europe/Americas positions will be close and the refinement will make them just perfect.

This solution has many upsides, but the one downside is the human cost of data maintenance. If we have two servers, then any editing done to one has to be done to the other as well. This just means that the effort I would have placed in the server now needs to go into the server and editor. Basically, all the editing operations that are now being done need to be made as simple and easy as possible. That way, the load for the data maintenance of one server goes down, and hopefully, the addition of the other is not a horrible problem.

There are several things to work out, but the number of systems that have to change in this plan is much fewer - only the Server/Editor. This makes it much more likely that the stability will remain high during the changes, which is a major benefit. We have to deal with the trade processing, but that shouldn't be hard - again, it's by region. Also, we need to have a trial SOD position file, and that should not be hard either as it's already capable of being generated by several systems at this time.

Not all my plans are jewels... this one is far better than my original. Far, far, better.

Today was spent working through a few issues on the conversion of the server from a nightly restart to continuous operation with multiple regions rolling over as the day progresses. It's really that latter part that's the pain in the neck as a single timezone implementation wouldn't be nearly as hard, but the problem is that we can't get people here to stop work early enough to allow the system to be ready for the start of the far east day.

Today was really just trying to get all the references to the global constants for time and date to be region-specific so that we can have n regions and not have to worry that there are just two - like there are at the present time. While I don't think there will ever be more than two regions, it's not a good idea to build in such a concept as it represents a serious limitation going forward.

Tomorrow will be more heads-down coding. Not a lot of time to sit up and look around, I need to get this code in place and working on a single timezone as soon as possible to check and see what the stability impacts are to the system.

Well... today was the first day of the real re-tooling of the server for 24hr operation. Up till now, it's been restarted each evening, but with the needs of the Hong Kong group coming so close on the heels of the end of the Chicago trading day, it was time to look at a more continuous Server. The first thing I needed to do was to extend the protocol between the server and it's clients to allow for something other than the 'instrument update' notification. I had to add in the capability for the server to send a 'remove instrument' notification when a new underlying is reloaded and the list of derivatives do not match those that used to be on the underlying, and so there's an instrument that has been removed from the server and needs to be removed from the clients as well.

This will happen when we have an expiration, or a split processing, where the strikes or expirations have changed and therefore the options will have changed. Right now, the server clients don't have any mechanism to remove instruments - the users have to zero out positions in the server, and then place new positions in the server. It's a hack, but born out of necessity. Today we got the instrument removal on a reload working so that the splits processing will be much easier and setting the stage for the nightly regional roll-overs where instruments will be dropped as a natural course of the week.

I also spent a little time updating the project web page to put in the overview of all the tasks I have planned out for the remainder of the project. It's a lot of little things, but thankfully, nothing seems to be too difficult that can't be solved in a day or two. Oh, it'll still take a month or so to get all this done and tested, but that's plenty of time before the DST switch this fall.

Today I got news that the plans I'd had for having MarketMash cover the Hong Kong user's open reliably were dashed because of the pokey nature of the Chicago folks in getting their trades into the systems. So I had to go back to the plans for a true 24hr Server and finish them. The first ideas I had for covering Hong Kong were to have the server reload itself without shutting down. This would be similar to the reloading all positions I'd worked out a little bit ago - something that used to require a recovery now is done in less than a minute. But the problem turned out to be the timing.

The traders and operations don't get done putting in trades until nearly 5:30 pm, Chicago time. This meant that even if I had a file ready for me then, the earliest I could have had it going was 5:31 pm, which is 7:31 am Hong Kong time - depending on the DST shift. So the hopes of doing it at 4:00 pm were out the window.

The next idea was to have the instruments classified into 'Regions' like ASIA and OTHER/LAST where they are in these regions based on the geographical region and the time we got marks and end-of-day positions set for them. So, late in the afternoon, we'd roll-over the ASIA region to tomorrow's business day and reload the marks and SOD positions. Then, a few hours later when the rest of the marks and positions were set, we'd roll-over the LAST region and everything would be ready for the next day. In the interim time, part of the instruments would be on tomorrow's date, and the others would still be on today's date.

This means that we'd have to make the server's clients (the middle-tiers) smart enough to know what region they are in and how to deal with the different regions and time of day. It's a bit of a mess, but it's the best solution I can think of, because the only other solution I can see is to have multiple servers with the same problem as nothing is going to be completely ready 24 hrs a day. That's just not the way this place works.

So today was spent doing a lot of thinking about the edge conditions and what would need to happen to each system for each case. Once it's all laid out, I'll update the project web site and the others can see what needs to be done and by whom.

This afternoon has been one of those really special moments where you know you're on the right track. I was talking to the Data Team earlier today and asking them if there's anything that they can think of that'll make the server's suite of tools easier, and they came up with the idea of automating the few tasks now done manually to execute a split. One of the first things they have to do is to zero out all the positions on all the instruments of the family. This really is a bit of a limitation on how the server's clients act - based on the fact that the server never "looses" a position.

So I started working on it, got it in the server, got it in the Java protocol, got it in the perl module calling the Java code, and then was putting it in the web site when I realized that this was really almost something that I've been meaning to add to the server for months. When something bad happens and the start-of-day (SOD) position file is bad, or the trades are all messed up, we have to correct the files and then recover the server. This is a 20 minute process that the users don't like to see.

The better solution is to enable the server to be zeroed out and then have all the positions in the new SOD file reloaded and then the trades replayed. This would allow us to fix the files and then say "do the BIG reload" from the client and the server would reload all positions and things would not require a recovery and 20 minute outage - it'd take 3 mins tops and the system would never be down.

What was amazing to me is that something almost totally unrelated comes back to be the key component in something that has been back-burnered for months. It's really something. And for a while, at least, I have a great feeling that I'm on the right track.

Certainly one of the nicer graphing packages I've used is the VantagePoint graphing library from Visualize Inc. It's 100% Java, it's fast, well designed, and very feature-complete. Sure, there are a few things I'd like to see them put into the product, but it's very few, and that's impressive. What I've been looking for is an Objective-C/Cocoa version of the same kind of thing so that the code that I'm working on in OS X can have stellar graphing as well. Specifically, right now, I'm looking at the heat/contour graphs in VantagePoint and wishing I had something like that in Obj-C for my Poission's Equation solver code - Potentials.

Potentials is meant to be a very simple solver of Poission's equation in 2D - and as such - be very fast. Once we have the solution in 2D, there's always the time factor which could yield interesting capacitance calculations if that was where we wanted to go, or maybe put in a little fluid and see charge distributed over time. It's something that might be very interesting to play with, but the first step is to be able to see the results in the code without having to run a stand-alone plotting package.

I came across Visual Data Tools, Inc. and their two products DataTank and DataGraph. While DataTank looks to be certainly more than I need, it's also much more expensive, and at $1195 for a license, that's a for a hobby project. And DataGraph looks pretty nice, but is lacking a few of the graph types that VantagePoint has - specifically, the 3D and 3D-mapped-to-2D like the heat and contour graphs. These are the ones that I wanted to use in Potentials, so it's bit of a bind. I wrote the developers to ask if there were these plot types in DataGraph and they mentioned that they were thinking of putting out a new version, targeted somewhere between DataGraph and DataTank, and that the new Bar graph would be in DataGraph soon. While this is good news, I had to write back and ask what the new product would have - I really need the different graph types.

I hope there's something coming that's reasonably priced and has the features I need and integrates well with Xcode. It'd be great to get this code to the next level and have the plotting contained within it. We'll just have to hope that they come back with good news.

UPDATE: it looks like the DataGraph developer is going to keep the low-end product without the two plot types I need for Potentials. I can respect that, but there's no way I can see dropping $1195 for this code... not without some proof that it'll make money, and I don't expect to sell it, so that's especially hard. So I sent an email to the Red Rock Software guys - they make DeltaGraph which has all kinds of nice graphs in it, asking if they had anything that would allow me to incorporate their graphs into my code. I didn't see anything on their site, but it's possible they are thinking about it - or have it. Who knows. It's priced as a nice, graphing package - $300+, but it's not the $1000+ of DataTank. We'll see what they come back with, but I'm not holding out a great deal of hope. I also looked at the VVI Vvidgets and while they seem to have the bases covered, it's more low level, and not nearly as clear-cut as the DataGraph or DeltaGraph tools.

I've spent a bunch of the day messing around with Java applets and jars and Firefox trying to get a page to work. Needless to say, I'm less than thrilled with Sun's Applet support in Firefox... or maybe it's Firefox's support of the Java Applet VM. Either way, it's a load of crud and needs to be worked on pretty seriously.

First, why on earth has Sun not yet made a 64-bit applet plugin for Firefox and other 64-bit operating systems. Even their own 64-bit Solaris doesn't have one. I've looked at the bug tracking for Sun's Java and there are requests as far back as 1.3.x-days for a 64-bit applet plugin. But nothing yet in 1.6 - you've got to wonder what their own developers are using - 32-bit Windows is the answer. Amazingly short-sighted. Thankfully, Leopard will be 64-bit and have a 64-bit Java applet plug-in for Safari, and I'm assuming Firefox. I am amazed to think it's going to be a Mac to bring this to a usable platform. Certainly puts Windows and Sun to shame.

Next, there is a limitation on the linux Java plug-in with regards to the line length of an applet PARAM tag. Don't know what the actual length is, but when I sent in the bug (with an example) to the Firefox bugzilla site, the guy that picked it up said that the problem was in the Java plug-in and that's all the further it got. On windows there's no such limitation, but I do a lot of my developing on linux (64-bit at that) and it'd be nice to be able to have the same page render on linux and windows the same.

So I'm trying to figure out why this page is bringing Firefox to a crawl and not IE. I'm now convinced that the interface to the Java plug-in in IE is significantly different than the one in Firefox, and the IE one is doing something to compensate for any limitations in the plugin. One such difference is that in IE there appears to be a static picture taken of the applet and then it's stopped. This makes for applets that simply display information (like graphs) quicker to load and less of a CPU hog. Since I can't get the Firefox team to look into it, I'll have to simply hope that the next release will implement the plugin interface better and some of these things will just work themselves out. I sure hope so.

Today I took out a few minutes to check on the convergence issues that I remembered were in shark, the simulator that I had originally written as part of the Ph.D. thesis while at Purdue. The big problem originally was that there wasn't a machine fast enough to give me real, solid, answers to the questions of the physics of the problem. At least not in 1988. Now, it's a little different story and yet I remembered that I was still troubled by some convergence issues, so I wanted to take a look and see if there was an easy solution to the convergence issues.

The code tries to 'step' the bias on the device from the initial equilibrium state to the requested bias point. This is a classic problem in modeling - if you can't get a reasonable initial guess to the solution, then back-off on the initial state and see if you can get that guy solved. If you can, then use that as the initial guess for the next 'step' in the bias ramp-up. Seems reasonable, and it works pretty well.

One clever trick is to look at the difference between the values at each node from one bias point to the next, and then scale that for a better initial guess at the next bias point. In theory, it should get you closer to the next bias point than the last state, and therefore get you to the solution even faster. Problem is that if you don't get the "jump" right you can make the situation worse for yourself. And that's what was happening in the code. I was overshooting the change, which is not really surprising as this is a massively non-linear system and I was trying to assume for the bias changes it was linear. Not good. When I took that out of the code, it got to the right values fast enough and a lot more controllable.

The right solution might be to look at the last two points and allow the change to be second-order. It's better than linear, but I have a feeling that even that isn't going to be good enough at predicting the solution to warrant the cost, and uncertainty, of using the "jumps" in the biasing of the device. But if I wanted to try something else, I'm guessing an a logarithmic scale is probably closer to 'right' than even a second-order approximation.

At this point, all the tests check out and it looks pretty good. The next thing to do is to put in the input deck for the drifting packet that was the problem that couldn't get solved in 1988. It'll be interesting to see how long it's going to take with modern desktop computer hardware. My guess is it'll work, and be on the order of days, but it'll generate answers.