What's it all about, Alfie?

Today has been a mix of a lot of changes - some small, some very large, but what really sticks with me about all these is that they are moving the Greek Engine project forward a great deal. It's really pretty amazing, and it all started because I wanted to make sure that I was getting accurate Future prices into the mix for the pricing of the underlying for some options.

What I found was that the Futures weren't getting quoted at all, and that lead to a dive into the code to see where it was they were getting held up. What it turned out to be was a one-line change - I wasn't "wiring up" the output of the Futures feed to the next step in the processing. I was getting the quotes and trades, but I just wasn't doing anything with them. Silly. Easy fix, and then I started seeing data.

Then I noticed that I was saving the state of the feed component to the cache service on red is, but I wasn't reading it back in. Once again, an easy one-line fix, but amazing that I hadn't taken the time to see whether or not this was really working. Got that down, and then things got really big.

One of the other developers let me know that the spreads he was trying to push through the system were going to have negative prices. Since all the code was using uint32_t for a price, that was a major change, and took me quite a while.

While I was moving from unsigned to signed integers for the price, I decided that it was probably a good idea to do the same for the sizes. You just never know, and the last thing I wanted to do was to do this again because some Bond or something will have a negative price to indicate some side or the other. So I changed both. It was massive. But I got it done.

Then it was a few more little things - make the query of Future data correct - I wasn't properly picking up the instruments, and then allowed for the data (quotes, trades) on Futures to create them in the environments if they don't already exist. I'm thinking that new trades and quotes on Futures is good data, and it makes sense to have too many than too few, so let's auto-add them.

In the end, a lot of changes, and I'm going to need to let it run for a bit to make sure things are working, but the effect is profound. We have far better data, better recovery, and things will tick very nicely. Lots of good work makes for a good day.

This morning I saw that Google Chrome dev 15.0.874.21 was out, so I picked it up. It looks a lot like it's getting close to the promotion stage, as backed up by several of the comments on the release notes page. I like that they are working hard on the Mac OS X Lion compatibility, and in recent weeks they've been updating the V8 Javascript engine as well. All good news.

Just wondering what they are going to do next. With the binary-mode web sockets, it's possible to do a lot with this… I just don't know what they are planning on doing. Hope it's interesting.

Today I spent the entire day working on the code to wire in The Broker's service that will inform me when updates to the underlying PostgreSQL database I've converted to, has changed. It's something like an extended trigger for clients of the data. I can read the data from the database and then subscribe to change notifications on a table-level basis. When I get a notification, it'll be up to me to read in the changed data, understand the scope of the change, and then apply the changes to all relevant instruments in the Engine.

It's a lot of detail work, and I had to work out a lot of the details today to get the roles and responsibilities right, but in the end, I think it's all there. The key data tables are in a pretty efficient scheme - only updating those complete units that had some change as opposed to trying to change individual values or blasting everything out on any update.

I still have a lot of testing to do, but I'm going to wait for my co-worker to get back as he's the one setting up the replicated PostgreSQL databases, and I'd like to have him helping me run the tests. SHouldn't be hard, I just want to be able to focus on my end of things while someone else focuses on the source-data end of things.

It feels good to get this all done.

This morning I finished up the code to put the market schedule in for every instrument in my Greek Engine. The idea is that there are default hours for the pre-open market, and then the standard trading day, and finally, the post-close market trading. Since these can vary by instrument type and Underlying, it made sense to make this a little more complex logic than a simple table look-up.

My co-worker had placed all this data into a PostgreSQL table in the replicated database that is the new data service for the application, and so I had to read that in, throw it into decent data structures for looking things up, and then adding methods that made it easy to get at it. Not bad, but it took most of the morning.

Now I need to get some odd pricing logic for certain kinds of instruments from the Quants, and also write the code to take push notifications on the changes to the static data and update as necessary. It's getting close, and I'm very glad to see it finish.

Today I spent most of the day adding in the ability to get and set individual instrument values from the Broker API for the Greek Engine. This is really all about the end of day processing where we need to know the last trade prices of all the instruments (that's the "get"), and then match that up with prices from Bloomberg, Thompson, and other sources, to make a unified "close", or mark, for the day. Then we need to set all these values back into the engine (that's the "set") so that they have the proper, verified, close marks for the day.

It wasn't all that hard, I just needed to expand a little on a few things I've already written for the Greek Engine. I had the bulk of the setter code already done, I just needed to make it a little cleaner and make sure it didn't freeze out any instruments from receiving further updates. The getter was a little more difficult, as it is meant to be a map-of-maps, and so I need to be able to build it up a family at a time.

Again, not horrible, but it was a solid day of writing and then updating test apps to test this functionality and then fixing any little issues with the code I'd written. It was a nice thing to finish up the week on a success. I need a nice weekend to recover from the beating I'm taking at work.

I haven't posted any updates to Google Chrome dev for quite a while, but I've been updating every chance I get. The most recent cut is 15.0.874.15 and has the latest V8 javascript engine, as well as lots of fixes for Mac OS X Lion (10.7). It's been at version 15 for quite a while, and they are starting to issue updates to the "fourth octet", so I'm guessing that it's getting close to time to cut version 16 and make 15 the 'beta' channel and keep on going.

I was reading Slashdot this morning when I came across this article on Reuters:

UBS $2 billion rogue trade suspect held in London

and I almost had to laugh. In this industry a lot of people think investing is legalized gambling, and hedging is throwing away your profits. They make fantastic returns until the day they are wrong, and then they make fantastic headlines.

This guy is no exception. Two Billion Dollars. That's a staggering sum of money, and certainly more than you should ever have un-hedged. But they play the game and hope that their role of the dice winds up in their favor. Too often, it doesn't. Now he's serving time with Carlos the Jackal.

Not my idea of fun.

This morning was a lot of work that turned out to be something almost funny in retrospect. I had just completed the conversion of the data source from the old MS SQL Server databases being accessed through SQLAPI++/iODBC/FreeTDS to using PostgreSQL using just the SQLAPI++/PostgreSQL libraries. Things were running, but the greek values were all wrong. I mean not even close to right. So clearly, I had messed up on one of the data import statements or values - somewhere. So I had to track it down.

The problem was that there were a lot of values coming out of the database tables, and I needed to very each one to make sure there wasn't an extra factor of 100 or 10000. So I started doing a lot of checking. Pretty soon, it appeared that there was nothing wrong with the data. Not a single thing. Very odd.

So I backed off and started looking a little wider. Maybe it was the set-up outside the calculations themselves. So I looked at the Holiday Calendar. In the previous holiday calendar SQL call, I received the list of holidays. When I looked at what I was putting in now, I realized that I was looking at a calendar - not a Holiday Calendar. Basically, I was telling the FE Model that every day was a holiday. Very bad. So I changed the SQL on the calendar to only pick out the market holidays and BINGO! The numbers fell right back into place. Very nice to see.

But after that I had to giggle… Every day a holiday. Kinda funny.

The next problem that was sitting on my "TODO" list was a core dump that we got from having a web client hit us as hard as possible - pretty hard, in fact. The problem was pretty simple, and as soon as I saw the back trace, I knew what the problem was: concurrent modification of an iterating container.

Basically, I have been using the TBB concurrent_hash_map to hold things, and in this case, it was really back-firing on me. The TBB containers are thread-safe, which is great, but when you're iterating them, it's possible to have one thread think it's got something, and another thread to delete it out from under them. No fun, and almost no way to clear it up with these data structures. What you really need, and what I went to, was a nice read/write spinlock that we can then ensure isn't getting modified during an iterating pass.

With this in place, we don't have to worry about the excessive load of the exclusive lock, we can typically use the read level on the mutex, saving the write for those times we add and remove pairs from the hash map. Then we can back off and use something like a boost::unordered_map for the container and be better off all the way around.

At the end of the day I started working on the next feature I was adding - the ability to pull out of a running Engine a provided list of values for a group (or all) instruments in the engine. For example, a way to say Give me all the last trade prices and sizes for everything in this profile. I'm going to need it, so I got started on it knowing I wouldn't be able to come close to finishing it today.

Today I'm trying to finish up on the migration away from hitting the legacy databases and towards the replicated, local, read-only, PostgreSQL database. It's been nice, but from time to time, it's required data that my co-worker is far better equipped to add than I am, so I've had a few spells of just sitting around. So it goes. Can't fix everything.

I'm hoping to get this done today, but who knows… there may be a few sources of data that are really difficult to get. We'll have to wait and see.

Today has been kinda fun, my Greek Engine reads instrument static data from some legacy databases, and then merges these many sources into one useful source of data. It takes about a minute, which isn't horrible, but it's an expense in getting the logic right, and adding new features based on this data is a serious pain. Thankfully, my co-worker had the idea of making a "data service" that's really just a replicated PostgreSQL 9 database where the master is written to by update jobs, and the data is replicated to read-only slaves on each server box for local access.

It allows us to do all this convoluted logic in one place and let it be shared by all the groups and applications that need it. It doesn't have to be normalized, and it's probably better if it isn't - the queries will be simpler and faster without all the foreign keys.

So he's been spending time getting this master database up and populated and updating with data from a lot of different sources, and today I've been working on integrating this into my Greeks Engine. It's a lot of fun because PostgreSQL 9 is fast, simple, and the support tools are just amazing. It's been a favorite of mine for many years, and the new version, 9.1, is no exception.

Replication is fast, easy, and efficient. It's complete and so far as we can tell now, fool-proof. Wonderful news. The only wrinkle is the development tools - on my linux box, they have both 8.4 and 9.0 and there was a little confusion this morning about what was on and what wasn't. I needed to get that cleared up before anything I wrote worked.

Updating the code was pretty simple - and greatly simplified. No longer do I have to have 50 line SQL queries to get the data. It's there and it works. No more ODBC drivers, either. Holy Cow! What a relief.

I need to do a little more work in the morning, but it's a great start.