What's it all about, Alfie?

I've been working on fixing up a script that takes a Bloomberg field definition file and generates a bunch of SQL statements to populate a database with this field definition data and it's right on the edge of really needing to be an application. It's all in bash now, but it really ought to be in perl or maybe even Java or C++. It's thousands of lines, so an application that can detect the existing data easily and only update the relevant records is going to get a lot way towards making this better. Perl could do it, but this bash scripting is just awfully limited.

Oh, I'm going to finish it, because I'm nearly done, and there's nothing in the requirements of the task that can't be done with bash and the other unix tools, it's just that had I originally known the issues with this updated file from Bloomberg, I'd have probably opted for a Perl script from the get-go.

There are spurious backslashes in the file, so I have to sed them out and make a temporary file of that. Then there are the problems with the intended primary key - this time around the file has duplicates on the primary key because it's really not meant to be unique from Bloomberg's point of view, I was just using it as a primary key because the first version of the file was unique. Silly me.

So something that should have taken five minutes is now in it's second hour as I find each of these issues in the 9000+ line file. The box it's running on is not slow, it's just there's a lot of stuff to do, and I'm not being really efficient because each time I'm blowing away all the data and regenerating it. Again, I was thinking this was 'easy'. Silly me.

The lesson in all this is that no one is right all the time, and even if you're right today, tomorrow will bring facts and circumstances totally unknown today and they will make the decision wrong. We have to be flexible and willing to see what's right and wrong and fix it - even if it means re-writing the entire process.

About eight years ago I was doing a project at the place I used to work and I needed to have a linux-compatible database that I could use that had C/C++ and Java bindings. At the time, MySQL was very popular, but when I looked at it I was struck by the lack of foreign keys and support for stored procedures. PostgreSQL had both of these, and while it wasn't as fast as MySQL, speed wasn't as important to me then as completeness of the database features. Times have passed, and MySQL has gotten better, new databases for linux systems have come into being, and PostgreSQL is still going strong. I've never regretted the decision I made those many years ago.

Both MySQL and PostgreSQL have corporate backing now, and I'm glad for each as the competition is good for the users. Most of the time, my PostgreSQL databases just hum in the background, but today I'm messing with one and it's really a much a joy to use today as I ever remember it being.

Everything is there that I need. And it's fast. Sure, it may not beat a clustered Oracle installation at 22.5 million rows, but then again, when I get a database that big, I'm sure I'll have the money to get something equally as expensive, and possibly it'll be PostgreSQL on it's clustering solution. But for the databases I'm using - under 100,000 rows per table, this is more than enough. It's fast, dynamically configuring, and it simply just works.

If only more commercial products did that. PostgreSQL... get it. You'll be glad you did.

Today I have spent most of the day working on a problem we ran into this morning about the FX conversion of the dividend curves in my server. I had to dig into this problem for quite a while to figure out what the problem was. At first, it seemed pretty clear - the FX rate for USD/CNY was 1.0 and it should have been a tenth that. The problem was Why?

The first thing I did was to be sure that the FX rate was now not 1.0. It wasn't, but this is where the complexity comes into play - I forgot to realize that the server is a multi-machine, multi-process entity, and the FX rate I was looking at was, indeed, not 1.0, but that's not the FX rate that was being used to convert the dividend curves. The FX rate used for that was still 1.0.

After looking at this for about 10 mins., I realized what the problem was, and realized that by restarting one of the components I could have it reload it's FX rates, and this guy was the one doing the dividend curve FX mapping. In the end, it was an easy fix, but it made me realize that I needed to have a better way to have these dividend curves mapped, and so that's what I set out to do.

Interestingly enough, one of the strengths of the server is that the components are very loosely coupled. This means that they can be independently restarted and the 'whole' will not suffer. Things are re-tried, re-sent, and life goes on. Very resilient. Problem is, this means that you need to have an exceptionally good communications system to make sure that what you want done to one component will be done to all effected components. Case in point, telling the system to reload an individual FX rate.

Clearly, restarting the components is an option, but that's not very user-friendly. What I was looking for was a way to tell the components when something was changed and tell them of that change. Problem was, this would represent a significant addition to the protocol that was already in place between some of the components. Not something to do lightly. Especially, if there's an easier way to accomplish the same thing.

So I kept digging. It turned out that the only reason this one component had the FX rates was for this dividend curve mapping, and if the data coming back from the database didn't need to be mapped, then this would no longer be an issue. Idea: Make the FX conversions in the database calls. Problem: can't slow things down, so I can't add a lot of processing and I have to be careful about the FX rates I use.

Turns out, the first worry wasn't too bad. I simply looked at the original data as a 'rough cut' of the data. If any FX adjustments needed to be done, I did them en masse. This meant that most things didn't experience any slowdown, and those that did, most times the curve would be converted all at once and not a point at a time. Nice.

The last problem was a little more difficult. I was able to work towards it, and I only had to make two assumptions - that the 'latest data' for both parts of the FX conversion had to occur on the same date (good idea anyway), and that it's always the 'posted source' so that we had the best marks in use. I thought these two facts should be true, but I wanted to run it by someone who's had a few more years at the data-side of things than I. After I explained this to him, he was convinced that they were valid assumptions/rules and I then knew I could complete the FX conversion within the database.

What's the point? Well... complex systems are complex. It's in the name. Even when I've worked with this guy for years, there are parts and interactions that you might not realize and it plays tricks on what you think should happen. Then you try it and you get a different result and start to think about it, and then in a flash it comes to you that you were wrong and it was right.

Kind of like those simulations where you don't program in certain behavior, but the higher-order behavior is a direct result of the low-level rules, and so the complex behavior is displayed on your simulation. Wild stuff. Cool, but when you're trying to make a simple change and the complexity of the system is staring you right in the face, it's giggling at you. You are the one that needs to adapt.

At the Shop, I've been part of an integration team that's trying to get this large system into place and running. There's a ton of connections to our other systems, and it's a big job, but the part that I'm working with most is the price injection - because I'm the guy that wrote the price feed, after all. Makes sense.

Part of this product is a messaging system - which makes perfect sense to me - build a strong backbone and then hang things off it and let the middleware take care of the delivery and communication. Good plan. Horrible execution. I mean the worst I've seen in ages. And I've got data to back it up.

The entire messaging system is single-threaded. In this day and age of multi-core, multi-CPU machines, this alone makes this message system a problem. The entire thing is a bottle-neck, as opposed to enabling multiple messages to go between multiple senders and receivers, this guy acts like he's the 93 year old mail clerk in the post office. He's only going to do one thing at a time, and it's a government job, so he can't be replaced.

Case in point, I sent a message - a simple message to be sure, probably less than 1kB, and it took 23 minutes to arrive. Now this is going from one box in the server room to another box in the server room. There's no reason it shouldn't be measured in msec. - let alone 20+ minutes. While I understand that the speed of delivery is based, in part, on the messaging database and the other traffic it has to deliver, there's still no excuse to deliver to a customer a messaging solution that's 20 minutes to deliver a 1kB message.

The funny thing is, I could read all the relevant data from the database (about 3000 records) in about 15 seconds, so while it's clear that something is holding it up, it's not the underlying database that's the problem.

I've had problems with this product before, and as such, I'm not about to name names, but if you're doing anything with a custom-built vendor-suppled middleware, it pays to look under the hood and make sure you can plug in something that's got known good performance like Tibco or IBM MQ. Make sure you're not stuck with this company's horrible middleware that's causing you problems for years to come.

[5/1/08] UPDATE: after a restart of the subscription connections, things seems to be working better today. I'm not sure what the problem yesterday might have been, but it's working better today. I'm going to watch it for several days, but now it's delivering the messages in under 7 sec., and while that's not the fastest message system in the world, it's certainly acceptable for this project. I hope it keeps up.

I've been working on my old simulation code a bit in the evenings and weekends now, just seeing if I could get the GaAs simulations to predict the oscillations of the 1D code. One of the things that hasn't bothered me - until now, is the timing methods used in the code. After all, I can time it on the wall clock and see how long it's taking, but I put in a decent level of effort all those many years ago to get the CPU split times for each phase of the simulation, and I thought it'd be nice to get them correct again.

I say 'again', because in the old f77 days, I'm sure dtime() was about as good as you could get. But in these new days of gfortran the values returned from dtime() are not in keeping with reality. I've looked at the GNU Fortran docs, and they say it's meant to return the elapsed seconds, but it's not. Maybe it's the build of gfortran I'm using, but I think it's more likely that dtime() is not what the new standard is using and I needed to move on.

So I did some digging in the code. Turns out F95 defined cpu_time() which returns the elapsed CPU seconds (as a real) for the execution of the app. This means that we need to put it into the code in a 'difference' mode - taking a reading at the top of the loop, and then at the bottom, and differencing the two for the incremental time that I was used to getting from dtime().

This wasn't all that hard, and in about an hour I had all the code re-fitted for the calls to cpu_time(). Thankfully, this is a much better timer and I get results that are making sense with the wall clock time I'm seeing for the runs. It's not like it's running any faster, or getting better answers, but it is at least more consistent, and I can look at the numbers and see what's more costly and from that see what I might need to do to alter the bias stepping, etc. Not amazing, but nice.

I was chatting with a friend today and he was telling me the story of a guy he worked with. This guy was holding up the project they were on because he refused to see that he really didn't have the SQL skills to complete this one part of the project, and didn't hand it off to someone with far better skills. He insisted that it would be better for the team if he learned how to do this, and in a sense, he's right. It could be nice for him to be able to do more for the Team, but he shouldn't get that knowledge at the expense of the project. Pick it up on something that has a nice, long timeline, and not the project they are on that's already behind and threatening to harm the business.

It's about understanding one's limits and staying within them. Oh, there are time to stretch your comfort envelope, but when other people are counting on you, and you have the option of handing something off or keeping it for yourself and trying to get it done - you need to think of the Team a bit. Otherwise, people aren't going to want to work with you and you'll end up getting only those things that other people know you can do.

I don't know how this particular story is going to end. I'll check back with my friend in a few days, but it's something I really feel sorry for the guy about. I mean, he probably doesn't even recognize how he is perceived. He probably thinks this is normal behavior. But it's going to get him into a load of hot water when management asks him why it's not done and he's still saying "...I'll have it done tonight."

The final test of my thesis simulator code was a simple gaussian pulse traveling down the empty channel of a Si MOSFET. The simulation of the first picosecond took close to a month on the school's Gould NP1 supercomputer. Today I finally tried running the simulation on my MacBook Pro to see if I could get the same results as well as see how long it'd take.

I had typed in the input deck this past weekend, but hadn't had a lot of time to run it and see what the results were. Today I just fired it off in the background and came back to it every so often to see if it had completed. What I found out was that I hadn't typed in everything correctly and I needed to increase the maximum number of allowed Newton iterations to get to the convergence criteria I had in the code. No biggie, but it would have been especially nice to have gotten it right the first time.

When I finally got the upper limit set right, it turned out that I did the complete 10 psec simulation in about 72 mins. It's all single-threaded code, so there's a possibility of optimization there if I spend the time to make the matrix solution multi-threaded, but the work to do that would be significant. Still... under 2 hours versus a month for a tenth the work. Yeah, computers certainly have changed in the last 20 years, but it's times like these that really accentuate it.

I'll probably try to do some of the simulations that I simply could not do at the time - like the GaAs channel, because in those results are the real questions of the thesis work: Was it a 1D simulation effect? and What will the frequency be when taking into account the 2D v(E) vector? In the end, I know what the results should say... it's real and it's reasonable, as after I left, the next student actually got one of my designs working. But still... it's nice to be able to finally close the book on that chapter of my life.

I have been playing with MacVim quite a bit recently, and reading a few things off the Google Groups mailing list (vim_mac) and noticed that some folks were saying that there was a noticeable update delay on MacVim on their boxes. They didn't give details on the machines they were using, but they did say that the faster update scheme that the other Vim for Mac OS X also uses is the ATSUI renderer - MMAtsuiRenderer (as it's called in MacVim). So I decided to give it a go, you never know, it could be amazingly fast.

So I got the preferences setting that isn't available in the GUI and set it up:

    defaults write org.vim.MacVim MMAtsuiRenderer -bool YES

and then restart MacVim. Turns out, it's faster, but it wasn't really slow on my MacBook Pro. Now, maybe the others are using slower MacBooks, or even G4 iBooks, I don't know. But it doesn't hurt to have the faster renderer being used.

I'm still amazed by the job they have done. Really exceptional.

UPDATE: it turns out that if you turn the ATSUI renderer ON (at least for version 7.1/26) then the mouse clicks don't move the cursor to the location clicked. This is not a "good thing" and I've posted a message on the vim_mac message board to see if I can get this either fixed in the next release, or if there's something else I need to set in order to get the click/moves working again. We'll have to wait and see, but for now, I'm turning it off as the updating wasn't bad at all, and I need to be able to click-n-move.

[4/25/08] UPDATE: I got news back from the vim_mac mailing list. It seems the mouse support for the ATSUI renderer is not in the code. It's a known issue that they are working on when they have the time. For now, I'll have to just use the NSTextView renderer which is OK with me.

It's interesting working with developers and users when it comes to market data. They are often convinced that they need real-time data for prices and calculations driven off prices and they don't really stop to think Why? I do that a lot, and have come to the conclusion that most, and by 'most' I mean anything involving a human being in the loop does not need real-time data. Period. Here's why.

First, there's the calculations. Most applications aren't simple tickers - those are the trading apps, and they need prices that are as close to the market as possible, but then again, if you're trying to watch a few hundred symbols, the odds that you have a powerful enough desktop machine to actually keep up with the ticks from a data source like Reuters is iffy at best. You need it to be as close to the market as possible while not crushing your machine and making it virtually a single-use terminal for data.

So if you have calculations, like exposure, or running P/L, and it's aggregated in any way, then there's very little chance that you have an efficient enough system to actually handle all the ticks that a real-time price feed can dish out. Getting backed-up isn't the answer because then you're behind the market and still have to play catch-up. Nope... you need to be intelligent about what you do.

Secondly, even if you could keep up with the flow, the human watching isn't going to be able to respond to all the ticks individually - heck, it takes us 0.7 sec to hit the brake in an accident situation, there's no way someone if going to respond to a tick for a very liquid stock twice (or more) a second. No way. Automated trading systems are a different beast, but they don't have a human in the loop.

So... the reality of the situation is that for price data feeds you really need a good set of prices. Something that's very close to the market, say less than 3 sec, but not real-time because that's too much. Problem is developers want to have real-time systems because they sound neat. Yeah... I can see that, but it's not reasonable, and when you try and tell them this they aren't at all interested. Rather, they tell you that they can make it happen... and they've thought it all through and have all it takes to do it. This is must likely when the buzzwords come out with newfangled messaging systems to boot.

So you have to back up and explain the realities of these feeds to them. It takes about 30 to 45 mins to get through to most decent developers, and then they start to see the real scale of the problem. Statements like But I'm only registering for 400 symbols turns into "Yes, but you're registering for the most liquid 400 symbols that is going to be a significant real time load." Oh... I didn't think of that. Yes... I know.

Once it's explained, you end up with the standard market data 'bet': Let's try it my way, OK? It's already built, debugged, and ready to go. If this isn't good enough for you and your customers, then we'll do it the other way, OK? In all my experience I've never had to come up with the 'other' way of doing things.

But today I thought 'Why even get into it? Make something that appears to be streaming, even if it's not?' and so I did. I started by creating a nifty set of coalescing queues (FIFO and LIFO) where the push() method takes a key and a value. The key is the primary identifier and the property on which the coalescing will take place. For prices, this is the name of the ticker, but for other things it could be the address, or primary key from a database. The idea is that the order of the queue will be preserved but if you push() a value onto the queue that's already there (as defined by the key) then the value will be replaced, but the order in the queue will be maintained. This means that if you're using this for market data, the prices will keep updating even when you're not servicing the queue, so that when you do service the queue, the order is maintained but the data is the most recent data possible.

I put this into the client code for my price server and then made it possible for users to subscribe for prices and then 'turn on' the delivery of updates and simply "watch the queue" for updates. The queue has all the thread-safety and conditional code in it to make it very easy to simply ask for something from the queue and then as soon as something is ready, it's returned and you can process it and start at the top of the loop again. It's easy to put this in a simple service thread that does nothing but pick things off this queue as they arrive.

It's the illusion of real-time without the headaches. I let the users think they are getting real-time prices and not polling when they really don't know the mechanism that's getting those prices into the queue in the first place. Additionally, they aren't having to deal with identical prices and filtering them out - I do that before I put the prices on the queue in the first place. What it does is really short-cut the argument a bit by saying "try this, and let me know" - and then not hearing from them ever again.

Beautiful solution. I love it.

I've been a big fan of vi and Vim for a while. I started using it back in grad school and it's been on any Unix system (or Windows for that matter) that I've ever come across. Mac OS X has shipped with it in the 'console' mode since it came out, and yet they haven't spent the time to make it really a Cocoa app - and it's understandable. Priorities.

So when I came across the Vim for Mac OS X web site I was really jazzed. They had, essentially, gvim for Mac OS X. Nice. There were several things that kept me from using it full-time on the Mac and those were primarily limitations in gvim itself - to have multiple windows you had to have multiple gvim instances. But then today I was checking to see if the Vim for Mac OS X web site had an update from the 7.0.224 it's had for a while, and I went to the Vim wiki and it led me to MacVim.

Amazing. Nothing short of brilliant and stunning.

This guy, has put gVim to the level of a regular Mac OS X text editor. Multiple windows in the same running application instance... tabs to show multiple file buffers in the same window... transparency on the windows - it's amazing! It's a complete Mac app, but it's Vim!

I can leave it running without a window open, I can open multiple files in a window - open multiple windows, Cmd-W to close the window... it's everything that I had hoped for in Vim and it's working on Mac OS X now. It's even got code to check for updates! This is without a doubt the way to enjoy Vim on the Mac.

UPDATE: OK... I'm about as jazzed as I've been in a long time. This release of MacVim is amazing! For BBEdit, I built etags and get them into the Makefiles of a few projects. While it's not perfect because it'd be very difficult to be able to determine the context of the method invocation, for a lot of things, tags are really useful. It's nice to be able to jump around the code easily without having to move your hands from the keyboard. I guess that's the thing I like most about Vim - it's all Old School - just like me.