What's it all about, Alfie?

Given the state of web services and protocols these days, it's really pretty amazing that Twitter is as unreliable as it is. Before I even started using it, I had read many reports both praising the idea and cursing the implementation for it's unreliability. New things can start out on shaky ground, but as time goes on, they should stabilize out and get more reliable. But I don't see that happening with Twitter. I'm using Twitterrific and while I like most parts of the app, and understand they are working on an iPhone app for release in a few months, I have to say that I don't for a second think it's Twitterrific's fault.

So why is Twitter so unreliable? Can it be the load? Possible, but that's something that can be scaled up, and given all the attention it's received, I'm guessing that they aren't hurting for money to buy additional servers. So I don't think it's a matter of hardware.

Could it be the idea behind it? Doubtful, IM clients handle much larger packets of information and they aren't this bad. Also, given that it's not a 'push' technology, it's hard to understand the possible issues. Maybe it's the SMS or IM integration issues... don't know.

I've heard that it's all about being first. I disagree. That's a big factor, but it's not the end-game. Look at VisiCalc. Can't? Exactly my point. They created the spreadsheet and Microsoft beat them so badly that now most folks won't even know their name. Twitter? Twitter who? That's what might happen if they don't get the reliability up. People will like the idea and the barrier to entry is nothing for a Google, Yahoo, or Microsoft.

Ideas are great, but eventually you have to deliver. Twitter needs to get busy and deliver on the reliability.

While it might not seem like much, being able to upgrade WordPress from 2.3.3 to 2.5 was nice to get done today. It takes a little longer for Fantastico at HostMonster to get the updates, but they work like a charm, and I haven't lost a thing. Can't beat that. Unfortunately, there's already 2.5.1 out, and so I'm still behind. Not to worry, it'll catch up... and if the Google Summer of Code is right, we'll get caching in the default WordPress which would be really nice because the existing caching schemes make it seem a little non-standard.

I'm not about to get Slashdotted anytime soon, so I'm safe. When it's built into the WordPress release, I'll get it. But I do with they had better facilities for Privacy. There's really only the ability to mask out the search engines. I'd like to be able to work on an opt-in policy where I had to give people accounts on the install and then they could read it. That would be nice.

When I got home today my Kindle was waiting for me. I'm currently in the middle of Game of Thrones by George R. R. Martin - about 120 pages in, so I didn't think I'd be using the Kindle until I finished off this 800+ page book. Amazon, and luck, proved me wrong.

What I think one of the nicest parts of the Kindle is that I can get recent releases at paperback prices. I'm not one to buy a hardback book for myself. I just don't. I wait until they come out in paperback and then get them. That means I'm about a year and a half behind 'current'. For about the price of a paperback, I can get new releases on the Kindle. That's really nice.

So I open up my Kindle and find out that it's already been set up for me. That's probably nothing for Amazon to do, but it's an amazing feeling to have it there, out of the box, and ready for me. Very cool. I read the introduction on the Kindle - on the Kindle, and then decided to check out a few books I've wanted to have 'on hand' to read. So I go to the Kindle store.

Very nice experience. Clean, decent. All I could have expected in a wireless delivery system. Very nice.

So I look up Clive Cussler and get his latest two that are out in hardback. It's next to nothing to get them, and $20 later, I have them. It's simple, fast, and enjoyable. I can see why people are saying this is the 'iPod of books'. It's not a perfect device, but it's darn close, and they have a very nice online store.

I then go to look up Goerge R. R. Martin and find out that I can get two eBooks - the one I'm reading and the next in the three-part series for the cost of the second one. I get them both. I then use the Table of Contents on the Game of Thrones to get to where I am in the 'dead tree' version. My Kindle is ready to use.

Amazing experience. Really. About the only downside I've seen is the carrying case. I'm not sure that I'm going to use it - I don't use one for my iPod Touch and just take care of it, but it would be nice to be able to 'lock' the Kindle so that accidental presses didn't mess with things. But maybe they can come out with that in a software update.

In any case, I can see that my reading will be changed forever. I'll pick up a 4GB SD card this weekend and plop it in the Kindle and be set for life. Really. I can't imagine ending up with more than 4GB of books on this thing, but that's a staggering thought. I only wish there were a way to get my physical books on the Kindle without re-buying them. That's the one beauty of the iTunes system and music. I can convert them by ripping. Can't do that easily with books. Shucks.

I've spent most of today debugging a problem I saw in a vendor's API to a messaging system. It's not the best vendor I've ever worked with, in fact, I don't think they are even in the top 80%, but they are the vendor I have to work with at the time, and I've had to try and make the best of it. So here's the problem we ran into today.

We have a price injecting system. It gets price data over a custom socket interface (I wrote) from a price feed server and the reformats it into the format needed by the vendor's product and sends it on it's way. It's a simple transformation system. Nothing fancy. But the vendor's API is socket-based as well, and as we were to learn, not done nearly as well as mine.

When the transformer/feeder app was running in Chicago, and the price source was in Chicago and the database for the vendor's product was in New York, everything worked fine. When the feeder was in London and everything else was the same, my feeder missed the messages coming from the vendor's messaging system when I was injecting prices.

Inject prices - miss messages. Stop injecting prices - get messages. Make another test app subscribed for the same messages and it always gets them - because it injects no prices. This was getting crazy. So on a whim I decided to make use of a price source in London for the London feeder - Bingo! Now it worked. It appears that the vendor's API can't handle delays in the socket delivery from another completely different source. Yup. It's got nothing to do with the use of the vendor's API - it's the activity on a completely different socket that's effecting the vendor's API.

Note that in all this, my code is working fine. It's the vendor's that's stopped working properly. Nowhere in the documentation do they say that excessive waiting on socket communications will invalidate the delivery of messages - why should they? They probably never tested it, as they probably never had reason to. But when you charge $20 mil for something, you really ought to take a more pro-active view on things. For example, don't use a home-grown messaging buss when there are so many commercial ones that you can include in your $20 mil cost and not effect the bottom-line much.

In the end, I think I'm stable now, but there's really no way to know. They aren't going to fix this, I didn't expect them to. They took almost 2 months to fix the last bug we pointed out and that was a simple recompile with the right data type for a 64-bit version of the API. This would require real changes in how they do things, and change is not a word I'd use with this vendor.

I just have to say that I really hate the fact that it's expected that we figure this out. I'm not getting any part of that $20 mil, and yet I've saved their bacon by figuring out how to make it work in our environment. Crummy vendor.

I was talking to a friend today and he was fuming over a decision that one of the other guys he works with made. Of course, this other person really isn't in a position to make these kinds of unilateral changes, and then have other systems change as well, without at least some discussion with the lead developer (my friend), but that's what he did. It's the Better to ask forgiveness than permission style of work.

My friend tells the story that this guy needed to add a field to a RPC call, and while it was meant to be an integer, he decided to make it a string because he didn't want to get into parsing problems. The "fear of the integer" so consumed this guy that he told others down-stream of the system he was working on to use the string and then parse it as necessary. There were problems because this is clearly a dumb solution to the problem, and they had to change it to what it should have been all along - an integer. But the guy didn't want to change the RPC interface - 'leave it a string' he says. My friend was fuming.

Since this isn't happening to me, I can giggle about this to no end. There's a guy that shouldn't be developing because he's not really a developer, and doesn't really understand why you'd want to make the interface match the datatype - he's one of those guys that would make all the data in a database table two varchars() - one for the name, and the other for the value, and put everything in there. It's almost comical, if you didn't have to deal with the fallout.

So my friend has to spend the next several days unwinding all these changes and getting the right data type in there - in multiple systems, all the while cursing under his breath. Yeah... it's funny if you don't have to be involved in it.

There were a few things that I had wished were different about MacVim - specifically, the filenames in the tabs had the complete path name, nicely abbreviated, but still, there. I asked the support group about that and they came back with the most incredible answers. Now it's true, I've been using vi/vim for about 23 years, and I'm by no means a wizard at Vim - I use it, and I'm quick with it, but there are a lot of things about it that I'm not aware of. Today was an education.

The Current Directory of the File

The first thing that struck me was that when you opened a new tab with the File Open dialog, you'd get the complete path to the file as part of the name in the GUI tab. Yes, the path was nicely abbreviated, but it was still there, and if I had tabs that I'd opened from the command line, they didn't have the path. When I read the answer it was obvious - the current working directory of the command-line tabs was well know - the ones from the file open dialog wasn't. So what to do? Well, the answer was to change the directory on those files as well.

The brilliant answer came from one of the contributors on the mailing list: have an auto-command change the directory for each buffer you enter. The lines in my .vimrc that controls this are:

    if (has("gui_macvim"))
        :autocmd BufEnter * :cd %:p:h
    endif

The Format of the Tab's Filename

The second thing that can help is to force the tab to have just the filename and not the complete abbreviated path. This is done with a simple set command:

    if (has("gui_macvim"))
        :set guitablabel=%t
    endif

With these two additions I have two things that I really didn't like fixed. When they get the ATSUI renderer working with the mouse support, this is going to be one incredible editor. It's pretty awesome already.

I've been working on fixing up a script that takes a Bloomberg field definition file and generates a bunch of SQL statements to populate a database with this field definition data and it's right on the edge of really needing to be an application. It's all in bash now, but it really ought to be in perl or maybe even Java or C++. It's thousands of lines, so an application that can detect the existing data easily and only update the relevant records is going to get a lot way towards making this better. Perl could do it, but this bash scripting is just awfully limited.

Oh, I'm going to finish it, because I'm nearly done, and there's nothing in the requirements of the task that can't be done with bash and the other unix tools, it's just that had I originally known the issues with this updated file from Bloomberg, I'd have probably opted for a Perl script from the get-go.

There are spurious backslashes in the file, so I have to sed them out and make a temporary file of that. Then there are the problems with the intended primary key - this time around the file has duplicates on the primary key because it's really not meant to be unique from Bloomberg's point of view, I was just using it as a primary key because the first version of the file was unique. Silly me.

So something that should have taken five minutes is now in it's second hour as I find each of these issues in the 9000+ line file. The box it's running on is not slow, it's just there's a lot of stuff to do, and I'm not being really efficient because each time I'm blowing away all the data and regenerating it. Again, I was thinking this was 'easy'. Silly me.

The lesson in all this is that no one is right all the time, and even if you're right today, tomorrow will bring facts and circumstances totally unknown today and they will make the decision wrong. We have to be flexible and willing to see what's right and wrong and fix it - even if it means re-writing the entire process.

About eight years ago I was doing a project at the place I used to work and I needed to have a linux-compatible database that I could use that had C/C++ and Java bindings. At the time, MySQL was very popular, but when I looked at it I was struck by the lack of foreign keys and support for stored procedures. PostgreSQL had both of these, and while it wasn't as fast as MySQL, speed wasn't as important to me then as completeness of the database features. Times have passed, and MySQL has gotten better, new databases for linux systems have come into being, and PostgreSQL is still going strong. I've never regretted the decision I made those many years ago.

Both MySQL and PostgreSQL have corporate backing now, and I'm glad for each as the competition is good for the users. Most of the time, my PostgreSQL databases just hum in the background, but today I'm messing with one and it's really a much a joy to use today as I ever remember it being.

Everything is there that I need. And it's fast. Sure, it may not beat a clustered Oracle installation at 22.5 million rows, but then again, when I get a database that big, I'm sure I'll have the money to get something equally as expensive, and possibly it'll be PostgreSQL on it's clustering solution. But for the databases I'm using - under 100,000 rows per table, this is more than enough. It's fast, dynamically configuring, and it simply just works.

If only more commercial products did that. PostgreSQL... get it. You'll be glad you did.

Today I ordered a Kindle from Amazon. I tossed and turned over this for months. When I travel each day to work on the train, I read. A lot. So I have tons of books that are sitting on my shelves in my home office. I have been looking for an alternative, and the Sony feels nice, but it requires a PC - and I'm not about to go down that road. Also, I was very interested in the "shop and read" functionality in the Kindle. When I finish reading a book on the train, I have to sit for the rest of the trip because it's too bulky to carry two.

I'm not sure that the Kindle is going to be everything in eBooks, but I think it's got a decent chance. I was waiting for the second revision of the Kindle, but when Amazon came back and had them in stock, it was clear to me that the next version was not real soon now. So I decided to go ahead and get it.

It'll probably be like my iPod - got a 3G and used it for years before getting the Touch which is exactly what I want. I'm not convinced that Apple is coming out with a book reader - but it'd be nice if they did. I think that this is something that Amazon is in a unique opportunity to produce and provide. They have the bookseller reputation. So I'm going to give it a go.

I'll post updates as I get it and use it. Should arrive sometime this week.

Today I have spent most of the day working on a problem we ran into this morning about the FX conversion of the dividend curves in my server. I had to dig into this problem for quite a while to figure out what the problem was. At first, it seemed pretty clear - the FX rate for USD/CNY was 1.0 and it should have been a tenth that. The problem was Why?

The first thing I did was to be sure that the FX rate was now not 1.0. It wasn't, but this is where the complexity comes into play - I forgot to realize that the server is a multi-machine, multi-process entity, and the FX rate I was looking at was, indeed, not 1.0, but that's not the FX rate that was being used to convert the dividend curves. The FX rate used for that was still 1.0.

After looking at this for about 10 mins., I realized what the problem was, and realized that by restarting one of the components I could have it reload it's FX rates, and this guy was the one doing the dividend curve FX mapping. In the end, it was an easy fix, but it made me realize that I needed to have a better way to have these dividend curves mapped, and so that's what I set out to do.

Interestingly enough, one of the strengths of the server is that the components are very loosely coupled. This means that they can be independently restarted and the 'whole' will not suffer. Things are re-tried, re-sent, and life goes on. Very resilient. Problem is, this means that you need to have an exceptionally good communications system to make sure that what you want done to one component will be done to all effected components. Case in point, telling the system to reload an individual FX rate.

Clearly, restarting the components is an option, but that's not very user-friendly. What I was looking for was a way to tell the components when something was changed and tell them of that change. Problem was, this would represent a significant addition to the protocol that was already in place between some of the components. Not something to do lightly. Especially, if there's an easier way to accomplish the same thing.

So I kept digging. It turned out that the only reason this one component had the FX rates was for this dividend curve mapping, and if the data coming back from the database didn't need to be mapped, then this would no longer be an issue. Idea: Make the FX conversions in the database calls. Problem: can't slow things down, so I can't add a lot of processing and I have to be careful about the FX rates I use.

Turns out, the first worry wasn't too bad. I simply looked at the original data as a 'rough cut' of the data. If any FX adjustments needed to be done, I did them en masse. This meant that most things didn't experience any slowdown, and those that did, most times the curve would be converted all at once and not a point at a time. Nice.

The last problem was a little more difficult. I was able to work towards it, and I only had to make two assumptions - that the 'latest data' for both parts of the FX conversion had to occur on the same date (good idea anyway), and that it's always the 'posted source' so that we had the best marks in use. I thought these two facts should be true, but I wanted to run it by someone who's had a few more years at the data-side of things than I. After I explained this to him, he was convinced that they were valid assumptions/rules and I then knew I could complete the FX conversion within the database.

What's the point? Well... complex systems are complex. It's in the name. Even when I've worked with this guy for years, there are parts and interactions that you might not realize and it plays tricks on what you think should happen. Then you try it and you get a different result and start to think about it, and then in a flash it comes to you that you were wrong and it was right.

Kind of like those simulations where you don't program in certain behavior, but the higher-order behavior is a direct result of the low-level rules, and so the complex behavior is displayed on your simulation. Wild stuff. Cool, but when you're trying to make a simple change and the complexity of the system is staring you right in the face, it's giggling at you. You are the one that needs to adapt.