What's it all about, Alfie?

Well, it turns out that the ASCII-based exchange protocols NASDAQ, and some of the other lower-volume exchange feeds use is fine as far as that goes, but OPRA decided that it had pushed the limits of the ASCII protocol, and decided to make/adopt this FIX Adapted for Streaming - or FAST, protocol. In a sense, I can see why they'd adopt it - as opposed to writing their own, but I've read enough on the net to know that they really didn't adopt it 100% - just the compression of data part.

Basically, the FAST protocol is based on a few ideas:

• Very Little to no ASCII to decode - no longer will there be numbers represented as ASCII digits. Most numbers are now simply integers. In fact, they only allow for three data types: 32-bit integer, unsigned 32-bit integer, and a string. WIth those, and a few decoder tables, you can handle anything an exchange needs.
• Delta Encoding - there will be fields that are required in each message, but for some fields, the value present will be a simple increment, and in fact, it's possible to have nothing in the message, and have the assumption be that the value is simply incremented. This helps a lot. There are also values that are simple changes from the last value in the field, so duplicates can be removed. It's small, efficient, and makes for a compact encoded data stream.

The problem is, of course, that there is now state in the decoder. In general, this isn't bad, but what it requires me to do is to completely decode all the messages that I get, and the shortcuts I had that would extract just the sequence number, or just the flags for skipping the message - those are tossed out the window. I need to get all the data, and then deal with it.

This took a little while to work into my application, but in the end, I had the concept of a decoded message, and that message included the elements I had originally extracted, as well as the actual message. Thankfully, this is still pretty fast as OPRA isn't messing around with a lame decoder as it knows the point of this is to get more through the system.

I still need to do a lot of tests, and even finish writing my codec for the OPRA data, but at least I've got all the essentials of the FAST decoding working, and should be able to get moving forward again tomorrow with the messages.

This morning I got a tweet about MarsEdit 3.0.5 being updated with a nice little set of bug fixes:

• Fix bug with spaces being stripped from HTML markup for Flickr items
• Fix bug with tags that have spaces in them
• Fix bug that prevented split-view state from being saved/restored correctly
• Fix bug that caused the extended "more" text for WordPress Page entries to be blank
• Fix margin sizes in media manager window

So I naturally had to get it. Don't think I saw a one of these, but hey... it was easy.

This happened yesterday during my "move weekend", but even so, GraphicConverter 6.7.4 is now out with a few nice new features:

New features:

• scale after crop option added
• pdn preview display added
• sepia batch added

Updated features:

• paperport import improved (support for files with different resolutions and color depths added)
• raw importer updated

Bug fixes:

• possible issue with unsigned short import fixed
• unicode issue in preview/rename fixed
• general issues with the default color profiles fixed
• bug with scale to 1080p fixed
• possible printing issues with color profiles fixed
• xbm inverson bug fixed
• possible bug with scaling of animations fixed
• many other small changes and bug fixes

In the course of buying our new house, I found a wonderful use for GraphicConverter - generating PDFs from a directory of scanned images. In my case, scanned documents that needed to be pieced back together for shipping to loan officers, underwriters, lawyers and realtors. Very handy feature.

During my latest "move weekend", the Twitter folks have been testing their new OAuth system, and the guys at Iconfactory have released a new Twitterrific for Mac 3.2.3 - but there was a slight problem, and they had to release 3.2.4 very soon after that due to rate limiting and OAuth issues.

Not to worry, I keep up to date on the tools I use every day, and this is one of them. It's still the best twitter app on the desktop and the iPhone. I like it's simplicity, but I'll admit, that there are a lot of new features in the iPhone version that I'd love to see int he Mac version. But please leave the UI the same. It's the best part.

I need to have VLC if only for HandBrake so I can rip my DVDs to play them on my MacBook Pro and iPhone. But it's nice to see that they have been squashing bugs, and that VLC 1.1.3 is out. For what I need, it's the only thing out there, but it's really good to see the movement.

Well... I'm donning the old fedora again, and off in search of the Lost Codebase. It's really quite amazing the skill that some people have to hide code. I'm sure they don't think of it that way - they probably consider it to be exactly where they want it to be - the right spot. But if I can't find it after working in the repository for nearly two months, then it's time to call it "hidden". Yup... hidden. And that means I need to get out the fedora and get exploring.

The first thing I check is of course, the most obvious - the name of the directory. Clearly, this is a trap, for who in their right mind would put the code in a clearly labeled directory. No, that's the location for some of the code. Maybe. Hard to tell as the class files are nearly completely empty, and one would wonder if the code even compiles. I'm not fool enough to fall for that trick - typing 'make' could end of wiping out my entire machine's drive. I'm no fool.

Next, I check the similarly named directories. No luck there, but not nearly as complex, and some of the traps aren't even well constructed. In one there's no Makefile - a dead giveaway, if ever there was one. In another, they foolishly only include a handful of files. This is too easily scanned and I can see what I'm looking for isn't there. In all, a minor detour, but I have no idea where to go next.

Next I have to go with the big guns - I grep for a keyword in the entire source tree. As expected, this yields far too many hits, and I need to filter it down. Doggedly, I wrestle the filter on the grep to give me something I can work with. I struggle weeding out the false hits. I finally think I may be onto something only to have my hopes dashed when it's a simple comment and not the real code I'm looking for.

It's frustrating, and in the end, I realize I've met my match. I have to back off, regroup, and hope that when the author(s) decide to come in for the day, they have some answers to where the hid the secret directory to the code.

Oh yeah... I even checked for the hidden directories... no luck.

It looks like they have fired up the 'dev' channel again as Google Chrome dev 6.0.495.0 was released this morning. I went back to the dev channel after moving to the beta when it was released a few days ago. I have to say, this has become an incredibly stable platform. It's fast, looks like a Mac app, and it just plain works. Nice.

The release notes indicate that, for the Mac at least, we're getting a fix for the download shelf, and a few fixes for CSS and plug-in handling. Looks good to me.

I've been working (still) on getting more exchange feed codecs into the system, and while it's not really hard work, it takes a little thought, and a lot of attention to detail. So when I get some kibitzing from those that would love to see me fail, but are too afraid to really stand up to this project, it's like swatting flies - not hard, they aren't going to do me any harm, but it's annoying nonetheless.

When it gets bad, I just get up, take a little walk, get a pop, and clear my head. That usually does it. Oh... and getting another feeder done in less than a day makes me feel good. It shows the "flies" that they really might want to take notice of the different way I've put this together. But that's really hoping for too much, I suppose.

Time to get some bug spray.

This morning I noticed that Google Chrome beta 6.0.472.36 was out - still no word on a new 'dev' release, so it appears that for now, they are simply sticking with the 6.0.x branch and not starting anything new for the time being. It seems reasonable that if they aren't making major changes, they can keep the 6.0.x branch moving along from dev to beta to stable. It's only if they have great new ideas that it makes sense to open up the dev branch again.

So it's out there - a few little UI fixes - nothing major.

Today I spent all day working on getting two exchange feeders written and tested. This kind of speed is not because I can copy/paste very fast, it's because I've got a solid design that allows me to leverage the work I've already done and customize it very quickly and easily. Given that the last developers of these feeds took months to achieve what I've done in a day, there's a lot to be said about the power of the design. The previous one was particularly ill-suited to this task.

So it was a hard day, but I'm getting a lot closer to the point that I'm caught up with all the exchange feeds we have. At that point, I can look to data enrichment, and really start to add value to the data feeds.