This morning was a banner morning for open source browsers. Starting with Camino 2.0.4. Today's update has quite a few nice things about it - an updated Java plugin, fixed Flash issues, latest Gecko engine... it's a good update. Now, I'm still not sure I'd use it day-to-day, but that's because Google Chrome and Safari are the best two I've seen today, but it's nice to see Camino stay up to date and push the rest to not get complacent.
This morning I saw that Apple had an update to iWork '09 (9.0.4) on Software Updates, so naturally I had to get it. While I'm not sure I'll be doing a lot of ePub work, it's nice that they have closed the loop on the iBooks and content creation. Now it's simple to write something, put it in ePub format, and get it into iBooks. Very slick. But then again, that's what we all expect from Apple. Nicely done.
Today I was doing some testing on my latest data codec in my new ticker plant, and I ran across some performance issues that I didn't really like. Specifically, the processing of the data from the UDP feed was not nearly fast enough for me. As time went on, we were queueing up more and more data. Not good. So let's see what we had in the mix that we needed to change...
First, the buffer I was using was assuming that the messages from the exchange were not completely within a UDP datagram. This was a nice "luxury", but it's not true, and it was costing us time in the processing. It's better to assume that each UDP datagram is complete, and queue them up as complete units to process, than to have the logic in the buffer to "squish" them together into one byte stream, and then tokenize them by the ending data tags.
That was really quite helpful because at the same time I decided that it was a bad idea to use the mutex/conditional I had set up to allow the one producing thread and one consuming thread to efficiently access the data. Instead, I grabbed a very simple lockless circular FIFO queue off the web and cleaned it up to use for this UDP datagram buffering. It's easy enough to use - there's one thread that moved the head, and another that moves the tail. Simple. As long as the head and tail aren't cached on the CPUs, it'll work without locking. Simple enough.
But when I get rid of the locking/waiting, then I have to handle the case where the queue is empty and we need to try again. My solution there is to start simple and put a simple 250 msec wait. When I started testing this, I saw that there were significant pulses in the incoming data because a lot of datagrams arrived while we were waiting. So I got a little smarter.
I added an expanding delay - starting small, and building, so that we can hit it quickly if it's a short delay, but when the close comes, we'll only do a few checks before it goes to only a few times a second. That's very reasonable.
I did more tests and finally ended up with a variable scheme that had no delay for a few hits and then started stretching it out. Very nice.
In the end, I had something that emptied far faster than the UDP data source, and that's critical for a ticker plant. There's enough to slow it down later in the processing, so it's essential to start out as fast as possible.
Well, it's taken me a few days, but I've finally finished the code in my ticker plant to handle the options data feed. It's a biggie because instead of doing the same ASCII encoding that the other exchanges do, they switched some time ago to a FAST (FIX Adapted for STreaming) encoded stream to reduce the bandwidth needed to move the data from them to us. This just added a new wrinkle as we had to incorporate their FAST decoder implementation (initially), just to get the data into a binary format that we could do something with.
Then we had to adapt the code to allow for the fact that some messages from the exchanges, specifically OPRA right now, generate multiple messages to flow downstream. This wasn't hard, but it was in all the codecs, so it took a little time to get it all right and working properly.
I got it all finished, compiled correctly, and looking like it's ready to test. Time to commit it all to git and then get to the business of testing.
This morning there was another round of security updates from Apple - covering PHP, PDF, etc. It's not a lot, but hey, when there's an exploit in the wild, you have to fix them, and I'm going to make sure I get the very latest updates. Just makes good sense.
Well, it turns out that the ASCII-based exchange protocols NASDAQ, and some of the other lower-volume exchange feeds use is fine as far as that goes, but OPRA decided that it had pushed the limits of the ASCII protocol, and decided to make/adopt this FIX Adapted for Streaming - or FAST, protocol. In a sense, I can see why they'd adopt it - as opposed to writing their own, but I've read enough on the net to know that they really didn't adopt it 100% - just the compression of data part.
Basically, the FAST protocol is based on a few ideas:
The problem is, of course, that there is now state in the decoder. In general, this isn't bad, but what it requires me to do is to completely decode all the messages that I get, and the shortcuts I had that would extract just the sequence number, or just the flags for skipping the message - those are tossed out the window. I need to get all the data, and then deal with it.
This took a little while to work into my application, but in the end, I had the concept of a decoded message, and that message included the elements I had originally extracted, as well as the actual message. Thankfully, this is still pretty fast as OPRA isn't messing around with a lame decoder as it knows the point of this is to get more through the system.
I still need to do a lot of tests, and even finish writing my codec for the OPRA data, but at least I've got all the essentials of the FAST decoding working, and should be able to get moving forward again tomorrow with the messages.
This morning I got a tweet about MarsEdit 3.0.5 being updated with a nice little set of bug fixes:
- Fix bug with spaces being stripped from HTML markup for Flickr items
- Fix bug with tags that have spaces in them
- Fix bug that prevented split-view state from being saved/restored correctly
- Fix bug that caused the extended "more" text for WordPress Page entries to be blank
- Fix margin sizes in media manager window
So I naturally had to get it. Don't think I saw a one of these, but hey... it was easy.
This happened yesterday during my "move weekend", but even so, GraphicConverter 6.7.4 is now out with a few nice new features:
New features:
- scale after crop option added
- pdn preview display added
- sepia batch added
Updated features:
- paperport import improved (support for files with different resolutions and color depths added)
- raw importer updated
Bug fixes:
- possible issue with unsigned short import fixed
- unicode issue in preview/rename fixed
- general issues with the default color profiles fixed
- bug with scale to 1080p fixed
- possible printing issues with color profiles fixed
- xbm inverson bug fixed
- possible bug with scaling of animations fixed
- many other small changes and bug fixes
In the course of buying our new house, I found a wonderful use for GraphicConverter - generating PDFs from a directory of scanned images. In my case, scanned documents that needed to be pieced back together for shipping to loan officers, underwriters, lawyers and realtors. Very handy feature.
During my latest "move weekend", the Twitter folks have been testing their new OAuth system, and the guys at Iconfactory have released a new Twitterrific for Mac 3.2.3 - but there was a slight problem, and they had to release 3.2.4 very soon after that due to rate limiting and OAuth issues.
Not to worry, I keep up to date on the tools I use every day, and this is one of them. It's still the best twitter app on the desktop and the iPhone. I like it's simplicity, but I'll admit, that there are a lot of new features in the iPhone version that I'd love to see int he Mac version. But please leave the UI the same. It's the best part.