What's it all about, Alfie?

In the past, I've built UUID classes in Obj-C, Java, and C++, and in the recent project of my ticker plant, I did it again. It's not hard - you put 16 bytes in a union and allow them to be accessed as bytes, or words, and add a few methods and you're done. But today I realized that those 16 bytes could really be much more if I created a uint128_t. Basically, I created a general 128-bit unsigned integer and then subclassed the UUID off that. It wasn't that hard, but I was trying to encode a generic instrument name, and realized that a uint64_t wasn't going to fit, and it made sense to see if I could make it work with the new 128-bit version.

It didn't take me long, and the code was really pretty primed for it's use. The only thing that remains to be seen is the issue of performance. In general, there's going to be a performance edge to the built-in uint64_t because it's an internal CPU type. But exactly how much performance I give up for that doubling I don't know. We'll have to see as I put it into play.

I was watching the ZeroMQ IRC channel today and hear mention of an interesting alternative to Google Protocol Buffers - MessagePack. I'm not making any claims here, but the wealth of the client libraries for MessagePack is quite impressive. Additionally, while Google Protocol Buffers is working really well for Google, the marketing fluff for MessagePack seems to indicate that it's got a few features that Google's offering doesn't.

Also, they are as focused on speed as the Google folks.

I'm not sure what's going to become of this knowledge, but I do know that our in-house Broker is doing a lot of the same things, and it might be time to look at an alternative to the home-grown cross-platform serialization schemes. Maybe.

This morning I noticed that Google Chrome dev 8.0.552.11 is out, and the release notes say the changes are minor - with the standard stability fixes. There is still the graphics rendering issues when scrolling in a zoomed view, and without a fixed zoom level, I'm stuck with 83% as the first zoom level "out" from 100%. I would like to be able to say 85% or 90% - but no such luck.

But I'll keep hoping that someday soon these things will get fixed.

Today was spent dealing with this external data service that's delivering massive data sets to my ticker plant for it's use. It's basically a bunch of look-up tables - 400,000 rows in all. It's massive, and so there's going to be lots of issues with loading and accessing, but also the time required to get it all in-memory and running.

Details. It's all in the details.

I've got it to the point that it all loads, but there are a few things I'm not really happy about. First and foremost, is the asynchronous loading of the data. I had to do a few tricks to make sure that we didn't immediately reload the data after loading it once. When you place a simple lock on the loader, that's exactly what you can get: a back-up of loads. Not ideal.

Then we're left with the idea that the second call to the loader thinks the load is done because someone else has taken care of it. But it's not really done, it's just being done. So data might not be there. It's a non-trivial problem, and I'm going to have to deal with it sooner or later.

Like I said... it's all in the details.

This morning I noticed that Apple updated the Java deployment on OS X to 1.6.0_22 to bring it current with the latest offering from Sun/Oracle. There have been a lot of little improvements and security fixes for the JVM and it's very nice to see them kept up to date on the Mac.

Like it or not, Java is here to stay and it's an important part of my daily life.

UPDATE: This is not a good development:

As of the release of Java for Mac OS X 10.6 Update 3, the version of Java that is ported by Apple, and that ships with Mac OS X, is deprecated.

This means that the Apple-produced runtime will not be maintained at the same level, and may be removed from future versions of Mac OS X. The Java runtime shipping in Mac OS X 10.6 Snow Leopard, and Mac OS X 10.5 Leopard, will continue to be supported and maintained through the standard support cycles of those products.

Now I can understand the "Back to the Mac" idea of not putting Flash on the newly shipping MacBook Airs, but that's easily fixed - just install it if you want. They aren't trying to stop you on your laptop. But this... this is bad news. I've been using Java on the Mac for a lot of years, and it's been a fantastic platform for that. I just don't know why they'd want to ditch it unless it's about the Oracle acquisition or something.

I just have a bad feeling about this...

I noticed yesterday afternoon that Transmit 4.1.3 was released but I was too busy at the time to do much about it. This morning, however, I took the time to upgrade as it's simply a fantastic file transfer utility. Far better than Cyberduck (which just had it's own update) and it's all about the polish and the completeness. Transmit is simply the cleanest, nicest, most well-thought app of it's kind. Simply amazing.

Today I spent quite a bit of time tweaking the codebase to make sure that there wasn't something I was doing that could be done more efficiently with a little work. I've been working very hard on getting things done in the ticker plant, and it's client, that I haven't had a lot of time for the make it faster part of Make it work then make it faster mantra, so today I wanted to take some time to try and see what I could do to get anything more out of the system.

I also got a few requests from the new users, and I added those changes into the system, and I even actually like one of them. Based on the person that asked for it, I was surprised that I liked anything he had to say, but that's personal. Professionally, he's a dud too, but that's a different story, no?

After I was pretty much done with that, he came back to ask about something else, and I told him the API I had implemented for him - but hadn't finished to this point. It's some of the details that I knew I'd have to come back to, and it just hadn't been the right time. Clearly now, was that right time.

What I found in digging into the source of the data I needed was that there's no way for me to handle it by conventional means. It's data stream is about 55 MB, and that needs to be deserialized into a map of maps representation of about 800,000 elements. Way too big. All I need is the name-value pairs in the data, and I don't need to blow it all into a map-of-maps and then walk the maps to get the pairs. I can do that in a more streaming manner.

So I started to think about that and came up with an absolutely wonderful idea - I'll make an iterator for the data. It'll look and act like a standard STL/boost iterator where you'll initialize it with the data stream, and then 'increment' your way to the end. Each 'step' will have all the data you need for that step, and then the trick will be that the processing of the data will be up to me, the caller.

This will work wonderfully, as I won't have to deserialize all the data at once and I won't even have to spend the time to deserialize all the data at once. I can hold onto the data stream and deserialize what I need. If the user breaks out of the iterating loop I stop decoding. Simple.

I spent the time to get most of the header file done, and tomorrow I'll need to implement it. But I'm very excited about this as it'll fit very nicely into the scheme of things and really help what I'm trying to do.

Cool stuff.

Yesterday I had the real pleasure of working with the lead developer of ZeroMQ on a problem I was having with my TickerPlant. Specifically, I was sending one message and seemingly receiving more than 10. Very odd. When I started off with a note to the mailing list, it then transitioned into a nice chat on IRC, and when he needed to see code, I simply posted the programs I had been using on github as gists. It was a completely amazing experience.

I haven't worked remotely for a long time - about a decade, really. But when I did, the experience was a wonderful one. I got a ton of work done, I didn't feel out of the loop, and everyone was happy. This experience was even better, and it's because of the tools that are now available: github, the Mac and iSight cameras, universal development tools on most platforms - it's really made it possible for someone like me to do my job completely physically removed from the traditional workplace.

Once again, I long for the days of working at home. There's just so much to like about it.

I'm trying to narrow down my problem with the excessive messages I've receiving in my ZeroMQ receiver based on the messages I know I'm transmitting. Basically, the difference is about a factor of ten. It's bad.

So I made a very simple ZeroMQ-only receiver application with everything set up the same to give to the same results and tested it. Yup, same problem. Now, how to share it to the mailing list?

Gist!

I remembered that you can throw up code samples, etc. up there and then share them. So I pulled it up on my work box, sent up the app, and then saved it as a public gist. Bingo! I have a super simple URL to post on the mailing list and they'll be able to see what I'm trying to do.

I sure hope someone has some ideas.

UPDATE: I've been able to see that the transmitter is using just two of the 27 URLs and when I limit the receiver to those two URLs, the numbers match. It's as if the broadcasting is duplicating on all 27 of the channels and the receiver is picking them all up. I've sent something else to the mailing list and we'll have to wait and see.

[10/20] UPDATE: On the mailing list, I got a response from Steve-o that indicates that I really need to be using unique ports as well as unique addresses. The switch will filter on the address, and the OS on the port, but the listening on the port is by the port number, and it appears that the suggested use from Tibco is to have unique ports. This matches with the exchange multicast channel mappings, and I can do that.

When I put that into my receiver and re-ran the tests, the results were rock-solid and accurate. Martin still thinks there's a better way to allow for this in ZeroMQ, and I'm all for that, but until then, I'm very happy with the unique ports as it seems to make the OS/NIC as happy as the switch is with the unique addresses.

Sweet!

Once again, the Google Chrome guys have pushed out a new release, this one's only "release note" is: Fix minor stability issues. Some day, we'll read that this is really something akin to wiping out your hard drive, and they just wanted to downplay it so as not to harm "the brand". Like so many things in life, perception is far more important than reality.

Still, it's nice to see it getting better. I'd really like to see the OpenGL accelerated rendering and playback in the Mac OS X port, but I can wait.

UPDATE: I'm still disappointed in the default zoom rendering. It has all the graphics in a very 'rough' state until the window stabilizes, and then it redraws it to the right scale. Annoying, and I'd have thought they could put in the default CSS, but that's not working in this version either. Oh well... I wait for fixes.