What's it all about, Alfie?

I noticed this morning that iTerm2 Alpha13 is out with a ton of really neat features. The searching, the playback - Holy Cow! These guys are taking the terminal to places I never even thought about. What an amazing piece of work. And it's slick. Very slick.

Excellent work.

I was coding this morning - adding a new synchronous calling scheme to my existing asynchronous library, and really enjoying the capabilities of boost's asio library. I knew what I wanted to write, and I had the basics of the skeleton there for the asynchronous versions, but there were a few differences that arose simply because it was synchronous.

So I was writing this up, got done, compiled it and ran the test app to verify that I hadn't changed any of the behavior, and BINGO! It just worked.

Now, this wasn't cracking the atom - it was only about 100 lines of code, but still... the number of times I've been able to write, compile, and run code and have it just work is pretty rare. It was a very fun feeling to have. Just type it in and run it - it's pretty wild.

Good times...

There comes a time when you are done with the major issues and start to do the full-up tests when you run into little bugs that take all of about 15 mins each to solve, and you spend hour after hour fixing these little nagging problems. It's the polish in the project. You could leave these in and have the system restart, but that's got no class. You need to spend the 15 min and just fix it.

That's what I've been doing all afternoon - fixing these issues. I've solved a bunch of them, but I think there are a few more that I need to handle. Certainly when it comes to communicating with the Broker. That guy needs some help when it comes to massive hits from all my ticker plants starting up. I'm not sure exactly how I'm going to deal with them, but I know I'm going to have to.

It's all a part of the process. At least I'm no longer writing bash scripts. That was also necessary, but wasn't nearly as fun.

This morning I noticed that Adium 1.4.1 was out, and that's a major surprise. The blog post says what it's all about:

While the primary purpose of this update is to fix ICQ authentication[1][2]

[1] For those that are unaware, ICQ is no longer owned by AOL. The breakage that users may have experienced was due to the change-over to the new authentication servers. (login.oscar.aol.com – > login.icq.com)

[2] Users of Mac OS X 10.4 using Adium 1.3.10 can change their server setting to “login.icq.com” in the “Options” pane of their ICQ account’s settings.

I didn't know that ICQ had broken off of AOL, but so it goes. I was able to maintain the SSL connection to AOL even though the SSL connection to ICQ is now gone. Not bad, I guess.

It's nice to see the improvements.

Today I'm spending a lot of time hardening the TickerPlant for production. It's not the most glamorous work... in fact, it's kind of mind-numbing, but it's as important as getting the rest of the core right, as the program isn't going to be run by me - it's going to be run by operations, and that means it needs to be a lot more secure than I might normally make it for myself.

I started with a little better application shell - it's about 50 lines of C++ code that starts things off after reading the command line arguments. I added in the standard 'usage' function to act as the "help" of the application. I then added a lot more comments in the code and it was looking pretty nice.

Then it was off to the shell scripts. I need to make the start/stop/restart scripts so that the user doesn't need to worry about the environment or the location - it's all just automatic. I've used scripts like these in the past and they have been really helpful, but getting them all into bash will be a little work - they were a mixture of bash and c shell in the past.

It's not glamorous, but it's necessary.

UPDATE: I still need to add in the sending of the ticker plant's stats via SNMP and add in the Jabber client so that we can communicate with the TickerPlant easily.

I have been looking at the memory footprint of my best bid/offer server and thinking that there might be a memory leak in there. I've done a lot of unit testing, but it's very hard to tell exactly what's happening when you hit it with 128,000 symbols and more than 30 ticks a symbol. After all, I expect the memory usage to rise as I put in new instruments. But I was worried that I was still creating data structures for existing instruments.

I looked at the code, and traced it for a few cases, and it seemed to work, but that wasn't nearly as satisfying as I had hoped. I really wanted to know for sure. So something finally came to me: Hammer it! I mean, really hammer it.

So I added a few loops to the test app and used the existing 128,000 instruments. I ran through another dozen ticks that would very predictably effect the best bid/offer. I ran that for each instrument ten times and let it run.

What I saw was that the memory climbed during the initial phase, and then during the "re-run" phase, it was rock solid. I mean it was amazing.

I'm satisfied. The small-scale tests are good, and the massive scale is as well. There aren't any leaks I can see. What I have now is what the system requires.

It seems like the VLC group got the Mac support it needed to make a solid Universal app with support for both 32-bit and 64-bit systems. It's been in most of the releases of late, and while I still check, I think it's likely that it's always going to have this kind of support. Good news.

I'm not a huge fan of VLC per se, it's Handbrake that I use, and it needs the libraries in VLC to read the DVD. It's simply the most convenient way to get the libraries.

This morning I saw that BBEdit 9.6.1 was released. It's got an impressive list of fixes, and while I can't say that I've hit one of them, it's nice to see that someone is really kicking the tires, and they are fixing these issues quickly. Good enough.

I was looking at the memory usage of my NBBO server and noticed that after a little bit it was hovering on the high side of 4GB in memory. While that may not seem like much, it's far too big for what this process was doing, so I decided to dig into what that number was all about today. The end result was that I really needed to change the serialization scheme I was using for my engine component.

In order to have everything all "fit" together, I have a Broker service that holds my configuration data. I've been placing the current running state in that service for each instance, and it's been working well for me. This time, however, I realized that while it works for me, it's not really very efficient, and that's what's killing me.

What I was doing was to place all the values into lists and maps and then just bundle them all together and let that be the payload that was sent to the configuration service. While this is just fine, it leads to data structures that are pretty involved to deal with, and really no more readable than the byte-level encoding I'm doing with my messages. So I decided to go back to that scheme as it should be easier for someone to understand as it's also used in the message serialization code.

The wrinkle to this problem was that I had several objects/structures and each one needed to be serialized and deserialized properly and while it's not hard, it was time-consuming. Each one needed to have it's own format, and then I had to write it up, and finally test it. Slow and sure is the best way to do all this stuff.

In the end, the serialization footprint went down from 2GB to about 50MB - significant reduction in the transient memory usage of the process. At the same time, the speed of the serialization and sending went down - not bad, either.

Looks like Oracle and Apple have come to terms about supporting Java on the Mac. Now it looks like we'll have a standard distribution of Java from Oracle (Sun) just like the Windows and Linux builds. That's nice.

I have no doubt that it'll be more current than the versions Apple ships - they try to stay pretty stable on a single version of the OS, but Oracle (Sun) really moves regardless of the underlying OS version. I'm just not sure about the overall quality, but I guess that's going to be as good as the linux port. Which is to say, not bad.

Good enough. Glad to hear it.