What's it all about, Alfie?

Yesterday I was starting to work on my next little project - a price injector from our price feed to a new vendor app that we're integrating.

Another developer is building a nice C++ wrapper around the proprietary messaging API. It's not terribly complicated, but he's going to wrap it so that we can make a message system that we can use within the Shop without having to worry that we're overloading the vendor's product. So with that part done, or at least getting done, I can focus on the application that will takes prices from the price source (CacheStation) and then send it to the app via our message bus.

Configuration for these apps is something I've worried about over and over for each new app. You want something that's easy to edit... something that's got a central location for easy maintenance... yet not introduce more system dependencies than you absolutely have to. Most of the time this is a configuration file, but I have used configuration databases at times and while I like a lot about that, I think the better solution is to have a flat file that can hold complex objects and structures and then make it possible to have this reloaded on-demand or on a timer and in that way you simply edit the file and the changes will be read in.

So in doing this I was thinking I'd mimic the BKHashMap and BKHashTree in BKit. Both have been very successful for me, and so I started looking at the hash_map in the SGI extensions to STL that have recently been adopted by C++. The problem was that in order to use a CKString as a key in the map (or an element in a hash_set) I needed to specialize the hash<T> template. It was pretty simple once I looked at the header for hash_fun.h:

  /*
   * While I'm not a big fan of the way they have chosen to put in the hash_map
   * and hash_set templates - I think it's too clunky. They should have made
   * it possible to put a hash operator or method on the class, but that's an
   * opinion. Anyway, given how they chose to do it, this template specialization
   * allows us to create sets and maps using the hashing function below.
   */
  namespace __gnu_cxx
  {
    template<>
    struct hash<const CKString &amp;>
    {
      size_t operator()(const CKString &amp; arg) const
      {
        return __stl_hash_string(arg.c_str());
      }
    };
  }

With this, I can create the has_map by simply saying:

  #include <ext>
  #include "CKString.h"
 
  __gnu_cxx::hash_map< CKString, int, __gnu_cxx::hash<const CKString &amp;> > map;
 
  map["when"] = 10;

Once I have the hash_map I can build the CKHashMap and CKHashTree classes. These would then form the basis of the file-based preferences system for the application. I've already done the code for BKHashMap to read and write plists of objects, so it'd be easy for me to make nicely structure-rich preferences files and then simply read them in and manipulate them.

First, this morning NetNewsWire 3.1 was released, with what was announced as primarily a performance-improvement based release. This is great news. Not that NetNewsWire was a pig, or anything, but when they take the time to work on the code code of an app (like I have been doing to the server) it shows a commitment to the app at a level you don't see when you're just trying to add new features to make users happy. So, Great Job to Brent S. on the release. I'm running it already.

The next bit of news that comes from Brent and the NewsGator folks is that NetNewsWire is now freeware. That's right - totally free. All of the NewsGator 'personal' products are free. They look at this as the way to get mind-share and name recognition and it's probably going to work. There are a lot of people using something like NetNewsWire for personal reasons, and if they see a professional app from the same guys, they'll be more likely to give them the nod - all else being equal.

And at the same time, there's a great article on Rogue Amoeba about the devaluation of all software for a platform when a good piece of software is available for free. The posting says:

There's certainly a place for free software. But when a fully-featured product such as NetNewsWire is suddenly free, it effectively reduces the value of other for-pay software products. "Why should I have to pay for quality tools, when this quality tool is free?", the thinking goes. When something is given away for free, its perceived value is lowered. If software is treated as valueless, it becomes much, much harder to sell. One need only look at a quote from news coverage discussing the move to freeware to see this in action: "And thank [NewsGator] for this grand gesture...While you're at it, do pray that a few other software companies get inspired by this move and follow suit."

So very true. He continues with the browser battle and how when Microsoft started, it was far far behind Netscape, but they threw their weight (and money) behind IE and it became the best choice - and eventually the standard. Then Netscape withered and died. The competition stopped making IE better, it languished, became a petty dictator of the net, and we have what we have today. Power corrupts, and absolute power corrupts absolutely.

Yet at the same time we see that Brent is happy about this decision. Saying in his post:

But I will say that, for me personally, this is a dream come true. Every developer wants to be able to work on the software they love, make a living at it, and give it to the world for free.

Usually you get to pick two out of three — if you’re lucky. Me, I get all three.

So what we have is a situation where the goals of an individual (Brent) collide with the goals of the Mac Development community - possibly even the long-term goals of the Mac Users themselves.

I remember that back in the late 1980s, Prodigy originally test marketed their online service for free - just get the free disk, load it on and get five free accounts. Everything was paid for from the ad revenue. Interesting fact. No one wanted to use it. So they charged $12.50/month for the same thing and they had people signing up by the droves.

Why?

Perceived value. If you don't have to pay for it, can it really be that good? I mean really, if it's free, what's it worth? But if they are asking for money for it, I'll give it a look and see if it's worth that to me.

The decision about NetNewsWire is not that big a deal. Yes, it's going to change the market slightly, and yes, it probably means that RSS readers are going to be driven by NetNewsWire for as long as it's available. But the same it true for TCP/IP stacks. You used to have to pay extra for them on PCs... then they became free downloads... then they were part of the OS... then they were configured automatically out of the box.

The point is, yes, things change. Industries are born, grow, age (hopefully gracefully) and then die. It's life. No matter if it's people, animals or software. And life goes on. Always has... always will.

It couldn't happen to a nicer bunch of guys. From ArsTechnica today:

A study conducted last year by the Electronic Frontier Foundation and independently verified by the Associated Press revealed that Comcast interferes with BitTorrent and Gnutella sessions by sending TCP "reset" packets to users. Despite the growing body of unambiguous evidence, Comcast still denies allegations that it specifically targets BitTorrent. Comcast vice president David Cohen told us that the company's traffic control mechanisms conform to the FCC's definition of "reasonable network management" practices (which are allowed by the FCC), but critics don't agree that Comcast's management is anything "reasonable." The FCC today indicated that "reasonable" practices should be transparent.

I don't mind a company making a buck, but when they say they're selling access to the internet only to actually allow access only to those things they want you to have access to, then I think I should get a big discount - and disclaimer. Also, I should be able to opt-out and get access to everything.

Clearly, I'm not the only one who thinks so, and I'm glad that the FCC is going to get involved. The problem is, with so much having already been done, it's not an issue of if they are doing it, it's a question of what will be the outcome of the "investigation". Again, from Ars:

The FCC's investigation will have significant implications for the network neutrality debate. Proponents of network neutrality legislation believe that the FCC's current standards and the manner in which those standards are enforced is largely inadequate to deter Internet service providers from engaging in abusive network
manipulation.

Will there be net neutrality, or will the practice of filtering be given a green light? I'm hoping that we get the former. I'm worried we'll get the latter.

Comcast announced that they will be rolling out hardware enabling up to 160 Mbps over cable. Given my recent experiences with Comcast's port blocking/interception, it's hard to believe that they'll use this for anything more than a token speed bump to get it on par with FiOS from Verizon. As much as I'd like to believe it, I just don't believe this will mean I'll be able to get even 20 Mbps symmetric or even 20/5. It's just not in their business model to allow their customers to use the connection for what they want.

But for those wanting to download stuff from iTunes or updates to applications, I'm sure this is going to be a significant upgrade. Thankfully, Chicago is a major Comcast market, but if they are trying to go up against FiOS, they won't put the new hardware here first.

I guess the best news I can read from this is the fact that new infrastructure is coming to Comcast, and if regulation follows - or they ge their act together, it'll mean better connectivity for me. That's at least something to look forward to.

This morning I came in to find that the CPU usage on the server's kernel is at an all-time high! OK, thankfully, it's still running, but it's not running the way I had expected it to be running this morning. I had done too much optimization and that lead to backups in the production system. Unfortunately, there's no way to test the server like it's used in production other than to put it into production. There are clients all over the globe, and it's just not possible to load the dev server to the same level. So what was looking like it was working very well in development, wasn't working so well in production.

Not a big surprise, but it's something I need to go back and mess with. Basically, the system was balanced - no one component that much faster than the others. When I made the pub/sub system faster, that allowed the price ticks to move faster, and the clients had to keep up. Well... what if they're on the other side of the globe and the WAN line just isn't fast enough to get them the bandwidth to the server that they need to keep up? Yup, backups.

It's not horrible, as the server becomes a self-regulating system, but it's annoying that there's so little I can do to check these kinds of interactions.

So today I'm backing out a few of the changes to the pub/sub system and putting in a few more subtle changes to the pub/sub system. We'll put in an index to the primary feeder queue so it's faster to look up duplicates (if it's configured that way)... we'll also have a little better look at the outgoing client queues now that they hold the instruments again (back to a few large queues)... all these will make the code more understandable and not quite as high performance, but in the end, it'll balance the system better.

I'm still going to try to get the tick rate up, but I may be able to do that with these changes installed. We'll have to see how the tests go.

UPDATE: OK... I'm getting the code back in balance. There are a lot of things I can do to make it work less and that's what I'm focusing on at this time. It's looking much better in development, but the acid test is, as always, production tomorrow.

I've made some pretty dramatic improvements in the server's price tick processing in the last few days. Problem is, it leaves me feeling... well... unsatisfied. The problem is that there's no end to it. Maybe I made it faster, but there's still a bottleneck, and that would be nice to remove, if possible. It's a never-ending stream of minute fixes that makes it a little faster, but it's not really faster enough to justify the time spent on it.

So I have to stop when it's really not done - it's just done enough. That's what leaves me feeling very unsettled. I'm a very completion-orientated person. I like to create and fix things, but performance tuning is just getting a little bit more out of something that's already working. Oh, it's nice to see a factor of four or more, but after those first few big gains, the next gains are almost always much smaller, and continuing to be smaller as you continue to work on the code.

I've just got to come up with a self-imposed criteria for stopping this work. Then, when I reach it, I can emotionally let go of this phase of the project and move on to the next. Good plan.

Today I noticed that my previous efforts to speed up the tick flow through the server helped, but there were still times when the CPU usage climbed to those levels that looked like there was a large look-up being done. I checked the logs and about the same time those were happening, I was getting large ClientProxy queue sizes. This makes sense - if the outgoing queues to the clients start to grow, the search time for a duplicate (they must be unique) would go up, and that would slow down the injection rate, and in turn make the queue grow even more.

So... I needed to make sure I was using the most efficient storage as possible. I did a little reading and found that std::set is an RB tree, and the find() uses this to get to the elements as quickly as possible. No room for improvement there, unfortunately. But I could not give up, there had to be a solution.

And while it wasn't earth-shaking, the solution was fun. Rather than try to get more efficient data storage, realize that maybe I was asking it to do the wrong thing. All I needed was to make sure that each Instrument appears only once on each ClientProxy queue. Why not leave it up to the Instrument, then? Rather than have a few large lists in the ClientProxys, have a lot of small lists in the Instruments.

Elegant. Simple. Re-phrase the question.

So what I did was to put a std::set on each Instrument with a protecting mutex on it and then make the ClientProxy tell each instrument that it's going on the queue, and coming off the queue. That way, we can be sure that the same effect occurs, but that we can have a simple queue in the ClientProxy and it doesn't have to look for duplicates.

This change allowed the server to not get into those periods of large queue size. This is good in that the entire system will respond better to the prices. Also, it means that we don't have those conditions where the CPU usage rises and slows down the tick processing. It's a great solution all the way around.

Looks like there's been an update on MarsEdit for a security problem. At about the same time I noticed that WordPress is at 2.3.2, but HostMonster doesn't have it available in it's application support tool. No matter, I don't use the 'drafts' section of WordPress at HostMonster - I use MarsEdit.

Lots of things seem to be updating today... Cool.

I can't think of a better poster child for great development on the Mac than Adium. I've been using it ever since I got an Intel Mac as my old multi-protocol chat client, Fire, didn't work properly on the Intel platform. As I was looking for a solution, I noticed that Adium was looking to be the preferred solution for most Mac folks.

I downloaded it, got all my accounts into it, fiddled with the preferences to get it looking and working like I wanted, and have been happily using it ever since.

Today they released 1.2, and the release notes indicate a ton of bug fixes and localization changes. Not that I had that many problems with it before, but it's nice that they are continuing to work on it. If you need something like this, get it - you won't be sorry.

Today I was blown away by a performance boost I got from not cacheing resultant data. When you look at it, it's very logical why it's faster, but it took me an hour or so to get to the point of even questioning the cache as a bottleneck in the processing.

For the past day or so, I've been looking at the speed my server is processing incoming price events (ticks). It's a lot better than it used to be, but there were times when it took minutes to get a price through the server. It didn't seem to be a locking problem. It was just that some ticks took much longer than others to process. So I started digging.

The first place to look was in the processing of the incoming prices. The prices come from the feeder apps, go onto a compression queue (updates new data, but retains queue order) and then are pulled off by a thread and sent through the system. My first thoughts were that the queue was holding onto the prices longer than needed - primarily because the processing thread wasn't getting back around to the queue quickly enough. I thought it might be in the thread synchronization of the pushing and popping threads, but after a little bit of experimentation, I realized that this wasn't the case. The problem was elsewhere.

Next, I turned my attention to the updating of the prices. There's nothing in the code that would block the updates so drastically. I looked and looked further downstream, but in the end, this was just not where the problem was.

Finally, I started looking at the general CPU usage of the kernel process - the kernel of the server - not the OS. This lead me back to something I've know about for months, and just haven't gotten around to fixing: when there are clients attached to the server, the CPU usage goes way up - primarily due to the messaging to those clients of what's changed in the server. Unfortunately, I couldn't get good handle on why this was happening - until this morning.

There are two parts of the client communication with the server: the outgoing notification that a particular instrument has changed, and the request/response with the client for the complete detailed information about a particular instrument. I 'short circuited' each of these in turn, and found that the CPU usage was really attributed to the notifications and not the queries. I was a little surprised. Then I started thinking about the details of the notifications.

Originally, when the server would update an instrument's data, it would ask each connected client if they were interested in hearing about this guy's update. This then migrated to having a client proxy on the server side asked the question, which was an important step, but it was still a little pokey. The next step was to have the client proxy remember the answers for each instrument, and going on the idea that once an instrument is 'interesting', it's very unlikely that it will stop being 'interesting', and so we cached the 'interesting instruments' and first checked if the updated instrument was in the list of 'interesting instruments' before doing the more thorough check.

The list was a simple STL std::set of strings, but when the client had been connected for a while, it was possible that there would be literally thousands of 'interesting instruments' in the list. Scanning this list each time seemed to be the problem, so that's where I went to work.

The goal was to make checking the individual instrument so fast and easy that there would be no way a cache was going to be faster than simply checking this guy each time. The first step was to realize the fact that all the clients that connect to the server really are looking for all positions, so I made a few methods on the instrument class that would simply and very quickly tell me if there were any positions on that guy, and not bother with the more complete checking that was done when we wanted only a subset of positions from the server.

The second modification was in putting an STL std::set where I had a simple list - which had a linear sort, for those instruments we wanted to be notified of - even with no positions on them. When I put in these changes to the system and restarted I was amazed to see that fully 66% of the CPU usage of the 'connected' server was gone. Amazing.

The CPU usage of the kernel now very clearly tracks the incoming price events. It's at least 2 to 3 times faster for prices through the server, and it's simpler, smaller (no cache of instruments), and can now move more prices than ever before. Really very cool.