What's it all about, Alfie?

Here's something that happened to me today that makes me a little scared of the vendor relationship I have with our price feeder. First, it's another group in the Bank, so it's not like they are a real vendor - and for that I'm eternally grateful. But even so, today I saw a problem in the feed and I was able to figure it out faster than they were. More to the point, when I explained it to them, they had to take a step back and think about it before they realized what the problem was.

It's all about trying to be clever and efficient - which normally, I'd be in support of, except where it causes problems for others. Get it right then get it fast. That's the way to do it. Unfortunately, they have it right, but using it then becomes very difficult as what they send (only the diffs from the last tick) isn't really enough to do the job of interpreting the data.

There are two fields of importance in this example: the price and the time it was sent from the exchange. Let's say they aren't properly mapping out the time from the Reuters FIDs, and so all I get is a date - no time. When a new tick comes in, the date hasn't changed and the time hasn't been mapped, so the date/time hasn't changed from it's last value, and so it's not sent. The price is, but without the time, when was this price issued? Today? Is it current at all?

Had they sent the date/time every time then this would not be an issue - but they can't do that - inefficient. So because they want to only send the diffs we have to put in a lot of extra logic to figure out what it was they meant to send.

It only gets more complex when you think that a "tick" may come form the exchange with a date/time of 'today' and a price of 0.0 - meaning it's not really a tick. So I properly ignore that. But if the date/time is never resent, then all subsequent ticks don't have an 'old' date/time to run with. As I said... it's problematic.

But it's the fact that this doesn't seem to represent a problem to them that bothers me. You should be in the mode of helping your customers, not making it harder for them. In this case, they were definitely making it harder by not being a little less efficient and sending the date/time again. Sigh...

One of the things I like to do when I've made changes to several files in a CVS directory - and haven't yet checked them in, is to make a tarball (a .tgz file) of them so that I can move them to another machine for testing. For example, working on cross-platform code, it's nice to get it working on one platform and then move it to the next and verify it before checking it in. This way all the changes for the modification are in the one CVS check-in.

I wanted to make a simple alias for TCSH that did this on my Mac. I was doing it all the time and it just seemed smart to finally do this right once and then just use it from here on out. The problem I got into were these "Apple Dot" files (._file) that tar was including in the tarball even though they weren't listed in the list of files to include - nor could I exclude them with the command options.

So I did some googling and found the COPYFILE_DISABLE environment setting in Leopard. If this is set to 'true', then these Apple Dot files will not be copied. But, since I didn't want to make a big change to all my shells, I decided to simply make it defined for the one command.

Finally, then, here is the alias:

  alias tcu 'setenv COPYFILE_DISABLE true; tar zcvf \!* `cvs -q update |
      grep -v \^\? | sed -e "s/^[AM] //g"`; unsetenv COPYFILE_DISABLE'

(it's all one line, of course)

Yesterday I was starting to work on my next little project - a price injector from our price feed to a new vendor app that we're integrating.

Another developer is building a nice C++ wrapper around the proprietary messaging API. It's not terribly complicated, but he's going to wrap it so that we can make a message system that we can use within the Shop without having to worry that we're overloading the vendor's product. So with that part done, or at least getting done, I can focus on the application that will takes prices from the price source (CacheStation) and then send it to the app via our message bus.

Configuration for these apps is something I've worried about over and over for each new app. You want something that's easy to edit... something that's got a central location for easy maintenance... yet not introduce more system dependencies than you absolutely have to. Most of the time this is a configuration file, but I have used configuration databases at times and while I like a lot about that, I think the better solution is to have a flat file that can hold complex objects and structures and then make it possible to have this reloaded on-demand or on a timer and in that way you simply edit the file and the changes will be read in.

So in doing this I was thinking I'd mimic the BKHashMap and BKHashTree in BKit. Both have been very successful for me, and so I started looking at the hash_map in the SGI extensions to STL that have recently been adopted by C++. The problem was that in order to use a CKString as a key in the map (or an element in a hash_set) I needed to specialize the hash<T> template. It was pretty simple once I looked at the header for hash_fun.h:

  /*
   * While I'm not a big fan of the way they have chosen to put in the hash_map
   * and hash_set templates - I think it's too clunky. They should have made
   * it possible to put a hash operator or method on the class, but that's an
   * opinion. Anyway, given how they chose to do it, this template specialization
   * allows us to create sets and maps using the hashing function below.
   */
  namespace __gnu_cxx
  {
    template<>
    struct hash<const CKString &amp;>
    {
      size_t operator()(const CKString &amp; arg) const
      {
        return __stl_hash_string(arg.c_str());
      }
    };
  }

With this, I can create the has_map by simply saying:

  #include <ext>
  #include "CKString.h"
 
  __gnu_cxx::hash_map< CKString, int, __gnu_cxx::hash<const CKString &amp;> > map;
 
  map["when"] = 10;

Once I have the hash_map I can build the CKHashMap and CKHashTree classes. These would then form the basis of the file-based preferences system for the application. I've already done the code for BKHashMap to read and write plists of objects, so it'd be easy for me to make nicely structure-rich preferences files and then simply read them in and manipulate them.

First, this morning NetNewsWire 3.1 was released, with what was announced as primarily a performance-improvement based release. This is great news. Not that NetNewsWire was a pig, or anything, but when they take the time to work on the code code of an app (like I have been doing to the server) it shows a commitment to the app at a level you don't see when you're just trying to add new features to make users happy. So, Great Job to Brent S. on the release. I'm running it already.

The next bit of news that comes from Brent and the NewsGator folks is that NetNewsWire is now freeware. That's right - totally free. All of the NewsGator 'personal' products are free. They look at this as the way to get mind-share and name recognition and it's probably going to work. There are a lot of people using something like NetNewsWire for personal reasons, and if they see a professional app from the same guys, they'll be more likely to give them the nod - all else being equal.

And at the same time, there's a great article on Rogue Amoeba about the devaluation of all software for a platform when a good piece of software is available for free. The posting says:

There's certainly a place for free software. But when a fully-featured product such as NetNewsWire is suddenly free, it effectively reduces the value of other for-pay software products. "Why should I have to pay for quality tools, when this quality tool is free?", the thinking goes. When something is given away for free, its perceived value is lowered. If software is treated as valueless, it becomes much, much harder to sell. One need only look at a quote from news coverage discussing the move to freeware to see this in action: "And thank [NewsGator] for this grand gesture...While you're at it, do pray that a few other software companies get inspired by this move and follow suit."

So very true. He continues with the browser battle and how when Microsoft started, it was far far behind Netscape, but they threw their weight (and money) behind IE and it became the best choice - and eventually the standard. Then Netscape withered and died. The competition stopped making IE better, it languished, became a petty dictator of the net, and we have what we have today. Power corrupts, and absolute power corrupts absolutely.

Yet at the same time we see that Brent is happy about this decision. Saying in his post:

But I will say that, for me personally, this is a dream come true. Every developer wants to be able to work on the software they love, make a living at it, and give it to the world for free.

Usually you get to pick two out of three — if you’re lucky. Me, I get all three.

So what we have is a situation where the goals of an individual (Brent) collide with the goals of the Mac Development community - possibly even the long-term goals of the Mac Users themselves.

I remember that back in the late 1980s, Prodigy originally test marketed their online service for free - just get the free disk, load it on and get five free accounts. Everything was paid for from the ad revenue. Interesting fact. No one wanted to use it. So they charged $12.50/month for the same thing and they had people signing up by the droves.

Why?

Perceived value. If you don't have to pay for it, can it really be that good? I mean really, if it's free, what's it worth? But if they are asking for money for it, I'll give it a look and see if it's worth that to me.

The decision about NetNewsWire is not that big a deal. Yes, it's going to change the market slightly, and yes, it probably means that RSS readers are going to be driven by NetNewsWire for as long as it's available. But the same it true for TCP/IP stacks. You used to have to pay extra for them on PCs... then they became free downloads... then they were part of the OS... then they were configured automatically out of the box.

The point is, yes, things change. Industries are born, grow, age (hopefully gracefully) and then die. It's life. No matter if it's people, animals or software. And life goes on. Always has... always will.

This morning I came in to find that the CPU usage on the server's kernel is at an all-time high! OK, thankfully, it's still running, but it's not running the way I had expected it to be running this morning. I had done too much optimization and that lead to backups in the production system. Unfortunately, there's no way to test the server like it's used in production other than to put it into production. There are clients all over the globe, and it's just not possible to load the dev server to the same level. So what was looking like it was working very well in development, wasn't working so well in production.

Not a big surprise, but it's something I need to go back and mess with. Basically, the system was balanced - no one component that much faster than the others. When I made the pub/sub system faster, that allowed the price ticks to move faster, and the clients had to keep up. Well... what if they're on the other side of the globe and the WAN line just isn't fast enough to get them the bandwidth to the server that they need to keep up? Yup, backups.

It's not horrible, as the server becomes a self-regulating system, but it's annoying that there's so little I can do to check these kinds of interactions.

So today I'm backing out a few of the changes to the pub/sub system and putting in a few more subtle changes to the pub/sub system. We'll put in an index to the primary feeder queue so it's faster to look up duplicates (if it's configured that way)... we'll also have a little better look at the outgoing client queues now that they hold the instruments again (back to a few large queues)... all these will make the code more understandable and not quite as high performance, but in the end, it'll balance the system better.

I'm still going to try to get the tick rate up, but I may be able to do that with these changes installed. We'll have to see how the tests go.

UPDATE: OK... I'm getting the code back in balance. There are a lot of things I can do to make it work less and that's what I'm focusing on at this time. It's looking much better in development, but the acid test is, as always, production tomorrow.

I've made some pretty dramatic improvements in the server's price tick processing in the last few days. Problem is, it leaves me feeling... well... unsatisfied. The problem is that there's no end to it. Maybe I made it faster, but there's still a bottleneck, and that would be nice to remove, if possible. It's a never-ending stream of minute fixes that makes it a little faster, but it's not really faster enough to justify the time spent on it.

So I have to stop when it's really not done - it's just done enough. That's what leaves me feeling very unsettled. I'm a very completion-orientated person. I like to create and fix things, but performance tuning is just getting a little bit more out of something that's already working. Oh, it's nice to see a factor of four or more, but after those first few big gains, the next gains are almost always much smaller, and continuing to be smaller as you continue to work on the code.

I've just got to come up with a self-imposed criteria for stopping this work. Then, when I reach it, I can emotionally let go of this phase of the project and move on to the next. Good plan.

Today I noticed that my previous efforts to speed up the tick flow through the server helped, but there were still times when the CPU usage climbed to those levels that looked like there was a large look-up being done. I checked the logs and about the same time those were happening, I was getting large ClientProxy queue sizes. This makes sense - if the outgoing queues to the clients start to grow, the search time for a duplicate (they must be unique) would go up, and that would slow down the injection rate, and in turn make the queue grow even more.

So... I needed to make sure I was using the most efficient storage as possible. I did a little reading and found that std::set is an RB tree, and the find() uses this to get to the elements as quickly as possible. No room for improvement there, unfortunately. But I could not give up, there had to be a solution.

And while it wasn't earth-shaking, the solution was fun. Rather than try to get more efficient data storage, realize that maybe I was asking it to do the wrong thing. All I needed was to make sure that each Instrument appears only once on each ClientProxy queue. Why not leave it up to the Instrument, then? Rather than have a few large lists in the ClientProxys, have a lot of small lists in the Instruments.

Elegant. Simple. Re-phrase the question.

So what I did was to put a std::set on each Instrument with a protecting mutex on it and then make the ClientProxy tell each instrument that it's going on the queue, and coming off the queue. That way, we can be sure that the same effect occurs, but that we can have a simple queue in the ClientProxy and it doesn't have to look for duplicates.

This change allowed the server to not get into those periods of large queue size. This is good in that the entire system will respond better to the prices. Also, it means that we don't have those conditions where the CPU usage rises and slows down the tick processing. It's a great solution all the way around.

Today I was blown away by a performance boost I got from not cacheing resultant data. When you look at it, it's very logical why it's faster, but it took me an hour or so to get to the point of even questioning the cache as a bottleneck in the processing.

For the past day or so, I've been looking at the speed my server is processing incoming price events (ticks). It's a lot better than it used to be, but there were times when it took minutes to get a price through the server. It didn't seem to be a locking problem. It was just that some ticks took much longer than others to process. So I started digging.

The first place to look was in the processing of the incoming prices. The prices come from the feeder apps, go onto a compression queue (updates new data, but retains queue order) and then are pulled off by a thread and sent through the system. My first thoughts were that the queue was holding onto the prices longer than needed - primarily because the processing thread wasn't getting back around to the queue quickly enough. I thought it might be in the thread synchronization of the pushing and popping threads, but after a little bit of experimentation, I realized that this wasn't the case. The problem was elsewhere.

Next, I turned my attention to the updating of the prices. There's nothing in the code that would block the updates so drastically. I looked and looked further downstream, but in the end, this was just not where the problem was.

Finally, I started looking at the general CPU usage of the kernel process - the kernel of the server - not the OS. This lead me back to something I've know about for months, and just haven't gotten around to fixing: when there are clients attached to the server, the CPU usage goes way up - primarily due to the messaging to those clients of what's changed in the server. Unfortunately, I couldn't get good handle on why this was happening - until this morning.

There are two parts of the client communication with the server: the outgoing notification that a particular instrument has changed, and the request/response with the client for the complete detailed information about a particular instrument. I 'short circuited' each of these in turn, and found that the CPU usage was really attributed to the notifications and not the queries. I was a little surprised. Then I started thinking about the details of the notifications.

Originally, when the server would update an instrument's data, it would ask each connected client if they were interested in hearing about this guy's update. This then migrated to having a client proxy on the server side asked the question, which was an important step, but it was still a little pokey. The next step was to have the client proxy remember the answers for each instrument, and going on the idea that once an instrument is 'interesting', it's very unlikely that it will stop being 'interesting', and so we cached the 'interesting instruments' and first checked if the updated instrument was in the list of 'interesting instruments' before doing the more thorough check.

The list was a simple STL std::set of strings, but when the client had been connected for a while, it was possible that there would be literally thousands of 'interesting instruments' in the list. Scanning this list each time seemed to be the problem, so that's where I went to work.

The goal was to make checking the individual instrument so fast and easy that there would be no way a cache was going to be faster than simply checking this guy each time. The first step was to realize the fact that all the clients that connect to the server really are looking for all positions, so I made a few methods on the instrument class that would simply and very quickly tell me if there were any positions on that guy, and not bother with the more complete checking that was done when we wanted only a subset of positions from the server.

The second modification was in putting an STL std::set where I had a simple list - which had a linear sort, for those instruments we wanted to be notified of - even with no positions on them. When I put in these changes to the system and restarted I was amazed to see that fully 66% of the CPU usage of the 'connected' server was gone. Amazing.

The CPU usage of the kernel now very clearly tracks the incoming price events. It's at least 2 to 3 times faster for prices through the server, and it's simpler, smaller (no cache of instruments), and can now move more prices than ever before. Really very cool.

Today I put the finishing touches on the addition of Sedol, ISIN, and CUSIP identifiers to the server. While this isn't rocket science, it's nice that they were added so easily. All I needed to do was to pull it out from the instrument master database a little bit of code made it variables and sent it to the clients.

The Hong Kong users wanted one, but I knew that the coverage of these identifiers is not 100% on any one identifier set, so it was nice that all three were available in the instrument master. I've worked hard to make sure that I can phase things in without having to make a "big switch" where multiple systems are changing on the same day - today I put the changes in the server, tomorrow I can roll-out the changes to the middle-tier and clients. Nice.

There seems to always be something waiting in the wings. For the last few days I've been noticing a problem with one of my apps and while I was confident I hadn't made any changes, it was possible that it was the exchanges, or the software that I use to pseudo-normalize the data from the exchanges. It's only happened in 2008, so I sent an email to my contact in the group who maintains the code I use and asked him if anything changed.

Interestingly enough, things had. But he didn't find them right away. Eventually, he did see that the new backup policy they implemented in 2008 wasn't properly returning the configurations for the users. This was the source of the problems I'd been seeing.

Great that he found the problem and fixed it, but I'm wondering how many other users of this code were getting bad results and just not knowing about it. It's always important to watch what you're running.