What's it all about, Alfie?

The first work day after Thanksgiving is always a great day for me. This year I was a day late, but I got there... it's Christmas Music Time! Yup, that's when I pull out the Christmas music playlist on my iPhone and get to listening to some of the greatest music of the whole year. It's also the time of year when we all are a little nicer, a little kinder, and everyone tries to put on their best face. It's a great time of year.

I remember being in a choir room of a rather big church in Indianapolis when I was in high school, and there was a picture on the wall with a quote:

For the common things of everyday,
God gave man speech in a common way.

For higher things men think and feel,
God gave man poets, their words to reveal.

But for heights and depths no words can reach,
God gave man music, the soul’s own speech. - Anonymous

Every time I read it I tear up. It's perfect.

For the last day and a half I've been working on re-writing my C++ IRC Client code to use boost's asio, and it's been pretty interesting. I will say there are a lot of plusses to using the boost socket functionality - even over my own socket library (imagine that!).

First, it's got the complete asynchronous mechanism for sending and receiving - you just have to love that. Also, they have done a wonderful job in making it all very rational and sane. Asynchronous methods perform as you'd expect, and have remarkably similar signatures to their synchronous counterparts. It makes writing with the classes very simple. Clearly, a lot of thought has gone into this stuff, and that's really nice to see.

Secondly, there's not the need for all the threads that I had in my old code. Primarily because of the single io_service thread that boost asio uses for all async operations. This really is a great timesaver. You can easily have multiple threads sending out chats to the IRC server with the async writer. Very slick.

Finally, the resulting code size is much smaller. That's more of a consequence of the other two, but the payoff can't be understated. Less code means less maintenance, less cost, and less time. You just can't beat that.

So I have it all done, but I haven't been able to test it yet as we don't have an IRC server up and running - yet. It's been discussed, and maybe today they'll see that I've gotten code ready, and they'll put one up. It's not hard - just takes a little time, that's all. But the benefits will be enormous. I look forward to getting the server going, testing my code, and integrating this into my TickerPlants and other libraries. It's just an amazingly powerful tool for support and problem solving.

In the past, I've used IRC in my applications to great utility. I created a simple framework that allowed me to have each application instance "pose as" a "user" on IRC, and when the application was running, you could see this in the chat rooms, and you could interact with it by as complex, or as simple, a means as you, the application designed, wanted. The protocol is simple, it's fast, and there's very little administration to the system.

SNMP is not so simple, but it's far more common in the monitoring and control of applications. The question is really: Is the complication worth it? I'm not at all certain that it is.

Several months ago, the decision was made to use Jabber as opposed to IRC. I went along with it as that's the right thing to do. But it's not the choice I'd have made. The ircd is simple, fast, and with a little additional code, you can make it log everything. From that point, you don't have any concerns about compliance, and you're free to use it as needed.

It seems that the powers that be are re-evaluating the Jabber solution as it's a little more complex in it's implementation, and the goal here is not to have something really complex, it's to have something really easy. So I'm hoping that I'll get to finish my IRC client based on the boost asio work. If so, it should be exceptionally fast and easy to use. Both are the hallmarks of a great utility.

I hope I get to see it come to pass.

The complexity of SNMP just seems like massive overkill.

I read a lot about this over the weekend, and the more I read the more it made me smile.

The foundation upon which this project is base comes from the GNUstep community, whose origin dates back to the OpenStep standard developed by NeXT Computer Inc (now Apple Computer Inc.). While Apple has continued to update their specification in the form of Cocoa and Mac OS X, the GNUstep branch of the tree has diverged considerably.

Yeah... they've diverged only in that Apple's Cocoa has moved forward and GNUstep is still using the OPENSTEP guide as it's reference point. And there's nothing wrong with that. OPENSTEP is fantastic... it's just that Apple has decided to keep moving and GNUstep hasn't. Sony isn't content on sitting still either:

We depart somewhat from the GNUstep adherence in that our goal is to thoroughly modernize the framework and optimize it to target modern consumer electronic (CE) devices. These modern conveniences include such features as touch displays and 3D graphics.

I use WindowMaker as my X11 desktop of choice. It's all about GNUstep. I think it's fantastic that Sony is picking this up. It means that GNUstep is going to be getting a much needed shot in the arm with all the added interest. Really great news.

NeXT was right. Took a long time, but they were right. Fantastic.

This afternoon I've been tracking down a good solution to a nasty threading problem. This part of my ticker plants is the UDP receiver and it tries to get the UDP datagrams off the socket and into a buffer as fast as possible. To that end, I've got a single-consumer, single-producer, lockless FIFO queue that should be thread-safe as the 'head' and 'tail' are volatile and there's only one thread messing with one of these guys at a time.

But that's just the theory. Here's what the code looks like:

  template<typename Element, uint32_t Size>
  bool CircularFIFO<Element, Size>::push( Element & item )
  {
    uint32_t  nextTail = increment(tail);
    if (nextTail != head) {
      array[tail] = item;
      tail = nextTail;
      return true;
    }
 
    // queue was full
    return false;
  }
 
 
  template<typename Element, uint32_t Size>
  bool CircularFIFO<Element, Size>::pop( Element & item )
  {
    if (head == tail) {
      // empty queue
      return false;
    }
 
    item = array[head];
    head = increment(head);
    return true;
  }

Here's what happens: I'll be running just fine and then the call to pop() will return true and because of that, the value (a pointer) will return as something. This presents a real problem. If it returns a NULL, that's easy to deal with. Problem happens when it returns junk.

Ideally, it wouldn't return a NULL or junk, but coding for that has turned out to be harder than I thought. First, I can just check for a NULL or what I think of as "junk" data, and not delete that pointer, but what happens when it returns "junk" that's not fitting my pattern of "junk"? Well... I'll delete it and BAM! SegFault.

Not easy.

I believe the problem is one of compiler preference. The data in the class is defined as:

  volatile uint32_t head;
  volatile uint32_t tail;
  Element   array[Capacity];

where the lack of the volatile keyword is the big deal here. What I need to do is to make the data look like:

  volatile uint32_t head;
  volatile uint32_t tail;
  volatile Element   array[Capacity];

and then correct all the castings in the code to make them work properly.

I've got something to compile that looks like:

  template<typename Element, uint32_t Size>
  bool CircularFIFO<Element, Size>::push( Element & item )
  {
    uint32_t  nextTail = increment(tail);
    if (nextTail != head) {
      array[tail] = *((volatile Element *)(void *)&item);
      tail = nextTail;
      return true;
    }
 
    // queue was full
    return false;
  }
 
 
  template<typename Element, uint32_t Size>
  bool CircularFIFO<Element, Size>::pop( Element & item )
  {
    if (head == tail) {
      // empty queue
      return false;
    }
 
    item = *const_cast<Element *>(&(array[head]));
    head = increment(head);
    return true;
  }

We'll have to see how this runs.

There are books about the subject. There are people that swear they are the Devil's handiwork. They are the source of much debate, and in my opinion, they are much maligned. They are pointers.

Pointers in C and C++ code are as important as any other language construct. They are an essential tool in the arsenal of a software developer, and they have to be mastered. Yes, if put in the wrong hands pointers can be a very dangerous thing. But so can knives, guns, and explosives. But that doesn't mean that you want to start digging out tons of rock with shovels because explosives are "dangerous". It means that you have to make sure you have skilled technicians that understand the way in which to safely handle the tools of the trade.

Same with pointers. You have to be skilled and trained in their use, but that doesn't mean you have to spend a decade learning how to use them. You just need to keep an eye on them and remember to have strict rules in their usage and lifecycle.

For example, if you're using pointers on a queue, make sure that your method comments indicate when you're taking ownership of the pointer, and when you're passing ownership to the caller. Then it's clear. Simple.

Pointers are really no different than any resource you have to track - pooled database connections, threads, all these offer the same level of difficulty as the lowly pointer. But all are essential tools of the skilled software developer.

Fear not the pointer. It is your friend.

Today I spent the nearly the entire day trying to find the remaining memory leak(s) on my ticker plant. Again, only a few were being effected and again, I focused on the code they exclusively have. Once again, this was a complete waste of time as it wasn't in the exclusive code but a very odd little bug in the shared code.

When I'd struck out after several hours of testing, I decided to start at one and and sweep the code with a fine-tooth comb. Starting at the incoming UDP feed, I cut the rest of the app off and checked the memory usage. Stable. Good. Now let's add in the next step. Leak? Ha... let's fix him. Continue until we have the leak spotted.

The first problem may not have been a leak, but it was an unreliable memory usage pattern. The UDP datagrams came in the boost asio socket and I placed the datagram into a std::string with the time as microseconds since epoch into a simple stl::pair. This was then placed into a simple std::deque. Something like this:

  typedef std::pair<uint64_t, std::string> TaggedDatagram;
 
  CircularFIFO< TaggedDatagram, 100000 >   mStaging;

where the CircularFIFO is a single-producer, single-consumer, lockless, circular FIFO buffer for fast pushes and pops of the data coming off the wire.

The problem with this design is that we have to create the std::string every time anyway, and the storage of this structure is very unpredictable. What I decided to do was to switch from the stack to the heap and change the structure:

  typedef std::pair<uint64_t, std::string *> TaggedDatagram;
 
  CircularFIFO< TaggedDatagram, 100000 >   mStaging;

Now it's basically two 64-bit ints and the heap will reclaim the memory as needed. This was a nice addition, but it wasn't the final problem.

It was at the end of the day, but thank goodness that I found it. It turns out that the ZeroMQ send() method was the culprit. Normally, the ZeroMQ has been very nice for me. Why these messages caused problems, I have no idea, but it's the one method and nothing else.

I know they are working on a new version (2.1) with the latest OpenPGM included, and that will be nice to see. Tomorrow morning when they are all online, I'll ask what the story is on the release of 2.1. Until then, I'll deal with the smaller, but still annoy leaks.

Whew!

[11/24] UPDATE: I talked to the ZeroMQ guys this morning and they say the release of 2.1 is scheduled for this week. Nice. I'll get it early next week and try it out.

Today has been spent trying to track down a very non-obvious memory leak in my ticker plant code. I've been watching it run over the last few days - fixing little things as I see them, and each time the app runs longer and better. Good... we're moving in the right direction.

But it's odd that a few of my ticker plants have a problem with a growing memory footprint. Very odd indeed. So I started digging into these exchange feeds. My first mistake was to ignore all common functionality - after all, if it's common with the exchange feeds that aren't leaking, then it can't be that code. Right?

Wrong.

What I found was that I needed to be exceptionally careful even when using the compare-and-swap atomic operations. It's possible that two threads, on two CPUs, are doing their own thing on that one variable, and if it's only in their cache, it's possible that there might be a time when the caches are updated and the main memory isn't. This could cause me to "loose" a message, and leak memory.

What I did was put a simple boost spinlock mutex on the value and then we got a lot more stability. That's good.

Unfortunately, that's not the end of the story... but it's the end of the day. A long day for a little solution. Tomorrow I'll have to see what the remaining problems are.

Well... it's here! I saw on the familiar sources that iOS 4.2 was released to iTunes 10.1 and initially I was pretty disappointed that they didn't release 4.2.1 - as I know they were working on it before the release, but when I installed 4.2 it was actually reported as 4.2.1!

Sweet.

I haven't used the new stuff, but I'm sure I'll get around to it. I'm guessing that with the news that iOS 4.3 coming out next month, I'll probably find out in that time-frame. I'm pretty busy now with work and the house.

Who knows... maybe I'll get an Apple TV? Might be fun.

Yesterday I worked on the Broker's C++ Service adapter to allow for these direct dial connections with the addition of the boost asio acceptor listening on a ephemeral port and putting that in the registration message to the Broker. That looked to be pretty good, and so today I worked on adding the client-side of the protocol.

I went with a simple scheme - if the user explicitly asks to locate() a service, then we'll ask the Broker for the direct dial location, and save it. Then, for every connection to that service, we'll have a pool of connections to use that connect directly to the service and do not go through the Broker.

There's a hitch - if the Broker is trying to load balance my calls, then I'm going to mess him up as I am creating a pool on the one and only location he's giving me. This isn't great, but the alternative is to not have any pooling of connections and ask the Broker for the location each time, and then create a connection and then throw it away.

It's possible, but I decided against it. I think there's more value in this pooling, but we'll see. Maybe I'm all wet and the real value is in asking the Broker each time. We'll have to see how it plays out in the usage patterns.