What's it all about, Alfie?

I was looking on the web today for some C++ library that would wrap the erlang interface (ei) that appears very full-featured and yet somewhat cumbersome for a C++ codebase. I looked at several until I found one that looked to be full-featured, well thought out, and complete. I stopped when I found eixx on GitHub. This looks to be very nice. I haven't used it - yet, but if we go forward with the idea of having each service connect into the distributed erlang network and use that as the way to talk to the services, and serve up data.

The real advantage with this scheme is that we no longer have to deal with async I/O on the sockets for the client or the server. It's all encapsulated in the erlang runtime/library and that takes all the responsibility from our hands and puts it in the hands of the guys who made the language. Very nice.

Anyway... it'll be interesting when we move on this. I'm actually looking forward to it.

I've been using Growl for a very long time. I think it's about the most interesting new technology - with style, that I've seen in workstations in the last few years. It's really amazing. Anyway, when I started using Colloquy I was very impressed with the stability, features and style of the IRC client. I've written IRC clients for a while, and this guy is just amazingly nice. Beautiful, in fact.

But there's one issue that I've never figured out - Colloquy supports Growl, but Colloquy 2.3 and Growl 1.2 on Snow Leopard (10.6) just doesn't work well. Both Growl 1.2.1 and Colloquy 2.3 are 64-bit apps, but Colloquy is constantly showing the "Bubbles" Growl notification style - even though I changed the default style. And the notifications fall on top of other notifications. It's like Colloquy 2.3 has a parallel version of Growl. Very odd.

So this morning I did a little digging and found a posting about an error in Colloquy with regards to Growl. I'm not positive, but I think it's related to Growl 1.2.1 being beta at the time 2.3 came out, and 32-bit at that. Then I found this article and it pointed me to the nightly builds of Colloquy.

I chose the latest - Colloquy-5138.zip and BINGO! Everything is working amazingly well now. Very nice. I still have 64-bit apps, and the style is respected as well as the location of the notifications. It's like it's working the way it was meant to. Lovely.

I'm sure they'll push out a new version soon, and when they do, I'll get back on the "released" version of Colloquy, but for now, this is just fine.

Today has been a mixed bag... I did get a little development done on the ticker plants - just a few little things to polish up some things, but I also had hour-long design meetings (not bad), and then multi-hour-long debugging meetings where there was no hope of actually finding a problem because the developers had so little experience developing on linux that there was no hope of a successful test, and they really just needed to understand the proper way to edit/compile/test code.

The design meeting was really kind of interesting. We have The Broker, and it's been an erlang process, and a Java process, but it's always been centralized. I mentioned today that I wondered why it wasn't using Distributed Erlang, and let the entire brokerage system sit on all the machines and handle everything in a more distributed manner. Let erlang handle the registration, and the message passing. We can use J interface (Java library) and el (C library) to make our server applications appear as distributed erlang nodes. This makes it much easier to do all the things the Broker does.

Sure, the clients still can use sockets to connect to the Broker - or should I say A Broker, where we can have one running on each server in the server room. The client (defined as processes that aren't nodes in the distributed erlang system) can then use the socket interface to connect to a Broker and work as it always has, but the wrinkle is that it's only really brokering his traffic. This gets rid of a lot of the problems we had faced in the past - and went to great lengths to try and solve.

So much gets easier if the main components of the system are all distributed erlang nodes. Very nice solution to the problem.

The debugging session was only slightly productive, and ultimately disappointing. The code was compiled on CentOS5 and run on Ubuntu 10.04.1 - I'm not at all surprised that things broke. Far too different a version of libc, gcc - everything, really. We couldn't even get the code to build on Ubuntu. They need to step back, build the code on the box they are going to run it on, and then move forward.

It's slow going sometimes.

This morning I saw that Google Chrome dev 11.0.696.12 is out, and it's got a good number of fixes - mostly UI components and front-facing issues, but that's OK too. Glad to see it. One thing I didn't expect as a brand new icon - seems the Googlers are going for a more geometric look and less of the shiny plastic look.

Good for them. I like it.

Focused on an interesting problem today. In the last few weeks, I've done a lot of re-writing on the UDP receiver in my ticker plant to get it better, faster, etc. And one of the things I've noticed is that I was accumulating, but not logging, dropped messages from the exchange. Now this is a serious issue because I'm looking at both the A and B sides from the exchange - they are meant to be fault-tolerant pairs so that should you loose a datagram on one, the other has it and you can get it there. So to loose packets is significant.

Made more significant in the nature by which I'm losing them. Let's say I start one of my apps that listens to a set of UDP multicast feeds. This guy gets started and it's running just fine. In another shell on the same box, I start another application that listens to a different set of UDP channels. As this second application is starting - the first app starts dropping packets! Within a few seconds, everything stabilizes and both applications are fine and neither app is dropping anything.

If I then stop the second app - the first app drops a few packets! Again, within a second or so, it's all stable again and nothing more is dropped.

From this, I have a few observations and a theory.

• It is not in the process space - two apps share nothing but the OS and hardware. So it's not "within" either process.
• It is socket related - because I loose packets on A and B channels, it's not the failure of one multicast channel.
• It is load related - the more load there is on the first and second apps, the worse the drops.

My theory is that it's the way the bonded interface is configured. Specifically, I believe it's set up to automatically rebalance the load between the two sockets, and in so doing, changing the load causes some of the sockets to be shifted from one physical NIC to another, and the packets are dropped.

It certainly makes sense. The question is: can I effect the configuration in a meaningful way? I looked at the modes for bonding NICs in Ubuntu, and depending on how they have it set up, I might just have to live with it. If so, at least I know where it's coming from.

UPDATE: the core issue is that I can't specify the NIC for boost asio to use for reception of the UDP traffic. If I try to use the address, I get nothing. If I use the "0.0.0.0", then I get data but the problems persist. It's an annoying limitation with boost asio UDP, but it's a limitation, and we'll have to deal with it. Crud.

UPDATE: the only option I found was in the joining of the multicast channel. It turns out that you can tell boost which address to join the multicast address on this takes the form of something like:

  socket->set_option(multicast::join_group(
                              address_v4::from_string(aChannel.first),
                              address_v4::from_string("10.2.2.8"));

where the second address is the address of the NIC you want to listen on. It works only marginally for me, and that's a drag, but it's a possibility if I need it. It's not boost's problem.

[4:20pm] UPDATE: I found out that it's the Intel NIC drivers! A guy in The Shop ran across this for his work a little bit ago, and found the solution in updated drivers for the Intel 10GbE NICs. I've talked to the Unix Admins, and they are building a patch for my boxes. This is fantastic news!

I was having a bit of a problem with The Broker today, it seemed. It appeared that when I saved my state to the Broker's configuration service, I was hanging, and the monitoring thread that fired off this save was hung. I got the guys to restart the Broker and things seemed OK, but I decided to take advantage of one of the really neat things of boost threads, and fire off the call in a separate thread so if it stalls, the monitoring thread doesn't.

The old code looks like this:

  if (secs - mLastMessageSaved >= 300) {
    saveMessagesToConfigSvc();
    mLastMessageSaved = secs;
  }

becomes:

  if (secs - mLastMessageSaved >= 300) {
    using namespace boost;
    thread   go = thread(&TickerPlant::saveMessagesToConfigSvc(), this);
    mLastMessageSaved = secs;
  }

and now the call to saveMessagesToConfigSvc() is now called by the separate thread, and as soon as the method returns, the thread is killed and cleaned up. Exceedingly sweet!

OK... this is what boost threads are all about, but in comparison to Java threads, or something that takes a little more scaffolding, this is elegant to the extreme. Just add a few constructs to the line and it's done. You can't get much simpler than that. Very nice.

This is something that was started by some of my kid's teachers years ago, and I still like it - Pi Day! We try to have pie in the house, and I'm happy to say that Pumpkin is the order of the day today. Yum!

Happy Pi-ing!

The past couple of days have been about speeding up the processing of the exchange data through my feed system. Because the code for the decoding of the messages is fixed, specifically for the big OPRA feeds (using OPRA's FAST decoder), most of this is accomplished by re-organizing the data flow and data structures. One of the things I had done a while back was to have the two channels of an exchange feed put their datagrams in separate queues, and then have one thread empty both so as to remove the need for locking on the sequence number arbitration code.

The problem was that this required twice the time to process the data through the decoder because both A and B sides went through the same thread. This can get to be really quite nasty. For example, on an OPRA feed, it takes about 40 usec to decode an average datagram, and at 50,000 datagrams/sec we're looking at 2 sec to process one side, but this design would have to do double that work. Nasty. Lots of buffering.

The solution is to have one thread per datagram stream. That immediately cuts the processing time in half. The problem is that we then need to lock for the sequence number arbitration. Nasty. Then I had a flash - merge the data!

First, tag one of the channels as primary, and have it control the arbitration. Every other channel decodes it's datagrams but then instead of trying to have the thread send it out, have that thread put the decoded messages into a queue that the primary will process as soon as it's done with it's datagram. The arbitration is very fast because it's as simple as checking the sequence number and a few flags. It's the decoding that takes the time. With one of the FIFO queues, we can have multiple non-primary channels, and have the primary take the results off and send them out.

Even more importantly, the primary can be the primary feed line of the exchange, and that makes things even better as the secondary feed is really only needed when there's a failure of the primary. What we've done then, is to make it more like the "normal" feed with a "backup" just in case.

Very neat.

This morning I noticed that Google Chrome dev 11.0.696.3 was released with a few issues addressed. Nothing major, but it's nice to see the attention to detail my the builders.

Today was a day where I spent about half my time tweaking for speed, and the other half recovering from the problems of making it too fast. I'm in the middle of a push to make the core of my ticker plants as fast as possible for a group that's going to start testing it next week. There have been a lot of little tweaks - each getting just a touch faster, but together, it's a non-trivial improvement. Unfortunately, these little changes pointed out threading issues as well. Those then needed to be fixed.

It's a game I've played for a while now, so today isn't unlike a lot of other days lately, but it's just "more of the same". It's all part of the improvement process.