Archive for July, 2010

Using Multiple Connections on a ZeroMQ Socket (cont.)

Monday, July 26th, 2010

I was able to spend another 20 mins on the code conversion to a single ZeroMQ socket for my ZMQ receiver this morning. I then ran the tests, and BINGO! it worked like a charm! Excellent. I still need to do a lot more tests, but this is an amazing step in the right direction.

Today I need to test multiple multicast channels sending to a single receiver, and then I need to get a little deeper into the actual data messages that we'll be sending as I need to do some load tests to see what it's all capable of.

Very exciting times.

Using Multiple Connections on a ZeroMQ Socket

Friday, July 23rd, 2010

ZeroMQ

The design we have for my ticker plant is to have the different products and exchanges spread out over a large number of reliable multicast channels, address and port combinations. In order to do this with ZeroMQ, I was under the impression that I needed to have a zmq::socket_t for each channel - which in ZeroMQ terms is really a URL like: epgm://eth0;225.1.1.1:55555. After all, the basic code for a multicast receiver looks like:

  #include <zmq.hpp>
 
  // make a ZeroMQ context to handle all this - use one thread for I/O
  zmq::context_t   context(1);
  // make a simple socket, and connect it to the multicast channel
  zmq::socket_t    socket(context, ZMQ_SUB);
  socket.connect("epgm://eth0;225.1.1.1:55555");
  // now set it up for all subscriptions
  socket.setsockopt(ZMQ_SUBSCRIBE, "", 0);
  // receive a single message
  zmq::message_t   msg;
  socket.recv(&msg);

So in many ways, the ZeroMQ socket looks and acts like a regular socket. But I've read in many of the mailing list posts, and even talking to a guy here at The Shop that has dug into the code, that these aren't really sockets - they are just the logical way the ZeroMQ guys made their code appear to the users of their stuff.

This is never more obvious than the mailing list post I read today about having multiple connections to a single zmq::socket_t. The maintainer of the library said that there can be multiple connections to a single socket:

  #include <zmq.hpp>
 
  // make a ZeroMQ context to handle all this - use one thread for I/O
  zmq::context_t   context(1);
  // make a simple socket, and connect it to the multicast channel
  zmq::socket_t    socket(context, ZMQ_SUB);
  socket.connect("epgm://eth0;225.1.1.1:55555");
  socket.connect("epgm://eth0;225.1.1.1:55666");
  socket.connect("epgm://eth0;225.1.1.1:77777");
  socket.connect("epgm://eth0;225.1.1.1:88888");
  // now set it up for all subscriptions
  socket.setsockopt(ZMQ_SUBSCRIBE, "", 0);
  // receive a single message
  zmq::message_t   msg;
  socket.recv(&msg);

and the recv() will then pick off the first available message on any one of the connected sockets. That's great news! As opposed to needing n sockets for n channels, I can have just one, and that simplifies my code a huge amount.

So that's what I'm working on. I'm nearly done, and then I can test. Can't wait to see how this works.

MarsEdit 3.0.4 is Out

Friday, July 23rd, 2010

MarsEdit 3

I just got a tweet from MarsEdit to say that 3.0.4 is out with a few nice little changes - especially for the rich text editor users:

  • Fix a bug that caused strange ? characters at ends of pasted paragraphs
  • Fix Drupal authentication issues by preserving cookies when connecting
  • Fixes to Tags field to handle whitespace better, and keep insertion point visible

As a WordPress user, I'm not really hit by this, and I don't use the rich text editor, but it's nice to see improvements. It's been a slow week, and any reason to work on my Mac is a welcome one.

Google Chrome dev 6.0.472.4 is Out

Friday, July 23rd, 2010

GoogleChrome.jpg

This morning I checked, and Google Chrome was updated to 6.0.482.4 with some interesting new features for the Mac:

All

  • [r52790] Chromium stops saving files for any large downloads (Issue 49216)
  • [r52693] Fix crash with SSL client auth (Issue 49197)
  • [r52850] Option clicking a link now saves a resource directly without triggering a “Save As...” dialog (Issue 36775)

Mac

  • [r52911] Implement the upgrade available notification on the Wrench menu (Issue 45147)
  • [r52485] Implement the new, unified Wrench menu (Issue 47848)

And I have to say, the new, unified "wrench" menu is pretty interesting:

Google Chrome Wrench Menu

I'm not sure how the non-standard menu items like cut, copy & paste, and the zoom are going to go over. It seems like they went for the "common denominator" here, but even then, the edit operations are all pretty standard on all the platforms - only the shortcut key-bindings are different. Hard to see why they went this way, but they did.

Anyway, it's nice to be up to date.

Running ZeroMQ Applications – Not a Trivial Thing

Thursday, July 22nd, 2010

ZeroMQ

I finally got ZeroMQ integrated into my application library and wrote a few test clients - one to send and the other to receive. When I tried to run them I got the following error:

  $ zmqReceiver
  ..(startup)...
  (process:8444): Pgm-WARNING **: DSCP setting requires CAP_NET_ADMIN of ADMIN
  capability.

I seemed to remember something about this in the mailing list, but try as I might, I wasn't able to find that bloody reference again, and so I was stuck trying to figure this out from scratch.

Basically, because of the way OpenPGM runs, it essentially requires "privileged" access on the network drivers. OK, I can believe that. So how to I accomplish that? My limited google fu yields that the command setcap should do the trick, but that command is not on the CentOS 5 install I have, and yet the command execcap is.

It seems that the execcap runs the single command with the specified capabilities, so if I do:

  $ sudo /usr/sbin/execcap 'cap_net_admin=eip' ./zmqReceiver

then all runs well.

Sort of.

We still run into the problem of LD_LIBRARY_PATH not being transferred, so I ended up making a simple script that set the LD_LIBRARY_PATH and then called the command and that script I put in the execcap command to get things working.

And they worked wonderfully.

Now I need to find a way to make it "stick" without running it as root.

[7/26] UPDATE: when I came in this morning, the linux server I code on had been rebooted. Odd... so I decided to check and see what the state of the CAP_NET_ADMIN capabilities was. Lo and behold... it was fixed! Seems the admin guy had to reboot the box to make the change, and made it he did. I can now launch these processes without the grief of the launch script or execcap. Sweet!

Excellent MacBook Pro Graphics Card Switcher

Wednesday, July 21st, 2010

In the latest MacBook Pro lineup, there's an automatic switcher from the built-in Intel graphics chipset to the NVidia chipset. Which is great, but there's been quite a bit of complaining that this is a power stealer - even the simplest of apps force a switch to the high-power, high-performance, graphics, and then your battery life drops to the floor.

So this guy has made a new menu bar app that tells you what is in use, what applications are forcing it's use, and even allows you to force it to a mode. Pretty neat.

If I am lucky enough to get one of these before Apple addresses the problem in some update, then I'll want to get this to make my life a lot easier. Slick coding.

Git Grief – Always Pulling to Master

Wednesday, July 21st, 2010

gitLogo.gif

This is something I hadn't run across in - well... forever... If you are on the master branch, and the central repository (that place you cloned from) has changes you haven't received, the natural inclination is to do:

  $ git pull

but if you do this from a simple clone of a repo, and you're on the master branch, then you might very well get the following message:

  You asked me to pull without telling me which branch you
  want to merge with, and 'branch.master.merge' in
  your configuration file does not tell me, either. Please
  specify which branch you want to use on the command line and
  try again (e.g. 'git pull <repository> <refspec>').
  See git-pull(1) for details.

What?! OK, so I'm back to googling as the obvious things weren't working. Thankfully, I found this pretty early on. I simply need to edit my .git/config file and add the section (if it doesn't already exist):

  [branch "master"]
    remote = origin
    merge = refs/heads/master

With this, I can do the "normal":

  $ git pull

and it'll work just fine. Once I pull in the changes, I can then:

  $ git push

and send up my changes as well.

OK. Whew!

Need a Cleaner Messaging System – Trying ZeroMQ

Tuesday, July 20th, 2010

ZeroMQ

I was looking at several different reliable multicast systems today and came across one that appears to be promising. It's called ZeroMQ, and I'm not sure if it's going to work out, but I wanted to give it a try as it is planning to be used by another group in The Shop, and they have really gotten deep into the guts of the code. So it's probably going to be OK. I just won't know until I really get into it.

The first thing I need to do is to build it, and I might as well build an RPM for it for CentOS 5. There's no such RPM in existence, and yet they provide a nice zeromq.spec file for building an RPM, so I decided to go with that.

For the most part, it works OK... but there's a problem that is a killer if you don't get past it. It won't even compile with OpenPGM activated (--with-pgm). The solution was on the mailing list, and there was no easy way for me to put the fix into the zeromq.spec file, so I had to un-tar the ZeroMQ tarball, edit the src/Makefile.in file and then tar it back up and then build the RPM.

The critical line is:

  1. @BUILD_PGM_TRUE@@ON_MINGW_FALSE@ -DCONFIG_HAVE_HPET \
  2. @BUILD_PGM_TRUE@@ON_MINGW_FALSE@ -DPGM_GNUC_INTERNAL=G_GNUC_INTERNAL \
  3. @BUILD_PGM_TRUE@@ON_MINGW_FALSE@ -DGETTEXT_PACKAGE='"pgm"' \

to:

  1. @BUILD_PGM_TRUE@@ON_MINGW_FALSE@ -DCONFIG_HAVE_HPET \
  2. @BUILD_PGM_TRUE@@ON_MINGW_FALSE@ -DPGM_GNUC_INTERNAL= \
  3. @BUILD_PGM_TRUE@@ON_MINGW_FALSE@ -DGETTEXT_PACKAGE='"pgm"' \

and then the rpmbuild -bb zeromq.spec will work just fine.

I'm interested to see if I can get reliable multicast working with this guy.

Google Chrome dev 6.0.466.4 is Out

Tuesday, July 20th, 2010

GoogleChrome.jpg

When I checked this morning the good folks at Google have updated Google Chrome dev to 6.0.466.4 and while the release notes don't look all that impressive:

All

  • Late binding enabled for SSL sockets: High priority SSL requests are now always sent to the server first.
  • The extension api “chrome.idle” has moved out of experimental and now has its own permission: “idle”.

it's still nice to see progress. I guess I'm kidding myself here a bit... this isn't going to make one tiny difference to how I use this browser. It's getting to the very last issues before cutting it over to 'stable', and then 'dev' will be 7.x. Oh well... we'll keep on watching.

Trying to Get a Handle on 29West Usage at The Shop

Monday, July 19th, 2010

Today has been a long and tiring day, but I think finally I have the answers I've been looking for. Specifically, I've been trying to complete my next source/sink pair for the ticker plant that I'm working on, and this pair is meant to use 29West. Their LBM (Latency Buster Messaging) product, specifically. In the past, I've used 29West in what I consider a fairly "standard" configuration - there's a resolver for a large series of services - but separated out for production, UAT, development and possibly even multiple sites. But the point is to make these resolvers handle as much as possible so they actually do the job their name implies: resolve.

Then you have a series of topics like all messaging products, and you start sending and listening for packets on those topics. It's not the most elegant messaging system I've ever used - I think it doesn't encapsulate enough for the user, but that's a corporate design decision, and while I don't agree with theirs, it is theirs to make.

When I started looking at the code from the existing system using 29West, I noticed that it was doing a lot of very specific multicast address stuff, and different resolvers, and I got the feeling that they weren't using it like this at all. In fact, I got the distinct impression that they were using it to bridge multiple multicast networks together.

When we got together and I asked a few questions I realized that they were using it as basically nothing more than a reliable multicast system. The topics were really overly simplistic, but they had independent 29West networks for each of the different message types, and then each of the different first letters in the symbol names. It was really just a reliable multicast library.

When I realized this, I knew the real solution was to get rid of 29West and go towards a more simplistic, yet solid, reliable multicast library. I just had to find it.