Archive for the ‘Coding’ Category

Created SyncKitTouch for Better Code Sharing to iOS

Monday, June 4th, 2012

xcode.jpg

This morning I finally put the finishing touches on the first good cut of SyncKitTouch - the iOS library based on the AppKit Library, SyncKit that I had built for a project Joel and I are working on. It's a simple cloud synchronization scheme where we know the data that's getting synced, and the database in the cloud is a peer to the clients on iOS and OS X. The interesting part of this is that I didn't initially expect to have to build a second version of the library - I thought that all the NSDataSources and ComboBox DataSources were going to be used on both platforms. Oh no, no, no… That would make too much sense.

It seems that the data sources, while pretty much functionally the same, they aren't actually the same, and since the headers for one platform don't exist on the other, the best way I found to make the code as reusable as possible is to have the complete library in one project/target, and then put the AppKit-specific DataSources into categories on the appropriate classes. Then in the UIKit-based 'touch' library, I simply included all the classes - and re-implemented the categories this time using UIKit DataSources and Controllers.

It wasn't too bad, as most of the code was in the original library, and while it's recompiled, it's not duplicated in the project, and that means change it in one place, and that's it. Much better than making a complete duplicate of the library - or making three - one for the base functionality, one for the AppKit extensions, and another for the UIKit extensions. That's too much - better to have two - one that's "Touch", and one that's not.

So I got all the code in and compiling - but I haven't really create a SyncKitTickle app to try it out. I'm just not sure how this will look and function. I'm hoping Joel can give me a few ideas there. If he does, the next thing will be to make SyncKitTickle and start trying this out on the simulator, etc. Just to make sure that what I'm building is really working as it should.

Still… it's a nice milestone to get done and send back to Joel.

Bug in Google Chrome dev 21.0.1155.2

Thursday, May 31st, 2012

I was a little shocked to see that Google Chrome dev has a serious rendering bug in it. I think it's most obvious (to me) on the Google Finance page, but I'll bet it's a lot of places. The blocking for the graphs is way off for the page. Hugh blank spaces in the layout should have been filled up with content:

Re: skitched-20120531-164934

I'm sure they have people working on it now, as I can't have been the first person to notice this, but it's a little surprising still… I find it hard to believe this got through Q/A. Or even simple checks of common web sites - like their own!

Installing Ubuntu 12.04 on Spare Laptop

Thursday, May 31st, 2012

iMac-G5-20.jpg

I have been thinking about decommissioning my wonderful G5 iMac, as I just don't use it any more, and pulling a spare Gateway laptop out of the closet and installing the latest Ubuntu release for a platform to test DKit on in a more generic C++/linux environment. My iMac has been a blast. It's been with me for a long time, and I've really done a lot of great things with it. But the truth is that with Mac OS X 10.7 or even 10.6 running on it, it's long in the tooth, and there's no real need to keep it around. It's not hitting the dumpster, but I've got very limited desk space these days, and it really makes more sense to have the Ubuntu machine on the desk than the iMac.

So today's the change.

Last night I downloaded the Ubuntu 12.04 32-bit and AMD64 distributions and burned them onto DVDs. I think CD's would have worked, but I didn't have any, and a spindle of DVDs is something we almost never go through, so it was easy to burn off two copies. I'm glad I got both, as I remembered that the laptop in question had an AMD64 CPU in it, and so I'll be able to make the 64-bit and 32-bit libraries for DKit. Also, I'm sure I'll be able to fire up WindowMaker on Ubuntu 12.04, as I had it working on Ubuntu 10 at my last job. So all in all, it should be just what I wanted to test out the builds and compiles of DKit.

Ubuntu Tux

The boot directly into Ubuntu 12.04 is nice, but it might have been nice to have been given the option of boot or install a little earlier in the process. Here nor there, it's up, and this is not fast hardware, so I'm willing to cut Ubuntu a lot of slack here. It's nice that it has to run/install option, and so easy to get going.

I've been doing a lot of linux installs in the last decade, and they all have come to the point that a nice GUI is standard. I think Ubuntu's installer is no different. I do think that the use of the twitter feed on the install screen is very clever. It's an easy way to see what's happening, and what people are saying about Ubuntu as you're waiting for the files to copy and download. Clever.

Interesting Installation Challenges

The first thing I ran into is that sshd is not enabled by default on Ubuntu 12.04 - so I had to google how to install it:

  $ sudo apt-get install openssh-server

Once I had sshd installed on mao, I was able to log in from my MacBook Pro and move over my ~/.ssh directory so I could have my keys. After that, I needed to make sure that I had Vim:

  $ sudo apt-get install vim vim-gtk

then I moved over my .vimrc, and ~/.vim directory. In order to use the gists from Vim, I needed to have curl:

  $ sudo apt-get install curl

and then to have nice bitmapped fonts, I needed to remove the "don't" condition on them and regenerate the font table:

  $ sudo rm /etc/fonts/conf.d/70-no-bitmaps.conf
  $ sudo dpkg-reconfigure fontconfig

I certainly needed to have the compilers and source control tools:

  $ sudo apt-get install g++
  $ sudo apt-get install libboost-all-dev
  $ sudo apt-get install git

And then finally, I wanted to have the 'dev' channel of Goggle Chrome:

  $ sudo dpkg -i Downloads/google-chrome-unstable_current_amd64.deb
  $ sudo apt-get install libxss1

One thing I noticed is that the motd was a little verbose, and I wanted to trim it up. Turns out in Ubuntu 12.04, it's a symlink to /var/run/motd, so I needed to do a little to set things up right. First, copy the motd to motd.static, then relink the /etc/motd to the static version, and finally edit the static version for content:

  $ sudo cp /etc/motd /etc/motd.static
  $ sudo rm /etc/motd
  $ sudo ln -s /etc/motd.static /etc/motd
  $ sudo vi /etc/motd.static

With this, I get the leaner and cleaner motd and things are good again. I just wanted to put all this here so I had something to work off of, if I decided to do this all again.

After I got all this down and installed, I was able to go to work on making sure DKit compiled and ran on Ubuntu 12.04. It wasn't all that hard, but there was one thing I was a little surprised about. In LLVM gcc 4.2.1 I was able to have the following struct and use it in the boost::unordered_set container:

  typedef struct {
    boost::thread   thread;
    auint32_t       use_count;
  };

But gcc 4.6.3 on Ubuntu 12.04 didn't like this at all. It didn't like the default constructor or the default copy constructor. What I needed to do was to explicitly include them in order to make the compiler happy:

  typedef struct base_thread_info {
    mutable boost::thread   thread;
    auint32_t               use_count;
    /**
     * We should throw down the default constructor and copy constructor
     * as the boost container is going to need these, and we can't allow
     * the thread itself to be copied. That's just not right. So we mask
     * that and just let the count be copied. This isn't great, but we
     * have to have this capability, and this is the simplest way to
     * make it work.
     */
    base_thread_info() :
      thread(),
      use_count(0)
    { }
    base_thread_info( const base_thread_info &anOther ) :
      thread(),
      use_count(anOther.use_count)
    { }
    virtual ~base_thread_info()
    { }
  } thread_info;

The problem appeared to be that the copy constructor for the boost::thread was private, and that meant that we couldn't just "use it". In this case, we didn't even try, and that was OK. We aren't really trying to copy these, anyway, but the container needed to have this so that it could create and then copy into, as needed.

Interesting, but it's all cleared up now. Good.

[6/6] UPDATE: I forgot about PostgreSQL, PHP and Apache on the box. In order to get these guys going, I needed to install the relevant packages:

  $ sudo apt-get install postgresql-9.1
  $ sudo -u postgres createuser --superuser $USER

This gets me PostgreSQL installed, and me as a super user on the database instance - as long as I log in from the box. It also starts up the instance with the default parameters that are good enough for what I need at the moment. If I need to mess with the security permissions later, that's OK. But I think it's pretty much default behavior for now.

Then I needed to get Apache, PHP, and include the integration pieces for these so that they all work together:

  $ sudo apt-get install apache2
  $ sudo apt-get install php5 libapache2-mod-php5 php5-pgsql
  $ sudo /etc/init.d/apache2 restart

In order to get rid of an annoying warning, I needed to make a config file with the fully qualified domain name of the server:

  $ cd /etc/apache2/conf.d
  $ sudo vi fqdn

and include the single line:

  ServerName mao.themanfromspud.com

and then restart apache again. At this point, a simple info.php file shows that I have everything up and running and it's all wired up so that it all works from within apache web pages and PHP. Nice. Now I'm ready to do the other XML-RPC work I've got going right now.

Added UDP Transmitter to DKit to Round Out I/O Package

Wednesday, May 30th, 2012

DKit Laboratory

This morning I wrote a simple, yet effective, UDP transmitter for DKit. The idea is simple - if we have a UDP receiver for datagrams, then it makes sense to have a transmitter of the same. Simple make it a subclass of the sink and we should be ready to go. We can have it share io_service threads like the receivers, but since the transmitters have a slightly different usage on the io_service threads, it's probably a good idea to leave them separate for now.

We can certainly make a simple io_service base class for both the receiver and transmitter and pull these into this base class if we need to, but since the transmitter's need to have a boost::io_service::work defined for them - lest the io_service's run() method return too quickly due to nothing to do, it's different than the receivers, and simple enough to pool on the transmitters, as needed.

The one trick was the virtual template classes problem that I solved previously. I do not want to make the inclusion of the UDP transmitter require that the user know what type to use, so I made a very simple, and very clean subclass that then becomes the basis of the UDP transmitter:

  namespace detail {
  template <class T> class basic_udp_transmitter :
    public sink<T>
  {
    public:
      basic_udp_transmitter() { }
      virtual ~basic_udp_transmitter() { }
 
      /**
       * This is the main receiver method that we need to call out to
       * a concrete method for the type we're using. It's what we have
       * to do to really get a virtual template class working for us.
       */
      virtual bool recv( const T anItem )
      {
        return onMessage(anItem);
      }
 
      /**
       * This method is called when we get a new datagram, and because
       * we are expecting to instantiate this template class with the
       * type 'T' being a <datagram *>, this is the method we're expecting
       * to get hit. It's just that simple.
       */
      virtual bool onMessage( const datagram *dg )
      {
        /**
         * This method will be overridden by the specialization of
         * this template class.
         */
        return true;
      }
  };

and then the main class looks like this:

  class udp_transmitter :
    public detail::basic_udp_transmitter<datagram*>
  {
 
    // …
 
    /********************************************************
     *
     *                Processing Methods
     *
     ********************************************************/
    /**
     * This is the specialized template method that will be called from
     * the generalized recv() method, above, and will, in fact, be the
     * place where we do all the work of sending out the datagram to the
     * UDP multicast channel.
     */
    virtual bool onMessage( const datagram *aDatagram );
 
    // …
 
  };

At this point, it's easy to use the udp_transmitter because I've wired it all up to send the datagrams out via boost ASIO:

  udp_transmitter  xmit(multicast_channel("udp://239.255.1.1:30001"));
  rcvr.addToListeners(&xmit);

The threads are all handled in the same manner as the UDP receiver, so we know they are clean with the reference counting and the clearing of the sockets. It's a nice, clean implementation. You just need to wire it up to a source and it's going to send out those datagrams as they arrive.

Sweet.

It's nice to get this done and really get the next cut of the library done. It's now got receivers and transmitters, and I can build on these as needed or add in more receivers and transmitters (ZeroMQ, etc.) as needed.

Google Chrome dev 21.0.1155.2 is Out

Wednesday, May 30th, 2012

Google Chrome

This morning I noticed that Google Chrome dev 21.0.115.2 was out which marks the next major version number update in a while. It's got a new V8 javascript engine, as well as a few more experimental features enabled by default. It's nice to see it move along, and the number of times that I've been disappointed in the update - primarily due to screen redraw, are getting fewer and farther between. That's a great thing.

Glad to see the progress.

Added Adapter Class to DKit

Tuesday, May 29th, 2012

DKit Laboratory

Today I spent the time to finish up the addition of the adapter class to DKit. The idea is pretty simple: place a source and sink back-to-back and have their template types independent, and there you go! The multiple inheritance virtual template class is a little tricky in getting all these templates working, but I got it - eventually.

The original implementation was designed to simply use the API of the source and sink, but then I ended up getting into trouble with the accessing of protected methods in the source and sink, so I needed to lean a little more heavily on the actual implementations of the source and sink. This works well, too, as we then only have one codebase that we have to maintain - and that's always a good thing.

The implementation is really just a combination of the source and sink, but it affords me the flexibility of making a base class that takes a certain type, and generates a certain type, and with a little template specialization, it all comes together very simply and easily for the user.

I worked up a simple test adapter, and it's pretty clean:

  template <class TIN, class TOUT> class MyAdapter :
    public dkit::adapter<TIN, TOUT>
  {
    public:
      MyAdapter() { }
 
      /**
       * This is the main receiver method that we need to call out to
       * a concrete method for the type we're using. It's what we have
       * to do to really get a virtual template class working for us.
       */
      virtual bool recv( const TIN anItem )
      {
        return dkit::adapter<TIN, TOUT>send(convert(anItem));
      }
 
      /**
       * This method is called when we get a new datagram, and because
       * we are expecting to instantiate this template class with the
       * type 'T' being a <datagram *> this is the method we're expecting
       * to get hit. It's just that simple.
       */
      std::string convert( const datagram *dg ) {
        std::string     out = "<null>";
        if (dg != NULL) {
          std::cout << "converting: " << dg->contents() << std::endl;
          out.assign(dg->what, dg->size);
        }
        return out;
      }
  };

As with the sink, we need to include the recv() method, and then call the specialized method that will do the actual work. In many cases, this specialized method can be a virtual method, making the implementation of the subclasses even simpler. I think it's a pretty solid design, and I'm happy with it.

I'd still like to add a UDP transmitter, and I hope to get to that tomorrow - just to round out the library for completeness.

Completed the UDP Receiver for DKit

Friday, May 25th, 2012

DKit Laboratory

This afternoon I finally finished the UDP receiver for DKit - and it's everything I'd hoped for in this incarnation. When I started all this work, I wanted to make something that was easy to use, free, and something I could build on easily should I ever find myself in the same place where I needed to create an exchange feed system. Some of the limitations that I've run into in the past are:

  • Too Much in the Receiver Class - this happens all the time at a job: you want to do it "right", but there's no time. So as soon as you get something going, you add to it, as opposed to taking the time to see where it's appropriate to cut the design off and leave the component alone. In the past, this has really hurt the complexity of my UDP receivers.
  • Too Much Copy-n-Paste Reuse - as with the lockless containers and pools, I ended up not working hard enough to make the important template classes, and rather just hammered out code that worked - regardless of the line count. That's a horrible mistake to make.
  • Queueing in the Wrong Places - because of these other two mistakes, I have ended up putting in queueing in the wrong places. This makes things far less reusable as there is a built-in limitation that can't easily be overcome. In the past, this has even further introduced complexity as I've tried to make these queues "switchable".

So this time I was trying to really get it right. And I think I'm a lot closer now than I ever have been in the past. This new implementation in DKit has all the things fixed that I didn't like about my previous versions: better templates, less lines of code, and most importantly: it's the right tool for the job.

Specifically, there are several features in this version that make many of the limitations and problems of the last no longer issues at all:

  • The ability to have any number of receivers on the same service thread - this was a problem of context switching with all the threads doing a lot of waiting. In the old version, I had one thread for each pair of UDP sockets, and that was far too few. I could have easily gone up to 15 or 20, but the architecture of the receivers didn't allow the sharing of the boost::io_service instances like DKit allows. This will make it much easier to effectively load up a box and not pay a high price for context switching.
  • The ability to have any type for a source and sink - the old implementation had a single type of message moving between all sources and sinks, and while this may seem OK, it really meant that I had to have a source generate a message - it couldn't generate a datagram. The ability to have the source generate a datagram means that I can leave that component alone and now make components that sink datagrams and source messages. These are then universal in what they do, and we don't have to worry about putting too much logic into any one component.
  • The minimal feature set helps efficiency and performance - there's a lot to be said for a simple component that does one thing very well, and the templates make the compile time a little long, but in the end, it's incredibly flexible. I've always said that the best designs are the simplest designs, and this is certainly true of the receiver as it exists now. There will be no need for subclassing this guy - he's done. I now just need to make the combined sink/source to act as the subclass for those components that will mutate/translate/convert the output of one source to the input of the necessary sink. This design is far better than what we had.

The usage of this is pretty simple. You first need to make a subclass of the sink and have that call out to a method of the right type. Then you just need to create one, start it listening, and you're there:

  /**
   * This is the tests for the udp_receiver
   */
  //  System Headers
  #include <iostream>
  #include <string>
 
  //  Third-Party Headers
 
  //  Other Headers
  #include "io/udp_receiver.h"
  #include "sink.h"
  #include "util/timer.h"
 
  using namespace dkit::io;
 
  /**
   * I need to have a subclass of sink<T> that's going to handle my messages.
   * Because it's still a template class, I need to call out to a specialized
   * method. It's all still in the class, but it's required for the virtual
   * template class to actually work.
   */
  template <class T> class MySink :
    public dkit::sink<T>
  {
    public:
      MySink() :
        _cnt(0),
        _last_time(0)
      { }
 
      /**
       * This is the main receiver method that we need to call out to
       * a concrete method for the type we're using. It's what we have
       * to do to really get a virtual template class working for us.
       */
      virtual bool recv( const T anItem )
      {
        return onMessage(anItem);
      }
 
      /**
       * This method is called when we get a new datagram, and because
       * we are expecting to instantiate this template class with the
       * type 'T' being a <datagram *>, this is the method we're
       * expecting to get hit. It's just that simple.
       */
      bool onMessage( const datagram *dg ) {
        if (dg == NULL) {
          std::cout << "got a NULL" << std::endl;
        } else {
          std::cout << "got: " << dg->contents() << std::endl;
          _last_time = dkit::util::timer::usecStamp();
          ++_cnt;
        }
        return true;
      }
 
      /**
       * This method will return 'true' if we've received ANY datagrams
       * and if the last one was more than 5 sec ago. That's the timeout
       * for "no more data is coming our way."
       */
      bool allDone()
      {
        using namespace dkit::util;
        static uint32_t __limit = 5 * 1000000L;
        return ((_cnt > 0) && ((timer::usecStamp() - _last_time) > __limit));
      }
 
    private:
      uint32_t    _cnt;
      uint64_t    _last_time;
  };
 
 
  /**
   * This is the main testing app where we'll listen on a specific URL for
   * UDP multicast data, and then process it until there's a timeout. It's
   * going to also use the "shared io_service" capabilities just to make
   * sure that the reference counting in the udp_receiver is working right.
   */
  int main(int argc, char *argv[]) {
    bool  error = false;
 
    /**
     * To wire up as a listener to the udp_receiver, we need to be a
     * subclass of sink<datagram*>… so now that it's made, construct
     * one with it's own io_service.
     */
    MySink<datagram*> dump;
    udp_receiver  rcvr(multicast_channel("udp://239.255.0.1:30001"));
    rcvr.addToListeners(&dump);
    rcvr.listen();
    /**
     * At this point, make a new udp_receiver but share the io_service
     * thread from the one we just made. This will mean that both these
     * sockets are serviced on the same thread. Just a nice way to prove
     * that the reference counting is working.
     */
    udp_receiver  hold;
    hold.shareService(rcvr);
    hold.init();
    /**
     * Now let's stay in this loop as long as we need to...
     */
    while (rcvr.isListening() && !dump.allDone()) {
      sleep(1);
    }
    std::cout << "shutting down due to inactivity..." << std::endl;
    rcvr.shutdown();
 
    std::cout << (error ? "FAILED!" : "SUCCESS") << std::endl;
    return (error ? 1 : 0);
  }

The next thing I need to add to DKit is the adapter which com pines the sink and source in a single template class to be able to do all the steps in between. Once I have that, it's a simple matter of making the exchange codecs exchange adapters and then I'm out the door with a ZeroMQ transmitter that can take datagrams or payloads of some kind, and send them out with a reliable multicast transport.

It's looking pretty good.

Working Around C++ Virtual Template Issues

Thursday, May 24th, 2012

DKit Laboratory

Today I had a real problem, and I didn't even know it. It took me to the point of getting it all written up to realize that C++ virtual methods in template classes just aren't really possible. I ended up doing a big search on google, and there are enough examples of people trying to do this that I was convinced that it really isn't possible. And that left me in a lurch.

The original design I was working with in DKit was a source and sink, as I've written about in the past. What I did with these classes was to make them template classes, just recently realizing that all their method implementations needed to be in the header, but the idea was that there would be a template for the source and sink to be used as the base class of a series of sources and sinks moving different values around.

In order to do this, the sources would allow the sinks to be registered as listeners, and the sinks would then me messaged when a source needed to send a message. The source template class method for adding a listener looked like this:

  namespace dkit {
  template <class T> class source
  {
    public:
 
      virtual bool addToListeners( sink<T> *aSink )
      {
        bool        added = false;
        // first, make sure there's something to do
        if (aSink != NULL) {
          // next, see if we can add us as a source to him
          if (aSink->addToSources((const source<T>*)this)) {
            // finally, make sure we can add him to us
            added = addToSinks(aSink);
          }
        }
        return added;
      }
 
  };
  }   // end of namespace dkit

and then the code for sending a 'T' to all the registered listeners looked like this:

      virtual bool send( const T anItem )
      {
        bool       ok = true;
        if (_online) {
          // lock this up for running the sends
          boost::detail::spinlock::scoped_lock  lock(_mutex);
          // for each sink, send them the item and let them use it
          BOOST_FOREACH( sink<T> *s, _sinks ) {
            if (s != NULL) {
              if (!s->recv(anItem)) {
                ok = false;
              }
            }
          }
        }
        return ok;
      }

While I've got the virtual identifier in the template - I've learned that this is OK - so long as we're still dealing with template class definitions. So they stay, but the problem was soon approaching.

When I want to make a sink for the particular source I've created (we'll get to that in a minute), I need to subclass the sink which has the one important method:

  namespace dkit {
  template <class T> class sink
  {
    public:
 
      virtual bool recv( const T anItem )
      {
        /**
         * Here's where we need to process the item and do what we
         * need to with it. We need to be quick about it, though,
         * as the source is waiting on us to finish, and send the
         * same item to all the other registered listeners.
         */
        return true;
      }
 
  };
  }   // end of namespace dkit

This method is called in the source's send() method, and is the way we'll be able to get what we need. My original plan was to implement the listener like this:

  template <class T> class MySink :
    public dkit::sink<datagram*>
  {
    public:
      MySink() { }
 
      virtual bool recv( const datagram *anItem )
      {
        // do something with 'anItem' - the datagram pointer
        return true;
      }
  };

but while it compiled, it didn't work. Why not? Because there is no support for virtual methods in partially implemented template classes. None.

No matter what I did, there was no getting around it. My test app was simple:

  int main(int argc, char *argv[]) {
    bool   error = false;
 
    MySink     dump;
    udp_receiver   rcvr(multicast_channel("udp://239.255.0.1:30001"));
    rcvr.addToListeners(&dump);
    rcvr.listen();
    while (rcvr.isListening()) {
      sleep(1);
    }
 
    std::cout << (error ? "FAILED!" : "SUCCESS") << std::endl;
    return (error ? 1 : 0);
  }

I created the sink - called MySink, and then wired it up to the UDP receiver (a source), and started listening. All the datagrams were sent to the base class implementation of sink::recv(). Nothing was coming to the subclass' method.

OK, it's just not in the language. Boy, I sure wish it were, but it's not. So how to fix things? Well… I have a FIFO template class in DKit, and it's got working virtual methods. But the trick there is that the subclasses are still template classes. There's no attempt at specialization. So maybe the answer is as simple: Don't specialize!

So if I don't specialize, then my MySink class is still a template class:

  template <class T> class MySink :
    public dkit::sink<T>
  {
    public:
      MySink() { }
 
      virtual bool recv( const T anItem )
      {
        return onMessage(anItem);
      }
  };

and I've introduced a new function: onMessage(T). This, I just need to create for the kind of data I'm moving:

  bool onMessage( const datagram *dg ) {
    if (dg == NULL) {
      std::cout << "got a NULL" << std::endl;
    } else {
      std::cout << "got: " << dg->contents() << std::endl;
    }
    return true;
  }

Now my test application has to include the type in the definition:

  int main(int argc, char *argv[]) {
    bool   error = false;
 
    MySink<datagram*>   dump;
    udp_receiver   rcvr(multicast_channel("udp://239.255.0.1:30001"));
    rcvr.addToListeners(&dump);
    rcvr.listen();
    while (rcvr.isListening()) {
      sleep(1);
    }
 
    std::cout << (error ? "FAILED!" : "SUCCESS") << std::endl;
    return (error ? 1 : 0);
  }

What I'm doing is keeping the template subclass as the same template so that the virtual methods are going to work for us. Then I make use of the fact that I only need to make the one version of the method I need, as the templates are resolved at compile time. If I need a new subclass, then so be it. But I can make these implementations very easily and very quickly, and it's nearly as good as the fully virtual template methods - just one small step in there to glue things together.

I'm pleased with the progress. I'll be checking all this into GitHub soon, and then it'll be up there for good, but for now, I'm happy that I've figured out a clean way to make this work for me.

[5/25] UPDATE: I was working on this this morning, and there's a slightly cleaner way to do the polymorphic behavior with template classes:

  template <class T> class MySink :
    public dkit::sink<T>
  {
    public:
      MySink() { }
 
      virtual bool recv( const T anItem )
      {
        return onMessage(anItem);
      }
 
      bool onMessage( const datagram *dg ) {
        if (dg == NULL) {
          std::cout << "got a NULL" << std::endl;
        } else {
          std::cout << "got: " << dg->contents() << std::endl;
        }
        return true;
      }
  };

and at this point, it's all contained in the subclass of sink.

Minor point, but it's contained in one class, and that makes things a little easier to read and deal with. I tried to put the call in the sink superclass, but it's the same problem all over again - virtual methods in template classes are just not very well supported. But this at least is not too big of a hack to get things working properly.

Building Solid Template Classes – Stick to Headers

Tuesday, May 22nd, 2012

DKit Laboratory

Today I was starting to compile my latest addition to DKit - the UDP receiver that's a source of datagrams in the DKit sense. This is really just the first step of many to get the exchange feeds re-written into DKit in a far better way than I had recently done. The second time is always better, and the third better still.

So I had built the source and sink template classes with headers and implementation files. This, it turns out, really isn't such a hot idea. Because the templates are expanded at compile time, it's not possible to compile the source once, and then use it over and over again. I should have know this, but it wasn't smacking me in the face until I started to try and compile the UDP receiver and it was talking about missing implementations of the source::~source() for the specialization (datagram *).

Then it was instantly clear - I needed to keep everything in headers so that they could be expanded and built as needed, at compile time. Thankfully, there wasn't a lot of work to do - just move the code from the implementation file to the header and drop the implementation files from the directory and Makefile. It didn't take long, but it was clear that this was the right thing to do. The compiles started working, and things progressed nicely.

Not done, that's for sure, as I still need to throw in the async reader method to the UDP receiver, and then I can pull in that pool of datagrams and we should be in good shape. But it's still a little bit of work.

But it's nice to know that template programming is not only tricky and convoluted, it's also like Java code - all in one file. OK… that's a stretch, even for me. 🙂

General Template Usage with Pointers

Monday, May 14th, 2012

DKit Laboratory

This evening I finished up adding pools to DKit, and there are a few things that make it so bloody cool I had t write about it. In short, the problem is one that I've had to deal with before: How to make a simple pool of things such that I only create what I need, but don't over create, and can use a manageable set of things from a pool.

Like say I wanted to have a messaging system. I might want to have a bunch of std::string values that are created with a minimum size to make it easy to move things around. Then, all I would need to do is to get one from the pool, clear it out, add in the data, and then when I'm done, return it to the pool. The location in the code that gets from the pool is at a totally different place than the place that recycles these instances, and it's possible that due to threading or queueing, we may need to have several in play all at once. But eventually they will all come back to be recycled.

In the past, I had one class for a std::string pool, and another for datagrams, and so on. Each of these was almost identical to the other, just in how the instance was created. Typically, I'd have some sense of the 'default' size of the container I was creating. I based each off a single type of FIFO queue, so that it would always be a SP/SC pool, etc. This was necessary because I hadn't super classed the FIFO queues as I have in DKit.

Still… this was a lot of copy-n-paste reuse, and it was clear that it wasn't anywhere near as flexible as it could be. So today I decided to try and see what I could get away with if I tried to make a complete template pool class. The challenges were pretty clear:

  • Include the Type of the Queue in the Template - I knew that now that I had the FIFO abstract template class in DKit, it was going to be possible to have the constructor make a queue of the right type, and then just "use it" via the FIFO abstract template class and be able to allow the user to define what access type they wanted for the pool.
  • Include the Max Size of the Pool as a Power of Two - this was in keeping with the queues I'd be creating, and so shouldn't prove to be too hard.
  • Allow Pointers and Non-Pointers to be Pooled - this was the biggest challenge I faced, to be sure. In the pool, I wanted to have two basic methods: next() to get a new one from the pool, and recycle(T) to return it to the pool. The problem is what if the type 'T' is a pointer versus a non-pointer? How do we make sure we can delete a pointer, but simply let a uint64_t fall on the floor and be cleaned up?

Thankfully, I was able to look on Stack Overflow and see this interesting question which lead me to the answer I needed. What it really boiled down to was that I could use the boost::is_pointer(), but that only really lets me know if I need to clean up the contents of the queue in the clear() method. The trick to constructing and destructing was to realize that the templates worked outside the scope of the class definition, and so after the class definition, I added in these functions:

  namespace dkit {
  namespace pool_util {
  /**
   * In order to handle both pointers and non-pointers as data
   * types 'T' in the pool, we need to take advantage of the
   * template methods and make create() and delete() methods
   * for pointers and non-pointers.
   *
   * For create(), it's pretty easy - we allow for nothing to be
   * done for the non-pointer, and a standard 'new' for the pointer.
   * For delete(), it's the same - we delete it and then NULL it
   * out if it's a pointer, if it's not, we do nothing.
   */
  template <typename T> void create( T t ) { }
  template <typename T> void create( T * & t )
  {
    t = new T();
  }
 
  template <typename T> void destroy( T t ) { }
  template <typename T> void destroy( T * & t )
  {
    if (t != NULL) {
      delete t;
      t = NULL;
    }
  }
  }      // end of namespace pool_util
  }      // end of namespace dkit

Then, in the critical next() and recycle() methods, I simply used these functions:

  /**
   * This method is called to pull another item from the pool, or
   * create a new one if nothing is in the pool. This is the classic
   * way of getting the "next" item to work with.
   */
  T next()
  {
    T      n;
    // see if we can pop one off the queue. If not, make one
    if ((mQueue == NULL) || !mQueue->pop(n)) {
      pool_util::create(n);
    }
    // return what we have - new or used
    return n;
  }
 
 
  /**
   * This method is called when the user wants to recycle one of
   * the items to the pool. If the pool is full, then we'll simply
   * delete it. Otherwise, we'll put it back in the pool for use
   * the next time.
   */
  void recycle( T anItem )
  {
    if ((mQueue == NULL) || !mQueue->push(anItem)) {
      pool_util::destroy(anItem);
    }
  }

By using the create(n) and destroy(n) functions, I allow the compiler to see the template functions and pick which one to use. In the case of a pointer for 'T', it chooses the pointer-reference argument, in the case of a non-pointer, it's the pass-by-value argument. This selectivity allows me to partially implement these any time I want in order to make the actual construction and destruction as complicated as I need without requiring it for the simple default constructor and destructor.

Once I had this, the remaining problems weren't too bad at all.

I created an enum for the type of access to use:

  /**
   * We need to have a simple enum for the different "types" of queues that
   * we can use for the pool - all based on the complexity of the access. This
   * is meant to allow the user to have complete flexibility in how to ask for,
   * and recycle items from the pool.
   */
  namespace dkit {
  enum queue_type {
    sp_sc = 0,
    mp_sc,
    sp_mc,
  };
  }       // end of namespace dkit

and then it was pretty simple to make the template and the constructor:

  template <class T, uint8_t N, queue_type Q> class pool
  {
    public:
      pool() :
        mQueue(NULL)
      {
        switch (Q) {
          case sp_sc:
            mQueue = new spsc::CircularFIFO<T, N>();
            break;
          case mp_sc:
            mQueue = new mpsc::CircularFIFO<T, N>();
            break;
          case sp_mc:
            mQueue = new spmc::CircularFIFO<T, N>();
            break;
        }
      }
 
  …
  };

In the end, the code worked wonderfully. I built a test app that made sure the destructor was properly being called when recycle() was being called and the queue was full - check. I also made sure that when the pool was destructed, any remaining elements were properly destructed - if they were pointers.

It was really an amazing little bit of code. This is far more flexible and better than the previous single-purpose pools I've written. Less code is always better.