What's it all about, Alfie?

I'm working on my latest new project - a simple lisp-like language in C++ that is intended to be fast to execute and very memory efficient. In the past, I've built something like this, but it was more of an interpreter, and this time I'm heading for a more JIT-style system where I'm going to be creating in-memory data structures and then running them over and over as the values come in. This matches up more with what the use case is: data stream processing.

What I'm realizing that as I build the code in LKit, I'm really building a new language based on C++ and the objects and operators that I define in C++ for this problem domain. Think about it: C++ has only the basic data types of C, but it ships with the ability to deal with complex data - yeah, a + bi complex data.

You can create these as if they were simple data values, add, subtract, multiply them… even stream them out. For all intents and purposes, the shipping compiler really supports complex data types. But it's really all just built on top of what's there, and that's the real power of C++ to me - the ability to really augment the language to be able to add in data types and operations that make perfect sense in the domain, but aren't in the initial cut of the compiler.

I've create a value. This value can be a bool, an int, a double, or a time stamp. I can add these, compare them to one another, and to the data types they represent. They look, act, and process exactly like they were part of the language. That's really quite incredible. But it comes at a cost: you have to code it all up.

You have to code up the operations. You have to code up every little thing about how to handle these new data types, and if you don't, then it's not there. It's a lot of code to throw down, and I can see why a lot of people shy away from it, but it's got a lot of power, and there's reason to be careful with what you are doing.

But in defining this language, I really have a tremendous power. I get to define how all this works, and what it all does. I can make my resulting C++ code look incredibly simple and clean by making sure all the operators and functions are there to support what one would naturally want to do with these data types.

This makes the language almost domain-specific. And that's one of the things that makes coding in C++ so amazing to me. Great tools.

Today I've spent a good bit of time working on a new project - a lisp-like language in C++ for processing scripts in as efficient manner as possible. The idea was really from some work I've done in the past as well as talking to some nice guys at a start-up that are looking to do the same kind of thing. What I was thinking was that I'd put something together, and point them to it so they could get an idea of how I'd do it. Then, they are free to use it or not, and I've done my part to help them out.

I interviewed with them, but I don't think I'm going to be getting an offer, or taking the offer, as it's likely to be lower than what I have from another place, and it's a small shop, and therefore, a significantly greater risk. But I want to help them, and this is what they were hoping I'd be doing if I joined the group.

The first steps were pulling up the ideas I had from the original code, but there were a lot of things about that code that I didn't like - specifically in the area of performance. I'd like this version to be as fast as possible as it's going to be working on very large data sets - typically time series data, and so the type of data is somewhat restricted, but the amount is enormous.

I got the first two classes built: value and then variable (a value with a name). Now I need to step back a bit and work on the rest of the component design: expressions, functions, etc. The question will be: Should I break out the evaluation methods from value into a different base class? I'm not sure if I need to do this, or if it's really going to benefit the design in any way.

Certainly, it'd be nice to have all the "evaluate-able" objects based off one super class, but I'm not sure that I need that level of flexibility. After all, a function takes a series of values as arguments, and produces a value. An expression is really a function and/or values in some relation to one another. It's not like I need to have this all that different from what it is now.

For instance, I can base the function on the value and have the result value put into the ivar for the value as a cached value to speed up the resulting calculations. That would be nice. Then expressions are something else… maybe they are just relations of values. Not sure.

In any case, I've got a start, and it's going to take a lot more work to get something ready to put on GitHub and show to the guys at the start-up. But I'll get there.

I have been wishing that GitHub for Mac would properly show my Gravatar since the first release so long ago. The problem is that on GitHub, my Gravatar shows up just fine:

but on GitHub for Mac, it's all the default I have no idea who you are… image:

The problem stems from the issue that Gravatar assumes that the email is all lower-case before it's sent to Gravatar for image lookup. This is compounded by the fact that Gravatar does not allow you to have two emails registered for the same Gravatar that only differ by the case in the email. This means that drbob@themanfromspud.com and drbob@TheManFromSPUD.com are totally different Gravatars, and the latter will never be found.

This is highlighted by the PHP code snippet that I found for successfully getting a Gravatar:

  $out = "http://www.gravatar.com/avatar.php?gravatar_id=".md5(strlower($email_addr));

I'm willing to bet that GitHub for Mac is not doing the lower-casing of the email address before it's taking the MD5 hash and sending that to Gravatar. To test this, I changed the user.email parameter on my MacBook Pro and did a few checkins:

The commits I did today have the lower-case version of the user.email setting in the commits, but the older ones don't. This is a clear indication that I'm right. Now it's possible, I suppose, to go back and completely update all the repos I'm using, but I think that's a bit excessive. I'd like Gravatar to allow for mixed-case, but they just take the gravatar_id, and it's already MD5 based at that point.

Nope, this is a GitHub for Mac issue, and they need to lower-case the email address before they send it to Gravatar and then all will be well. I've sent in the request to GitHub, and we'll see what they have to say. I'm hoping they fix this, I do hate to see the little grey shadow for all my checkins.

UPDATE: Great News! I heard back from the GitHub for Mac guys:

  From: Josh Abernathy (GitHub Staff) 
  Subject: GitHub for Mac - Gravatar Mixup

  Hi Bob,

  Ahh, thanks for tracking that down for me. I've created an issue for it.

  Thanks!

so it looks like it's going to get fixed in an upcoming release of GitHub for Mac. That's such wonderful news as it means that all my avatars will be me and not the gray shadow. Super nice!

This morning I decided to check out the Pull Requests on GitHub. Specifically, I've always wondered how they work, and what the interface is, and I've had a nice little example waiting for me in the XMLRPC library that I'm using for SyncKit. So now is as good a time as any.

The core of adding ARC (Automatic Reference Counting) support for a Cocoa project is that ARC support is really compiled into the code. ARC is basically the LLVM correctly identifying when retain, release and autorelease need to be called, and then putting that code into the compiled executable without it having to be in the source.

It's slick, clean, and makes the code a lot less prone to memory issues. But there are things you can't do in ARC code that you can do in retain/release code - like compare-and-swap atomic operations. This is bad for the really low-latency stuff, but when you get into iOS, the ARC stuff is really a much easier way to make sure your app - and not you, has good, clean memory management.

I picked up the following simple fix for ARC support from Gus of FlyingMeat fame, and his FMDB project on GitHub:

  #if ! __has_feature(objc_arc)
    return [NSMakeCollectable(temporaryUUIDString) autorelease];
  #else
    return (__bridge_transfer NSString*)temporaryUUIDString;
  #endif

Because the LLVM compiler knows what the illegal methods are, it's a simple matter of going through all the code and correcting it a line at a time. It took me about two hours to do, and then I had something that worked great for AppKit and UIKit. Sweet.

So this morning, I forked the repo on GitHub, then cloned it to my MacBook Pro, made the changes and cleaned up the whitespace a bit, and then checked it all in and pushed it up to GitHub. Now was the time for the Pull request.

Happily, it's very simple. GitHub knows the commits you have done since the fork, and it's simply a matter of sending those commits to the original project maintainer, and it's up to him to incorporate them or not. You can write a nice set of comments to go along with the pull request, and I did, but I'm not sure that's all that necessary for something this simple. But is was fun.

It's off, and now it's up to Eric to see if he wants to support this. I've got my forked repo now, so it's no big deal to me, but it'd be nice to be able to stay up to date and see the code get better because of my involvement.

Sweet!

UPDATE: It was accepted!

This is why I love public domain coding. I can work to make something better, and the author looks at it and says Sure, why not? This is a great thing we are doing - those of us that can code well… and we need to do more of it. Give more than you take. That's a great motto.

This morning I was putting together a GitHub pull request for the XMLRPC library I'm using in SyncKit, and I realized that I'm really tired of the tab size on the git diff being 8 when all my code uses a tab size of 4. So I finally started googling to find an answer, and it turns out that the presentation of the git diff is really nothing more than the Unix command less. Interesting. Makes sense.

So you can make the tab size 4 by simply running:

  $ git config --global core.pager 'less -x4'

and if you omit the --global argument, it'll set it just for the repo you are in. This will be great when I have a repo with a tab size of 2, for instance. I can set the global to 4, and then override it as needed in the repo. Very nice.

Once again, Git rules.

While I like Ubuntu 12.04 a lot, the lock screen on the screen saver is a little too much authentication for me. It's a laptop, and it's on my desk, so it's not going anywhere. But I couldn't find an easy way to get rid of it. Until this morning. Yeah!

Turns out it's a simple preference command:

  $ gsettings set org.gnome.desktop.screensaver lock-enabled false

And with this run once, the screen save is still active, but when I hit a key to wake it up, I don't have to type in my password to see the desktop. Nice! This is a really nice desktop linux system - in addition to being a great server platform. Very impressed.

This morning I noticed that yesterday while I was at the interviews, the Google Chrome team release 21.0.1180.11 to the dev channel. The changes are sounding pretty routine these days: new V8 javascript engine 3.11.10.12, more Retina (HiDPI) enhancements for the new MacBook Pros, and several other crash fixes. Not bad for an update. I'm pleased that they are keeping the speed up after those few sluggish releases, so we'll see what they have planned for the 22.x series.

This afternoon I decided that maybe it was time to see if I could get ZeroMQ built and running on my MacBook Pro running OS X 10.7 as well as my Ubuntu 12.04 laptop. I'm thinking it might be nice to write a few little test apps again with the latest ZeroMQ APIs between the machines to make sure that I have everything I need - should it come to that and I need to implement a little ZeroMQ action into DKit, or some other library.

The first step is downloading it from the ZeroMQ site. I picked the POSIX tarball as it's the one with the created ./configure script, and I needed that in order to get things kicked off.

Next, we try to build it on OS X 10.7 and Ubuntu 12.04. There are a few changes that have to be made to the OpenPGM code in order for it to compile on OS X. They are basically the includes needed, and not allowing duplicate definition of values.

In ./foreign/openpgm/build-staging/openpgm/pgm/include/pgm/in.h :

Replace:

/* sections 5 and 8.2 of RFC 3768: Multicast group request */
struct group_req
{
  uint32_t    gr_interface;           /* interface index */
  struct sockaddr_storage gr_group;   /* group address */
};
 
struct group_source_req
{
  uint32_t    gsr_interface;          /* interface index */
  struct sockaddr_storage gsr_group;  /* group address */
  struct sockaddr_storage gsr_source; /* group source */
};

with:

#ifndef __APPLE__
  /* sections 5 and 8.2 of RFC 3768: Multicast group request */
  struct group_req
  {
    uint32_t    gr_interface;           /* interface index */
    struct sockaddr_storage gr_group;   /* group address */
  };
 
  struct group_source_req
  {
    uint32_t    gsr_interface;          /* interface index */
    struct sockaddr_storage gsr_group;  /* group address */
    struct sockaddr_storage gsr_source; /* group source */
  };
#endif  // __APPLE__

In ./foreign/openpgm/build-staging/openpgm/pgm/sockaddr.c :

Replace:

#include <errno.h>
#ifndef _WIN32
#    include <sys/socket.h>
#    include <netdb.h>
#endif

with:

#include <errno.h>
/* Mac OS X 10.7 differences */
#ifdef __APPLE__
#    define __APPLE_USE_RFC_3542
#    include <netinet/in.h>
#endif
#ifndef _WIN32
#    include <sys/socket.h>
#    include <netdb.h>
#endif

In ./foreign/openpgm/build-staging/openpgm/pgm/recv.c :

Replace:

#include <errno.h>
#ifndef _WIN32

with:

#include <errno.h>
/* Mac OS X 10.7 differences */
#ifdef __APPLE__
#    define __APPLE_USE_RFC_3542
#    include <netinet/in.h>
#endif
#ifndef _WIN32

The final change is to the ZeroMQ source itself:

In ./src/pgm_socket.cpp :

Remove lines 88-92, make this:

pgm_error_t *pgm_error = NULL;
struct pgm_addrinfo_t hints, *res = NULL;
sa_family_t sa_family;
 
memset (&hints, 0, sizeof (hints));
hints.ai_family = AF_UNSPEC;
if (!pgm_getaddrinfo (network, NULL, &res, &pgm_error)) {

look like this:

pgm_error_t *pgm_error = NULL;
if (!pgm_getaddrinfo (network, NULL, addr, &pgm_error)) {

At this point, we can get ZeroMQ to compile on Mac OS X 10.7 as well as Ubuntu 12.04. But there's a slight wrinkle… while I'm fine with the linux library being a 64-bit only architecture:

  drbob@mao:~/Developer/zeromq-3.2.0$ file src/.libs/libzmq.so.3.0.0
  src/.libs/libzmq.so.3.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/
  Linux), dynamically linked, BuildID[sha1]=0x71a160f17833128c864811b25942cdacdb54
  f6d0, not stripped

I'd really like the Mac OS X 10.7 dynamic library to be Universal, with both 32-bit and 64-bit architectures in it. Currently, it's 64-bit only:

  peabody{drbob}82: file src/.libs/libzmq.3.dylib
  src/.libs/libzmq.3.dylib: Mach-O 64-bit dynamically linked shared library x86_64

Hmmm… it's really amazing why some folks choose to write Makefiles in a way that doesn't allow you to use multiple -arch arguments. There really aren't all that many way to mess this up, but it seems that the OpenPGM and ZeroMQ guys have done it really pretty nicely. I can't simply remove the offending compiler flags and add in the necessary -arch i386 -arch x86_64. So I have to make it twice: once for i386 and again for x86_64 and then use lipo to stitch them together.

In the Makefiles, I added the -arch i386 command to the CFLAGS and CPPFLAGS variables after successfully building the 64-bit version of the libraries. I then did a simple:

  $ make clean
  $ make

and then when I looked at the resulting libraries they were 32-bit. I then just created the Universal binary with the lipo commands:

  $ lipo -create libzmq.3.dylib .libs/libzmq.3.dylib -output .libs/libzmq.3.dylib
  $ lipo -create libpgm-5.1.0.dylib .libs/libpgm-5.1.0.dylib -output \
    .libs/libpgm-5.1.0.dylib

Now I can copy these away and install them in the right location for usage. Typically, I'll do a make install and place everything into /usr/local/ and then copy these Universal binaries over the installed mono-binaries and be ready to roll.

UPDATE: I can get ZeroMQ to work with the simple transports, but the OpenPGM encapsulated transport fails miserably. I'm looking at the code and I'm becoming convinced that no one is really using or testing this code. It's a mess and there's no way some of these method calls are going to work. It's all in the PGM portion, so if they stay away from that, it's fine. But that's not good enough for me. So I'm giving up for now. I have asked a few things in IRC, but there aren't any responses (yet), and I don't expect to get any about the OpenPGM part.

I think it's dead, and I need to just move on. Sad to see, but it happens.

[6/21] UPDATE: after a lot of work trying to get the OpenPGM 5.1.118 code to work, I see that the developer knows about the problems, and is just not ready to support Mac OS X 10.7 at this time. I can respect that, but the things he had to have done to make his code this non-standard must be pretty wild. So there's nothing I can do at this point. OpenPGM is a deal-breaker, and that's the entire reason I wanted ZeroMQ.

This morning I was once again updating my Ubuntu 12.04 laptop and realized that were it not for a crummy hardware platform - the trackpad is horrible, and the display is very inexpensive, this would be a really nice laptop/development box. It's got all the tools I could ask for, it's got some wonderful fonts for the Terminal and such, it's got Google Chrome for all the web stuff… it's really pretty slick.

I gotta tip my hat to the Ubuntu folks. This is a much better distro than RedHat/Fedora was. I'm guessing Fedora is getting better, and it's probably as nice as you need, but the ability to do the updates easily - and have them on by default, is really nice. It's nice to have it all right there.

Certainly, I'm not giving up my MacBook Pro anytime soon, but I've looked at BestBuy, and you can get nice Wintel hardware for $700 to run this on… all of a sudden, it's something I might actually carry from time to time. Certainly nicer to work with than the crummy trackpad and display.

It's a great compliment to the MacBook Pro. Nice to have.

With the update of my web site, it seemed like a good time to look into refreshing the look of this blog as well. Specifically, look a a complementary theme - blue on white, with a "minimalist" sense, to keep the focus on the content, not the design. I was pleased to see that I'd done this search quite a while ago, and had a nice theme already picked out. Sweet!

The theme is called White as Milk, and it's a nice minimalist theme. There are a lot of little customizations that you can do, and I've had to do a few to get things looking the way I want - specifically, getting the style of a few elements looking right, and making the entire blog a fluid width page. I just got done doing that modification yo my basic web page and wanted to keep the same attribute in the blog.

This matches the look of the web site very nicely, while not trying to be a complete match to every style point. I think it's looking nice, and I'll live with it for a little bit to see how I feel about it.