What's it all about, Alfie?

When I joined The Shop, I was expecting to work on something entirely different from what I'm actually doing. The change came on my first day when it was explained to me that they needed me to help with The Magic Schoolbus, which I've talked about already. After my first meeting about the state of the code and the architecture, I knew that the goals I'd been given initially: Rewrite it, were not going to be well-received. But hey, let's face it, it's a pretty simple concept: deliver price ticks from the exchange feeds to the clients in an efficient way, and make some provision for getting static data (index compositions, etc.) as well.

Face it: it's a price feed and a data service. That's it.

But the existing codebase makes this far far more complex than it needs to be. There is blanket copy-n-paste use at the application level which makes maintenance exceptionally hard, and there are code generators, etc. I've touched on a lot of this, and while I really did give it the 'ol college try to make changes within the existing codebase, I was always left with the realization that this was the wrong way to go about doing this.

What was needed was a complete re-write. A massive simplification of the problem and the tools. One message for price ticks - maybe two, but that's it. One message for "variable" (a.k.a. self-describing) data - that's it. There's so much that can be done when you eliminate all the complexity.

But at the same time, this stands as such a threat to the status quo that it wasn't surprising that when it came up, there was push-back. Even real anger, from the guys in the Team.

It wasn't my choice - it wasn't even my initial goal, but it was all that was left. They, of course, did not see it this way. I didn't want to tell them this - I had held off on talking to them about this until I had a really decent start on the system with a lot of headers written and even a few classes implemented to show where I was going, and that I was willing to put in the work to make it happen.

Didn't really turn out well. And even while I was in it, I had a feeling this wasn't going to go well. I just had a feeling.

Too often people that work on far too complex systems define themselves in terms of that project. "I'm the Magic Schoolbus Guy. We've seen it time and time again. When that position of power is threatened, it's not met with open arms. More like armed resistance.

I tried to point out that there are plenty of good ideas around the table - even where I'd used several folks ideas and gave them kudos for seeing the right solution. Didn't help. Still lots of hostility. I understand it, but it doesn't help. I've got a job to do, and this is just going to make it harder.

I guess I needed to be taught a lesson. That seems to be the way things are going for me these days.

Once again the happy boys at Panic have updated Transmit 4.0.6 and in this release there's several more significant issues fixed, but the overall count of changes isn't nearly as significant as the last release. Things seem to be slowing down a bit at Panic HQ. Still, it's great to see the improvements.

Today has been a very hard day where I've been trying to get on top of boost and it's asio package as well as threads and serialization. It's a lot, I know, but the point of this project I'm on right now is to stop using all these home-grown classes and start to work in something like boost which will be a part of the standard moving forward. It'll make bringing up the next guy a lot easier if it's all using boost and then they only need to focus on the usage of boost, not the low-level threading, socket, and serialization libraries. It's an admirable goal, but it's a ton of work.

So today has been trying to figure out threading. I can appreciate the model boost has for threading, but I'd also like something that looks a little more like the Java Thread object - where you subclass it, have a method you implement and then can start this instance and away it goes. It's just targeted at different market - you could do the same thing, in theory, by having a method in a class the launch point for the thread, and then have everything in that one method.

Possible, but there will be times that a simple class like this will be handy - so I wrote it.

Next, I needed to deal with the atomic integers. Boost has the 16-bit versions, which is nice, but there were already applications of an atomic integer in the code (in the transfer statistics class) where I needed to have an unsigned 32-bit integer. So boost was not going to get me where I needed to be. Additionally, I really haven't liked the very C-like usage of a lot of these atomic operations on non-atomic data types. Why not just make a class that is an atomic 16-bit (and 32-bit) unsigned integer? So I did that as well.

What I ended up using were the GCC primitives which did everything I really needed - but it took some thinking to figure out. For example, with only:

  T __sync_fetch_and_add (T *ptr, T value, ...);
  T __sync_fetch_and_sub (T *ptr, T value, ...);
  T __sync_fetch_and_or (T *ptr, T value, ...);
  T __sync_fetch_and_and (T *ptr, T value, ...);
  T __sync_fetch_and_xor (T *ptr, T value, ...);
  T __sync_fetch_and_nand (T *ptr, T value, ...);

how do you do the "atomic set"? It struggled with this for a while until I also saw:

  bool __sync_bool_compare_and_swap (T *ptr, T oldValue, T newValue, ...);

and then it hit me: I just want to make sure that it always compares as "equal". Easy:

  auint32_t & auint32_t::operator=( const uint32_t aValue )
  {
    __sync_bool_compare_and_swap(&mValue, mValue, aValue);
  }

So I'm using the idea that the existing ivar is always going to equal itself, and because of that, we'll have a swap that puts the new value into place. Hey, it's not rocket science, but it wasn't clearly laid out, either.

In the end, I had two really nice atomic unsigned integer classes with all the nice operators overloaded so you can do the prefix "++" and the postfix as well, and the sum and difference. It's all there. They look and act like their non-atomic counterparts, but they are thread-safe without using any locks.

I just noticed that this afternoon that Google Chrome dev 6.0.453.1 is out and so I updated right away. I noticed that the page refresh was a little snappier, and the release notes don't really give me a great lead:

• Continued feature parity work

So we'll see... I'm all for snappier, and I'm all for more features, I just wish I knew what they really changed.

This morning I decided that it's better to make an RPM of Boost 1.43.0 and install it than to try and package this stuff on my own. So I started out on what looks to be a long and painful journey. In the end, I'll have this as a record of what to do correctly and in what order. I'll have thankfully edited out all the errors and re-dos to come up with something that really explains what I've done.

At least I hope it's going to turn out that way.

Laying the Foundation of RPM Building

First, let's figure out what we need to build an RPM. Pretty simple: a place to do it. So I created a build directory and set up everything for a build:

  $ cd ~/vendor
  $ mkdir rpm
  $ cd rpm
  $ mkdir BUILD RPMS SOURCES SPECS SRPMS
  $ cd RPMS
  $ mkdir athlon i386 i486 i586 i686 x86_64 noarch

Then in my home directory, I created the file .rpmmacros that contained:

  %_topdir /home/rbeaty/vendor/rpm

where clearly, the point of the .rpmmacros file is to tell RPM where it's supposed to look for things as it goes about it's business. This is a huge load off my mind as I now can build in peace and not have to worry about deploying until I'm 100% sure it's ready to go.

At this point, I need the sources.

Getting the Sources in the Right Place

One of the tricks I'm using is to base this build off the boost-1.33.1 SRPM spec that I can get from CentOS. So I need to get the source RPM for the older version from CentOS 5's site:

  $ cd ~
  $ wget http://mirror.anl.gov/pub/centos/5/os/SRPMS/boost-1.33.1-10.el5.src.rpm

and then I can "install it" into the building location with:

  $ rpm -i boost-1.33.1-10.el5.src.rpm

At this point, I don't need boost-1.33.1-10.el5.src.rpm any longer.

If I go into the $RPM_BUILD_ROOT as we'll call it in this process (and it happens to be called that in the scripts) we can then:

  $ cd ~/vendor/rpm/SOURCES

and we'll see the contents of the source tree. Not bad for a few simple steps! Now we need to get the latest version of boost and place it here as well. With a simple:

  $ wget http://sourceforge.net/projects/boost/files/boost/1.43.0/
         boost_1_43_0.tar.bz2/download

(all one line) we will get the latest version of the code into the SOURCES directory as: boost_1_43_0.tar.bz2. It turns out that the bzip version of the file is important as the spec file expects to see it and we don't want to disappoint (or unnecessarily modify) it.

Update the Spec File for Boost 1.43.0

I decided to go a "minimal impact" plan on the spec file. I needed to change a few things. First, the version of Boost that I'm building. The spec file is in the SPECS directory in the main rom directory. The simple differences are:

%define tarball_name boost_1_33_1
 
Name: boost
Summary: The Boost C++ Libraries
Version: 1.33.1
Release: 10%{?dist}

becomes:

%define tarball_name boost_1_43_0
 
Name: boost
Summary: The Boost C++ Libraries
Version: 1.43.0
Release: 1%{?dist}

and since I've built it already I knew I didn't need a library I didn't have on this box, so I could comment it out. If you don't need to do this, even better.

BuildRequires: libicu-devel
Obsoletes: boost-doc <= 1.29.2
Obsoletes: boost-python <= 1.29.2

becomes:

#BuildRequires: libicu-devel
Obsoletes: boost-doc <= 1.42.0
Obsoletes: boost-python <= 1.42.0

In the %package section, I needed to change what this obsoletes:

%package devel
Summary: The Boost C++ headers and development libraries
Group: System Environment/Libraries
Requires: boost = %{version}-%{release}
Obsoletes: boost-python-devel <= 1.32.2
Provides: boost-python-devel = %{version}-%{release}

becomes:

%package devel
Summary: The Boost C++ headers and development libraries
Group: System Environment/Libraries
Requires: boost = %{version}-%{release}
Obsoletes: boost-python-devel <= 1.42.0
Provides: boost-python-devel = %{version}-%{release}

Then, in the %prep section, I commented out the patches that I didn't think needed to be made as it was running fine for me as-is.

%prep
rm -rf $RPM_BUILD_ROOT
 
%setup -n %{tarball_name} -q
%patch0 -p0
%patch1 -p0
%patch2 -p0
%patch3 -p0
%patch4 -p0
%patch5 -p0
%patch6 -p0
%patch7 -p0
%patch8 -p0

becomes:

%prep
rm -rf $RPM_BUILD_ROOT
 
%setup -n %{tarball_name} -q
#%patch0 -p0
#%patch1 -p0
#%patch2 -p0
#%patch3 -p0
#%patch4 -p0
#%patch5 -p0
#%patch6 -p0
#%patch7 -p0
#%patch8 -p0

Finally, the %build section could be drastically simplified due to the new build tools in boost:

%build
BOOST_ROOT=`pwd`;
# build jam
./bootstrap.sh
 
# build boost with bjam
./bjam

and then completely commented out the %check section as I have no need of running any tests on boost - it is what it is.

Building and Verifying

Now that I have the spec file modified, I can build the RPMs:

  $ cd ~/vendor/rpm/SPECS
  $ rpmbuild -bb boost.spec

when this is all said and done, you should have the RPMs in the RPMS directory in the main 'rpm' directory - organized by the machine architecture.

To verify the contents of the RPMs, simply go into the directory and have a look at one:

  $ cd ~/vendor/rpm/RPMS/x86_64
  $ rpm -q -p boost-devel-1.43.0-1.x86_64.rpm

What you'll see is all the files that are in the RPM and it's just what you need.

Going for the Whole Burrito

If I want to build the i386 RPMs as well, I need to do two things. First, I need to edit the spec file to tell Boost to build 32-bit and to put the libs in the right place:

%define tarball_name boost_1_43_0
%define _libdir %{_exec_prefix}/lib
 
Name: boost
Summary: The Boost C++ Libraries
Version: 1.43.0
Release: 1%{?dist}

and:

%build
BOOST_ROOT=`pwd`;
# build jam
./bootstrap.sh
 
# build boost with bjam as 32-bit
./bjam address-model=32

and then I need to build the rpm with a slightly different command:

  $ cd ~/vendor/rpm/SPECS
  $ rpmbuild -bb --target i386 boost.spec

Of course, now that I think about it, it'd be just as easy to have two spec files - one for 64-bit and the other for 32-bit and then have two commands that build their RPMs, etc. Not a horrible situation at all, really.