What's it all about, Alfie?

I've spent a bit of time thinking about what to do in my forced hiatus, and when I happened upon a posting at North Central College, I thought "Hey, maybe it's something I should do?" So I sent in an email inquiry, and talked to a friend that loves teaching as much as I do, and was getting excited about the thought of teaching, doing a little consulting, and enjoying life a while lot more.

So I found the citations for all my papers, and put them on my resumes, and started thinking about writing up the first of many essays they wanted as part of the application packet.

Then I thought - I wonder if the pay is good enough? I don't need to supplement my complete salary from coding, as the consulting will help out quite a bit, but I need it to be enough that I can afford to take the job. So I hit up the web to see if there were anything with typical, maybe even average, salaries for faculty positions on campus.

And there were.

And it wasn't great news.

The average for Assistant Professors that I could find was in the high-50's. Then I looked nationally, and again, it confirmed what I had known in my years at Auburn, and that's the simple fact that faculty members don't make a ton of cash from teaching, it's from the books, and the consulting. So it's not really a great deal for me. I'd need to have such an amazing consulting income that I'd likely just skip the teaching and do more consulting. Or coding.

I was a little disappointed, but not totally surprised. It's something that you can do after you make your money - or if you can live on a lot less than downtown Naperville housing prices. It's just not really feasible at this time.

I may still go through with it - just to see what the actual numbers may be, but I know it's not going to be something I'm able to count on as my primary income. That's going to have to be back to the finance world.

Darn. It would have been nice.

Since I've been looking for a job, I haven't been doing a lot of posting because not a lot has been happening - except all the stuff associated with getting a new job. But one of the things I wanted to do this time around was to create a GitHub project that could showcase a little bit of my coding skills as well as being generally useful for folks - me included, and just get something out there. So I created DKit.

The idea behind DKit is simple - it's a very specialized C++ library with classes and templates, that contains atomic and lockless data structures for very efficient C++ coding. I've used classes like these in projects at previous employers, but I started fresh, and expanded the classes significantly so that the atomic integers paralleled those in the stdint.h header - signed and unsigned, 8, 16, 32, and 64 bit sizes. These will make counters trivial as they are all atomic and so you'll get great counts even if you don't go through the overhead of locking.

I also added in two groups of containers, so far: single-producer/single-consumer, and multiple-producer/single-consumer. To the first, I put in a simple circular FIFO, and to the latter I put in a linked FIFO and a circular FIFO. These were ideas I had at a previous job, but I wanted to make them fresh so as not to get into any IP issues.

All told, it's a nice start. I want to keep it free of the issues that I might have had in the past, and just keep it OPen Source. I picked the MIT license as it seems to be the most liberal while protecting the original copyright and the removal of any liability. I think it's the start of something really nice.

This morning a good friend messaged me about something I might want to keep in mind as I do more massively parallel projects in the future, and that's a C++ library that fits in will with the STL, and is written by a couple of engineers at NVIDIA - Thrust. This looks very interesting.

The idea is that with the CUDA nvcc compiler, and a little pre-processing and some templates, they have made the STL std::vector equivalent for GPU/CPU usage. You can say where you want this vector to be held, and then process a functor on it to get things done very quickly. It's not as clean as OpenCL, in my opinion, but it's far more likely to be used as throwing NVIDIA cards into boxes - even server boxes, and using GCC/nvcc is a lot more likely in the typical business use-case.

Macs still aren't getting significant penetration there, but it's getting better.

So this is something to keep in mind, for sure. Thankfully, it'll run on Macs as well - so I just need to get a Mac Pro with NVIDIA card(s) and then I can start playing with it. Sweet.

This morning, sometime, I noticed that Google Chrome dev 19.0.1068.0 was released and the release notes seem focused on the Android stack and a few side things, but I guess not every release can come up with a new way of doing things, or a massive speed improvement. But progress is progress.

This morning I noticed that Google Chrome dev 19.0.1061.1 is out and it's got a few nice things in it. Like a new V8 javascript engine (3.9.13.0), and support for remote file systems - could be interesting stuff. Glad to see they are still making improvements in V8 - that's something that I think I'm going to end up in sooner than later, and it'll be nice to have some improved performance there.

This morning I wanted to try to get some of my CVS repos converted to Git so that I could have the complete repo on my laptop and not have to worry about internet connectivity. I'm a big fan of CVS, and it's simplicity, but Git is the clear next-generation of CVS, and it doesn't need the connection to the server that CVS does.

Recently, I'd read that there was a simple git command: git cvsimport that would convert a repo, and I just had to try. The first thing was that I needed to have a program called cvsps. This is some other tool - not part of CVS, not part of Git, that I needed to get. I realized this as I tried to convert my first repo, and failed saying it couldn't find this app. So first things first, get the tools I need.

Getting cvsps

A simple google search revealed that the source code for cvsps was held in a very simple web site: http://www.cobite.com/cvsps/. I downloaded the latest stable code, and read the README. It's a simple make; make install, and I'm on my way:

  $ cd cvsps-2.1
  $ make
  $ sudo make install

The cvsps executable is now in /usr/local/bin and /usr/local/share/man. That's all we need - over and above Xcode 4.3 and it's command line tools (cvs, git).

Get a Local Copy of the CVSROOT

While I've read that this can be done using a remote pserver $CVSROOT, it's a good idea to just get a local copy of the complete CVSROOT to work from. Since I'm in a stable state with that, it was pretty easy to copy it to my TimeMachine external drive:

  $ cd /Volumes/Reststop
  $ scp -r frosty:/usr/local/CVSroot .

It only took a few minutes, and now I've got all the "source material" I need.

Migrate a Single CVS Repo

The process is pretty simple - you have to act as if you're starting a new git repo, but instead of the git init command, you issue the git cvsimport command. It's got a few arguments, but it's pretty simple to use.

For this example, I'm calling the new Git repo the same name as the old CVS repo, but I'm guessing if you want, you can change the names.

  $ cd git
  $ mkdir MyProj
  $ cd MyProj
  $ git cvsimport -p x -v -d /Volumes/Reststop/CVSroot MyProj

At this point, you have a new git repo but the origin is not set, so it's not "going" anywhere if you try to push it. Since I'm using gitosis on my home git server, it's a simple process to update that to add in the project(s) I'm migrating into the proper groups, and then push those changes up to the server before I try to set the origin of the new git repo.

Setting the Origin

Assuming that you have the server set up, and it could be that you're using GitHub and not gitosis, then all you need to do is to set the origin and push it up:

  $ git remote add origin git@git.myplace.com:MyProj.git
  $ git push origin master:refs/heads/master

Final Steps

It's possible to now go in and mess with the git config to set the master right, but I've found is just as easy to remove this new repo, and clone it again from the server. If I got it right, the history will be there, and I'll be sure it works. If not, I can start over. Simple.

It's been a lot of fun getting these guys over to git. Now I can use all the tools and fun I've had with git in the last year to these projects. Very nice!

I was fixing a production problem that the Unix Admins hadn't remembered to fix after the last kernel update, and my Boss asked to see me. I asked him to hold on a second to make sure that I had the production systems back up and emails sent, and then I followed him. Right past his office.

I was going to say "You takin' me out back to whack me?", but as we turned the corner, it was clear that, in a way, that's exactly what he was doing. I was lead into the room with an HR lady with a folder, and it was clear - I was being fired! Wild.

The reason was lame - I was unhappy. He elaborated to say that eventually I was going to leave or they were going to have to fire me, so do it now and get it over with. Seems a little wild given that less than two weeks ago, I got a $75,000.00 bonus check. Hmmm… could it be that I was looking and they found out?

Seems so.

I had a lunch meeting with a friend - totally unrelated to the search, but it could have look suspicious. Also, a clumsy recruiter called me at work - bone-head move, but it could have exposed me. Also, there's the very real possibility that someone blabbed on me. The Chicago finance industry is tight-nit, and it's happened to me before, so I know it can happen.

So it goes. Thankfully, I have a few things to look at, and I'm sure I'll be employed soon, and that's OK with me. This was not the place for me long-term. Too many people that are in the wrong jobs.

One of my favorite things is to work with databases in code. Persistence and database hits are a blast as they get you a place to save stuff that, if you design it right, you can view from just about any tool on the planet. Can't say the same for redis or mongoDB. My Greek Engine gets it's instrument data from a local replica copy of a master postgres database, and should the local copy fail - or be down, it should auto-reconnect to the master and just function off that one. If he's dead… well… that's when it's time to get serious about getting things working.

The first thing I needed to do was to consolidate all the database activity to as few a number of places as possible. Thankfully, I had a simple execute() method that did about 90% of what I needed. It just took a few minutes to make that the only way to hit the database, and then I could focus on making that a little more fault-tolerant.

The idea is simple, really: put it in a retry loop, limit the number of retries, and then for each retry, hit The Broker for the correct database connection parameters to use. If the Broker is wrong, then I'm in real trouble, but it's not, so I'm OK. (Famous last words.)

Add a little logging, remove some error codes, and we're ready to go. It really didn't take me all that long, and the results are much better. When, and if, the database goes down, we'll fail over to the master. When we get the local copy up, we can issue an IRC command to repeat the process, and the local one will again be used. Simple. Clean.

Great.

This morning I came in to see that some of the exchange feeds on one of the staging boxes of mine hadn't shut down properly. When the exchange test data flooded in, it made a mess, and that was no good at all. The only code that seemed to matter was a simple iterator on the TBB concurrent_vector. I've had issues with this code before - and always moved away from it in favor of a simple std::vector and a mutex of some sort. Here was another case of the exact same thing.

Now I'm not saying that the concurrent_vector is a mess, but I think that it, along with the concurrent_map are a little trickier than normal to work with. The iterators have built-in locks, and that makes it very easy to write dodgey code. I think that's what happened, but I can't prove it.

Far easier to use a simple std::vector and then a TBB spin_rw_mutex_v3 to protect it. Virtually all the access to the vector is read-only, there's only really one method that adds to it, and another that removes from it. Those are easy write locks, and happen on start up and shutdown. Easy.

The rest of the time, the r/w mutex will be essentially a no-op, and that's fine with me. The refactoring was easy because all the same vector operations are the same, and most (say 80%) of the use cases are simple iterators on the vector's contents. All I needed to do was to put the scoped locks in the right place, and we're ready to go.

In the end, this is just as clean, probably faster, and a lot more well-understood. Good move.

I just finished spending a good hour tracking down a nasty little problem with the logic I had for creating new instruments on the fly. The problem turned out to really be me, and my preconceived notions about what the problem really was, but that's typically the case. The underlying problem was that I was thinking that the first new message for an instrument wasn't creating the underlying, but in fact, it was. That explained why I was seeing no errors.

No… the real problem was that I wasn't properly handling the case when I found it. It was made, but then the next time, I tried to find it, and it was missing - or so the code thought. In reality, I had failed to really detect that I'd found it, and act accordingly.

It's almost a coding standard in my mind now - For every 'if' statement, there had better be an 'else' clause. It would have saved me this headache, and when I saw it, it was clear that I was missing the else, and what to put in it when the value wasn't NULL.

Glad that's over. It was painful.