This morning I noticed that Google Chrome dev 19.0.1061.1 is out and it's got a few nice things in it. Like a new V8 javascript engine (3.9.13.0), and support for remote file systems - could be interesting stuff. Glad to see they are still making improvements in V8 - that's something that I think I'm going to end up in sooner than later, and it'll be nice to have some improved performance there.
This morning I wanted to try to get some of my CVS repos converted to Git so that I could have the complete repo on my laptop and not have to worry about internet connectivity. I'm a big fan of CVS, and it's simplicity, but Git is the clear next-generation of CVS, and it doesn't need the connection to the server that CVS does.
Recently, I'd read that there was a simple git command: git cvsimport that would convert a repo, and I just had to try. The first thing was that I needed to have a program called cvsps. This is some other tool - not part of CVS, not part of Git, that I needed to get. I realized this as I tried to convert my first repo, and failed saying it couldn't find this app. So first things first, get the tools I need.
A simple google search revealed that the source code for cvsps was held in a very simple web site: http://www.cobite.com/cvsps/. I downloaded the latest stable code, and read the README. It's a simple make; make install, and I'm on my way:
$ cd cvsps-2.1 $ make $ sudo make install
The cvsps executable is now in /usr/local/bin and /usr/local/share/man. That's all we need - over and above Xcode 4.3 and it's command line tools (cvs, git).
While I've read that this can be done using a remote pserver $CVSROOT, it's a good idea to just get a local copy of the complete CVSROOT to work from. Since I'm in a stable state with that, it was pretty easy to copy it to my TimeMachine external drive:
$ cd /Volumes/Reststop $ scp -r frosty:/usr/local/CVSroot .
It only took a few minutes, and now I've got all the "source material" I need.
The process is pretty simple - you have to act as if you're starting a new git repo, but instead of the git init command, you issue the git cvsimport command. It's got a few arguments, but it's pretty simple to use.
For this example, I'm calling the new Git repo the same name as the old CVS repo, but I'm guessing if you want, you can change the names.
$ cd git $ mkdir MyProj $ cd MyProj $ git cvsimport -p x -v -d /Volumes/Reststop/CVSroot MyProj
At this point, you have a new git repo but the origin is not set, so it's not "going" anywhere if you try to push it. Since I'm using gitosis on my home git server, it's a simple process to update that to add in the project(s) I'm migrating into the proper groups, and then push those changes up to the server before I try to set the origin of the new git repo.
Assuming that you have the server set up, and it could be that you're using GitHub and not gitosis, then all you need to do is to set the origin and push it up:
$ git remote add origin git@git.myplace.com:MyProj.git $ git push origin master:refs/heads/master
It's possible to now go in and mess with the git config to set the master right, but I've found is just as easy to remove this new repo, and clone it again from the server. If I got it right, the history will be there, and I'll be sure it works. If not, I can start over. Simple.
It's been a lot of fun getting these guys over to git. Now I can use all the tools and fun I've had with git in the last year to these projects. Very nice!
One of my favorite things is to work with databases in code. Persistence and database hits are a blast as they get you a place to save stuff that, if you design it right, you can view from just about any tool on the planet. Can't say the same for redis or mongoDB. My Greek Engine gets it's instrument data from a local replica copy of a master postgres database, and should the local copy fail - or be down, it should auto-reconnect to the master and just function off that one. If he's dead… well… that's when it's time to get serious about getting things working.
The first thing I needed to do was to consolidate all the database activity to as few a number of places as possible. Thankfully, I had a simple execute() method that did about 90% of what I needed. It just took a few minutes to make that the only way to hit the database, and then I could focus on making that a little more fault-tolerant.
The idea is simple, really: put it in a retry loop, limit the number of retries, and then for each retry, hit The Broker for the correct database connection parameters to use. If the Broker is wrong, then I'm in real trouble, but it's not, so I'm OK. (Famous last words.)
Add a little logging, remove some error codes, and we're ready to go. It really didn't take me all that long, and the results are much better. When, and if, the database goes down, we'll fail over to the master. When we get the local copy up, we can issue an IRC command to repeat the process, and the local one will again be used. Simple. Clean.
Great.
This morning I came in to see that some of the exchange feeds on one of the staging boxes of mine hadn't shut down properly. When the exchange test data flooded in, it made a mess, and that was no good at all. The only code that seemed to matter was a simple iterator on the TBB concurrent_vector. I've had issues with this code before - and always moved away from it in favor of a simple std::vector and a mutex of some sort. Here was another case of the exact same thing.
Now I'm not saying that the concurrent_vector is a mess, but I think that it, along with the concurrent_map are a little trickier than normal to work with. The iterators have built-in locks, and that makes it very easy to write dodgey code. I think that's what happened, but I can't prove it.
Far easier to use a simple std::vector and then a TBB spin_rw_mutex_v3 to protect it. Virtually all the access to the vector is read-only, there's only really one method that adds to it, and another that removes from it. Those are easy write locks, and happen on start up and shutdown. Easy.
The rest of the time, the r/w mutex will be essentially a no-op, and that's fine with me. The refactoring was easy because all the same vector operations are the same, and most (say 80%) of the use cases are simple iterators on the vector's contents. All I needed to do was to put the scoped locks in the right place, and we're ready to go.
In the end, this is just as clean, probably faster, and a lot more well-understood. Good move.
This morning I saw that Google Chrome dev 19.0.1049.3 was out, and with it, a few fixes, a new V8 javascript engine, and a few Mac GPU fixes that I'm not sure effect me, but it certainly can't hurt. I'm always glad to see the progress.
Today I was having a really bad day with my UDP feed recorders. They filled up a 3TB+ drive array and I could not figure out why. As I dug into this, I starting seeing a pattern that was really bad: the files in a directory didn't add up to the output of du. So I wanted to test the theory.
The trick was, I wanted to just run a simple command - not write a script or a program. Just a command, but it wasn't clear what to do. So I hit google, and there was the simple result:
ls -lsa | awk '{ sum += $6 }END{ print sum }'
The ls is obvious - the 6th column is the number of bytes, but the awk line is the real beauty here. I've used awk a lot before, but I didn't know it had dynamic variables like this. And then to have the END tag to put it at the end of the command. Simply brilliant.
This is why I love linux/unix. You don't have to write groovy scripts if you understand the system. Love it!
This afternoon, I ran into what appears to be a problem with boost ASIO. I'm reading a framed TCP message where the fixed-sized header includes the number of bytes in the rest of the message, and then trying to read the rest of the message with the boost::asio::transfer_all() completion code. The goal of this says to either return the full buffer or an error. What I'm seeing from time to time is that I'm asking for n bytes and I'm getting m where m is a good bit below n.
I have been able to catch this in the reader, flag it as an error, and then notify the client to re-issue the request. These retries always seem to work (after I reset the socket connection), so it's not the server or the client - it's the communication between the two. Not a lot of fun, but at least I have a semi-reliable solution with the retries. This will hold until I get back to this in a few days.
But I'm just shocked that there's a problem in the boost ASIO code. I know it's possible to just drop the connections and not face the problem, but that seems to be excessive. What I want is to track down why this is happening.
Hopefully, I'll get to it.
This morning a user wanted to be able to see if a Broker was alive and listening on the machine and port it was supposed to be on. This came up because Unix Admin decided to do a kernel update on all the staging machines last night, and we didn't have everything set to auto-restart. Therefore, we had no processes for people to look to. Not good. Thankfully, we have backups, but how does a user know when to hit the backup? After a failed call, sure, but with retries built into the code, that can take upwards of 10 sec. What about something a little faster?
Seems like a reasonable request, and to this morning I added a little isAlive() method to the main Broker client. It's very simple - just tries to connect to the specific host and port that it's supposed to use, and if something is there listening, it returns 'true', otherwise, it returns 'false'. Really easy.
Boost ASIO makes it a little un-easy, but still, it's not too bad:
bool MMDClient::isAlive() { bool success = false; // only try if they have set the URL to something useful... if (!mHostname.empty() && (mPort > 0)) { using namespace boost::system; using namespace boost::asio; using namespace boost::asio::ip; // getting the connection in boost is a painful process… tcp::resolver resolver(mIOService); std::ostringstream port; port << mPort; error_code err = error::host_not_found; tcp::resolver::query query(mHostname, port.str()); tcp::resolver::iterator it = resolver.resolve(query, err); tcp::socket sock(mIOService); if (err != error::host_not_found) { err = error::host_not_found; tcp::resolver::iterator end; while (err && (it != end)) { sock.close(); sock.connect(*it++, err); } } // if we got a connection, then something is there… success = !err; // …and close the socket regardless of anything else sock.close(); } return success; }
While using boost isn't trivial, I've found that the pros outweigh the cons, and it's better to have something like this that can handle multi-DNS entries and find what you're looking for, than to have to implement it all yourself.
This guy tested out and works great. Another happy customer!
This morning the Google Chrome Team is at it again, and released 19.0.1041.0 which is primarily a bug fixes release. It's impressive to see these updates coming so quickly, as it's only been a few days since the last significant release. They are certainly back at it, and moving things forward. Nice to see.
This morning I wasn't at all surprised to see that Google Chrome dev 19.0.1036.7 was released, as I'd noted a few days ago that the beta channel had a more recent version than dev, and that's just not how they typically release things. So they've upped the major version number and included the latest V8 javascript engine (3.9.4.0) and a few UI issues. It's interesting to see that there are more than a few Windows-only fixes, but that's nice to see too - Mac OS X was a little more stable - or is still buggy, hard to say.
Great to see the improvements keep coming!