What's it all about, Alfie?

I've been talking about The Broker for a while, and it's been through at least five iterations/revisions/re-writes that I can remember, but we're in our sixth now. What I've seen, as I hit it the hardest of almost anyone, is that the java version built on Netty is just too complex. The design is reasonable, but even that's complex because of the direct dial interface we put on The Broker to remove the massive load some clients can generate. It just wasn't working. So we tried a new approach.

I mentioned distributed erlang, and asked why we didn't build the original erlang Broker using that? It seemed to handle all the registration/location service issues. It also handled the problem of a centralized service in general. When one client makes a request to one service, it's only those two nodes that will see the load - not anyone else. It's a really nice idea - extremely simple, and therefore, probably the best.

So my boss started on this next version. He and I worked on it today and we're down to just a few things that need to be finished: the service registry and the service load (for load balancing). I'll do those tomorrow and we'll be feature complete.

The beauty of this design is that we can add as many native services to the Broker as erlang modules in the distribution package. Each one is very simple, and is loaded on demand from the elang runtime. This places us in the wonderful position of being able to add in the login service, the configuration service, and any other service that we want in the native erlang. Because this new design places a Broker node on each box running a service, this means that for the most used services, like login and configuration, we won't even go off-box to get the response.

This is going to have a huge impact to the reliability of the Broker. It's going to be able to be stopped/started on each node, as needed, it'll connect into the grid and share it's state, and will be able to handle all the routine services on it's own. This is a major win for me. I think it's going to make it great.

Well... the first week of the four week "delivery cycle" of the new Greek Engine has come and gone, and I have to say, if we get done with it in the next three weeks, I'll be surprised. Maybe not off by much, but there's so much to do, and with two new guys on the team, one very junior, it's a lot of work on my part just to keep them for imploding.

On the other hand, I have to say that things are coming together. They are taking shape in the code, and things are getting written - just not fast enough, and the hardware was always going to be an issue. There's just not enough time to get all the hardware purchased, installed, built and running. It'll be close, if they hurry, but I've already found out that I'm going to be delayed at least a week on some hardware coming from another data center. Doesn't sound like much, but that's 25% slippage.

But progress has been made. We have better integration between the two projects - my ticker plant and the greek engine, and we at least know the sections of code we need to build in order to make this thing work. That's a big help. It's just a matter of time.

This morning I saw the expected update of Google Chrome dev to 12.0.712.0, marking the move to 12.x. I say this was expected because they just announced that the 10.x series was in beta (or release), and that meant that they needed to "up the voltage" a bit on the 'dev' channel. The third number is all that seems to matter, and there are still a few nice things to find in this update.

What seems to be missing is any new features that they might want to add after Firefox 4 went final a week ago. I've played with Firefox 4, and it's OK, but the rendering is the biggest issue, and while that's not quite right, I'll stick with Chrome. But Firefox is smooth, and the "Spaces"-like support is very cool and keeps the tab count to a minimum. Very nice indeed.

Note: the old icon is back, too. Seems "plastic" now.

One of my very favorite database libraries is SQLAPI++ because it's very well designed, works with a huge number of databases, is thread-safe, and doesn't require you to link in the underlying database libraries when you build your code. It's nice. So when it came time to hit a database from linux here at The Shop, I naturally turned to an old friend. But there were some ugly truths lurking there for me, and I had to spend quite a bit of time getting things sorted out.

First, off, SQLAPI++ on linux doesn't talk to MS SQL Server through any libraries like FreeTDS. Nope, you have to go through iODBC. Ick. Thankfully, FreeTDS has the ability to work with iODBC, but getting things set up and tested was a pain, so here's what I had to do.

Get everything installed:

• iodbc
• libiodbc2
• sqsh

then start to configure things.

Get FreeTDS going by editing /etc/freetds/freetds.conf to look something like this:

  [devSQL]
     host = dbhost
     port = 1433
     tds version = 7.0

Now you can put in the default .sqshrc and prove that you can talk to the database with sqsh:

  $ sqsh -S devSQL -U me -P secret
  1>

Success. That's a good first step. Now let's configure ODBC. Edit /etc/odbc.ini to look something like:

  [ODBC Data Sources]
  devSQL = Devel SQL Database

  [devSQL]
  Description = Development database
  Driver      = /usr/lib/odbc/libtdsodbc.so
  Trace       = No
  Server      = dbhost.yoyo.net
  Database    = master
  Port        = 1433
  TDS_Version = 8.0

and then edit /etc/odbcinit.ini to look something like this:

  [FreeTDS]
  Description = TDS driver (Sybase/MS SQL)
  Driver      = /usr/lib/odbc/libtdsodbc.so
  Setup       = /usr/lib/odbc/libtdsS.so
  CPTimeout   =
  CPReuse     =

At this point, we can run the iODBC admin utility to see that it was set up properly:

  $ iodbcadm-gtk &

and the driver should be right there. Use this to "Test" the connection, and you should see that it's there and working.

Finally, use the SQLAPI++ test client to see that everything is working there as well:

  $ test64
  1.    Oracle
  2.    SQL Server
  3.    DB2
  4.    Informix
  5.    Sybase
  6.    InterBase
  7.    SQLBase
  8.    MySQL
  9.    PostrgeSQL
  0.    ODBC
  0
  Client version: unknown before connection
  Database name (connection string):    devSQL
  User name:   me
  Password:    secret
  Server: Microsoft SQL Server Release 10.00.2714
  Server version: 10.0
  Client version: 0.82
  $

I was talking to a developer today and we got to talking about the fact that the STL map spec says that it's iterator is not invalidated on insert or removal - except in cases where the iterator is on the removed object. So it got me to thinking - what happens when you invalidate an iterator?

So I wrote this code:

  #include "map"
  #include "string"
  #include "exception"
  #include "iostream"
 
  using namespace std;
 
  int main(void) {
    map<string, string> map_test;
    map<string, string>::iterator iter_map_test;
 
    map_test["AAAAA"] = "11111";
    map_test["BBBBB"] = "22222";
    map_test["CCCCC"] = "33333";
 
    iter_map_test = map_test.find("BBBBB");
 
    map_test.erase("BBBBB");
 
    try {
      string value = (*iter_map_test).second;
      cout << "got : " << value << endl;
      ++iter_map_test;
      cout << "next: " << (*iter_map_test).second << endl;
    } catch ( exception & e ) {
      cout << e.what() << endl;
    } catch ( ... ) {
      cout << "generic exception." << endl;
    }
    return(0);
  }

and the results are amazing (to me):

  $ g++ maptest.cpp -o maptest
  $ maptest
  got : 
  next: 11111
  $

So the iterator is resetting itself when invalidated. That's very interesting! Now it's not what I expected, and I'm not certain I want to risk it, but it's nice to see that there is something non-fatal about the process. That part, it seems, the spec is right on the money about.

Clever dudes.

Today starts the push for the next four weeks to get out a new greek engine at The Shop. I'm not at all sure it's even possible, but I'll give it everything I've got for the four weeks to see if it can be done. There are a lot of things that have to happen in the right order, and pretty quickly. I've been told they'll all get done, but I have my doubts. It's just a lot of things that have to happen just so.

This morning already I'm spending too much time getting the next release of ZeroMQ checked out. It should be a pretty simple thing to check out, but it's taking far longer than the previous versions. Still, I have to get this done, and then I have to get SQLAPI++ working for the FreeTDS drivers we have on the boxes. All this in order to load up the data from the MS SQL Server databases that holds it all.

Yup, it's going to be tough to get it all done, but I'm willing to give it a try.

This morning I noticed that Google Chrome dev 11.0.696.14 is out, and it seems to have several updates for a small minor point change. OK with me, I'm glad to see the continual improvement of the dev channel. I was also interested in the about:gpu, and tried it on my MacBook Pro - very neat. Glad it's used in rendering.

This afternoon I realized that git on my Mac Mini at home had the 64-bit version of git installed from the Mac OS X git installer, and that was a big mistake as it's a simple Core Duo CPU which can only run in 32-bit more on Snow Leopard. I had the 64-bit version for my MacBook Pro, and my iMac, but forgot to get the 32-bit version for the Mac Mini. Bummer.

Clearly, it'd be great if they had a universal build so I didn't have to deal with this, but hey... they build it and I can just keep things straight. My bad. Plain and simple. So I got the most recent builds for x86_64 and i386, and put the 64-bit package on my MacBook Pro, and tonight I'll install the 32-bit version on my Mac Mini. There. It'll work.

It's nice to see:

  $ git --version
  git version 1.7.4.1

Nice. Love it when things "just work".

[3/18] UPDATE: this morning I installed the latest updates on my Mac Mini (i386) and on my iMac (x86_64) so all boxes are up to date and working fine.

I was looking on the web today for some C++ library that would wrap the erlang interface (ei) that appears very full-featured and yet somewhat cumbersome for a C++ codebase. I looked at several until I found one that looked to be full-featured, well thought out, and complete. I stopped when I found eixx on GitHub. This looks to be very nice. I haven't used it - yet, but if we go forward with the idea of having each service connect into the distributed erlang network and use that as the way to talk to the services, and serve up data.

The real advantage with this scheme is that we no longer have to deal with async I/O on the sockets for the client or the server. It's all encapsulated in the erlang runtime/library and that takes all the responsibility from our hands and puts it in the hands of the guys who made the language. Very nice.

Anyway... it'll be interesting when we move on this. I'm actually looking forward to it.

Today has been a mixed bag... I did get a little development done on the ticker plants - just a few little things to polish up some things, but I also had hour-long design meetings (not bad), and then multi-hour-long debugging meetings where there was no hope of actually finding a problem because the developers had so little experience developing on linux that there was no hope of a successful test, and they really just needed to understand the proper way to edit/compile/test code.

The design meeting was really kind of interesting. We have The Broker, and it's been an erlang process, and a Java process, but it's always been centralized. I mentioned today that I wondered why it wasn't using Distributed Erlang, and let the entire brokerage system sit on all the machines and handle everything in a more distributed manner. Let erlang handle the registration, and the message passing. We can use J interface (Java library) and el (C library) to make our server applications appear as distributed erlang nodes. This makes it much easier to do all the things the Broker does.

Sure, the clients still can use sockets to connect to the Broker - or should I say A Broker, where we can have one running on each server in the server room. The client (defined as processes that aren't nodes in the distributed erlang system) can then use the socket interface to connect to a Broker and work as it always has, but the wrinkle is that it's only really brokering his traffic. This gets rid of a lot of the problems we had faced in the past - and went to great lengths to try and solve.

So much gets easier if the main components of the system are all distributed erlang nodes. Very nice solution to the problem.

The debugging session was only slightly productive, and ultimately disappointing. The code was compiled on CentOS5 and run on Ubuntu 10.04.1 - I'm not at all surprised that things broke. Far too different a version of libc, gcc - everything, really. We couldn't even get the code to build on Ubuntu. They need to step back, build the code on the box they are going to run it on, and then move forward.

It's slow going sometimes.