Archive for the ‘Coding’ Category

Lots of Progress on Conversational Data Service

Friday, September 24th, 2010

Ringmaster

Today I got a lot of good work done on the conversational data service. I decided to go with an instance variable approach to integration. Basically, I looked at the inheritance scheme but in the end I wasn't really happy with the way that was going to work. It wasn't easy to handle multiple services... it was going to require more code to be written by the user... it was going to be difficult to work it into the exchange feed's cache... lots of little things. But if I think of it as an instance variable to a class, then it's almost a client to the MMD that allows me to "publish" data to it. It might even be possible to make a single, unified client for the MMD - that which can request and provide. Might be interesting.

Anyway... today was a lot of good progress on the design and getting the headers written. Lots to do, but at least I have a nice, clear path now.

Google Chrome dev 7.0.517.13 is Out

Friday, September 24th, 2010

I was updating the configuration of WP Super Cache on my WordPress installs at HostMonster this afternoon and noticed that Google Chrome was saying there was an update - I said 'yes', and saw that it's up to 7.0.517.13. OK... nothing in the release notes on this, maybe I'm just a little early.

As for the "Don't be Evil"... the problem is that Chrome is a really good browser. It's a few in the management that I have issues with. So that's the rationalization I'm going with. Yup. It'll work.

Adding Ant, JUnit to Ticker Plant Project

Friday, September 24th, 2010

I had to take a little time-out this afternoon to install Ant and JUnit on the development machines we're using so that the Team can start working on the Java side of things. I have to say that while I'm not a huge fan of Ant, it's adequate, and the rules that people are building are pretty impressive. Of course, there's very little that you can't do in GNU make, as I've learned, but I can see that the Java/XML generation doesn't want to have to learn what I've learned in order to have a decent, stable build system for Java. So be it. It's not horrible.

JUnit is in the same boat, in my book. It's adequate, and yet everything it does can be done by your own test frameworks, it's just that they already have this one, and in that it makes sense to use it.

It took me a little time to get things all set up and configured properly - about 30 mins. Yeah, that's about as fast as I've ever done it. It helped to have used both a lot, and to know how to lay out the project to get it all working easily. Once I had it all in I just needed to run a few tests and check it all in.

Now they can get started on the Java client.

Working on the Simple Data Service

Thursday, September 23rd, 2010

Ringmaster

The first data service I wanted to create was the simplest as well - something that could take a simple C++ variant and expose it to the Broker under a provided service name for query and update. There are two basic ways the Broker's clients interact with the data services: 'call' and 'subscribe'. The first is a "give it to me" idea, and the second is "give it to me and keep it up to date" plan. At least as far as the simple data service goes.

What I wanted to build was a very simple API for the service so that a user would be able to simply provide the variable and a service name and the rest of the "magic" would be done by the base service class. It's a nice idea - one application could then "publish" a bunch of values and services by simply gathering the data and publishing it with one of these "one-line" calls. Sweet.

Well... today was that day. It's the easiest form of the service as it's a single variant and I just need to be able to send it back to the caller and then track updates. Thankfully, I've talked a lot with the designer of the broker and know just what I want to do for tracking the changes. It's going to be a very simple transaction system where the user will have to start a transaction on the object, make changes, and then commit the changes and when that happens, the changes will be sent to all the registered listeners for that variant.

One of these registered listeners will, of course, be the service so that when the changes are sent, it can package them up and send them to all the subscribers. Pretty simple.

There were a lot of little things that needed to be worked out - especially with the transactions. But in the end, I have something that's working just fine for the simple case.

Tomorrow: Start on the conversational mode.

Building the Other Side of the Broker – The Service

Wednesday, September 22nd, 2010

Ringmaster

Today has been spent putting together the design and initial coding of the other side of the broker - the service. Now that I've got the client working, and working pretty well, actually, it's time to be able to offer up data to the clients by becoming a service. There's really nothing new here - other than I can't control the multiplexing of the different requests. They are all going to come into one socket and that's controlled by the Broker. If I want to have multiple socket connections to the Broker, then I can split up the load, but I have no real control over the multiplexing. So I have to account for that in the design.

The problem with the design of the service is that there are two kinds of services I'd like to have, and they really need totally different ways of interacting with them. For one use case, I'd like to just give something a value and a name to publish it under. The service will then take this value (as a pointer or reference) and talk to the Broker, register this name, handle all the requests, and when and if I happen to change the value, it will send updates to those clients registered to receive updates.

The second one is more conversational in nature. This time, I'd like to be able to handle all the interaction with the clients. It's much more client-specific as we have to keep track of the state of the conversation with each client, but it allows me the greatest flexibility, and I have need of this guy as soon as I can write it.

So it's been a day of writing headers, working out the details, thinking again, fixing up the headers, and then, when it all seems solid, starting to implement the first class.

Updating the Broker Client to Handle Ping-Pong Protocol

Tuesday, September 21st, 2010

Professor.jpg

This afternoon has been spent upgrading the features of my broker client so that it can handle a greater variety of messages and conditions. The first thing I needed to do was to realize that there are more than the two types of connections that I had originally thought. There is the 'call' and the 'subscribe', but those names (I've come to learn) are a little misleading.

The 'call' is really a one-time pull of a specific answer from the data service on the other side of the broker. This is RPC. It can be thought of as a lot more than that, but that's what it really is. The protocol implements it as a 'call' request to the service, and the service responds with a serialized data structure that the client decodes. No problem.

The 'subscribe' is really a request for an open channel to the service, and for that channel to remain open until it's asked to be closed. In the simplest case, the 'subscription' can be for a data item, like a table of values, that then updates asynchronously at the client. Each time the data on the data service updates, it sends "deltas" to the subscribed clients, and they receive it and update their copy of the same data structure.

But it can be so much more than that. Unfortunately, that's as far as my C++ client took it. What I learned what that it can be a general communication channel with as varied and complex a communication between the client and the data service as the service builder wants. It can be a series of questions and replies, setting of values, just about anything you can imagine, and in the end, a 'close channel' command will close the channel and return the socket to the pool for reuse.

What I needed to do was to add in the ability to carry on this more general communication with the data service. It wasn't too hard - I just had to add the ability to send a message to the data service and not worry about what's returned or when. Then I had to implement a listening mechanism on the variant so that when the data service updates it, something can register to act on those updates.

With that, I then had a way to register for the updates (or errors) from the data service so I'd know when the 'pongs' arrived from the service, and I could now send all the 'pings' I needed. I still needed one more thing: an asynchronous read timeout.

When issuing the first request to a data service, I needed to have some idea of a timeout so that I don't hang there for ever waiting on a dead or locked service. With boost asio, it was a lot easier than I thought. My initial entry point for the async read looked like:

  void BEClient::startAsyncMessageRead()
  {
    bool       error = false;
 
    // first, make sure we have something to do
    if (!error && ((mSocket == NULL) || (!mSocket->is_open()))) {
      error = true;
      cLog.warn("[startAsyncMessageRead] the socket hasn't been connected"
                " to a service");
    }
 
    // if all is well, start the receive process for the "header"
    if (!error) {
      using namespace boost::asio;
      async_read(*mSocket,
                 buffer(mInbound.body(), mInbound.size()),
                 boost::bind(&BEClient::asyncHeaderRead, this,
                             placeholders::error,
                             placeholders::bytes_transferred)
                );
    }
  }

and to add in the async read timeout, I only needed to create a boost::asio::deadline_timer as an ivar with the same boost io_service as the socket itself, add a method to act as the target of the timeout, and then change the method to read:

  void BEClient::startAsyncMessageRead()
  {
    bool       error = false;
 
    // first, make sure we have something to do
    if (!error && ((mSocket == NULL) || (!mSocket->is_open()))) {
      error = true;
      cLog.warn("[startAsyncMessageRead] the socket hasn't been connected"
                " to a service");
    }
 
    // if all is well, start the receive process for the "header"
    if (!error) {
      using namespace boost::asio;
      async_read(*mSocket,
                 buffer(mInbound.body(), mInbound.size()),
                 boost::bind(&BEClient::asyncHeaderRead, this,
                             placeholders::error,
                             placeholders::bytes_transferred)
                );
    }
 
    // if we have a non-zero timeout, the start it now
    if (!error && (aTimeoutInMillis > 0) && (mTimer != NULL)) {
      // set the timeout in millis as a time in the future...
      mTimer->expires_from_now(boost::posix_time::milliseconds(aTimeoutInMillis));
      // if it expires, call the right method
      mTimer->async_wait(boost::bind(&BEClient::asyncReadTimeout, this,
                                     boost::asio::placeholders::error)
                        );
    }
  }

and then to capture the timeout, we only need to have a very simple method:

  void BEClient::asyncReadTimeout( const boost::system::error_code & anError )
  {
    // if we have a target, alert them of the error
    if (mTarget != NULL) {
      mTarget->fireUpdateFalied("asynchronous read timeout occurred");
    }
 
    // let's recycle this socket now... it's all used up
    mBoss->recycle(this);
  }

With this, we can make sure that the client has the ability to specify a timeout (in milliseconds) if they wish it, and if that timeout occurs, the async operation will be cancelled and all will be returned to it's starting place. It's not ideal, but hey... it's a timeout.

With this all in and tested, I could call it a day. What a day.

Updated My Variant Map to have Variant Keys

Monday, September 20th, 2010

Professor.jpg

This morning I started on some major improvements in my client to the cross-platform, load-balancing, resilient messaging system that has been built in-house for the next generation of trading systems at The Shop. The messaging broker is built in erlang by a guy that's far too smart for the locals - really pretty impressive guy. Anyway, the protocol in C++ is not as elegant as in erlang, but to be fair, it's pretty good for C++, as I've written a few in my days.

So I needed to start by retrofitting my variant's map structure with the concept that keys can be something more complex than STL std::string values - they needed to be variants themselves. This is just going to make the speed a little less, but it makes it much more like the Java-based data services which benefit from the existence of Object - something I do so wish C++ had, but I can see both sides of that now.

I modified the map to have variant keys, and then updated all the tests to work with the new key type. To be fair, the variant can hold an STL std::string, so most of the tests worked anyway, I just had to add a few to make sure I hadn't botched something really obvious.

So far, so good.

Massive Redesign of Another Project – Time to Play Catch-Up

Friday, September 17th, 2010

cubeLifeView.gif

Well... as the saying goes - Better late than never, or maybe Better sooner than later, but today I sat in with a few other guys and helped them imagine the complete redesign of their system that I'm going to be using for some critical data I need for my ticker plant. It's nice to be included, but as you can imagine, it'd be even nicer to not have to do this after the fact. But so it goes.

The new design is a lot simpler - socket connections are not of a specific "type". There are no "automatic" actions. You open a connection to the broker, he gets your message to the right guy, and you can maintain that connection for as long as you and the service provider wish. When you're done, the connection is severed, and the socket connections are put back into the pools from which they (most likely) came. There's no need to kill a socket, but you can do that if you want.

Like I said - Simpler.

Problem is, it's a massive change and that means that I've got a lot of coding to do to play catch-up and get my stuff ready for these changes. I'll be busy Monday, that's for sure.

So Much for ‘Don’t Be Evil’ – Switching Back to Firefox 4b6

Friday, September 17th, 2010

pirate.jpg

I've been a big fan of Google over the years. A really big fan. I have appreciated their clever interview questions posed as adds in trade magazines... I have really loved and used their Google Visualization API... but the company seems to have taken a turn for the worst. These problems with Skyhook are really too much.

If Google is going to behave that badly, I'm heading back to Firefox 4 and stopping using Chrome. It's not that I need that great a secondary browser, and I just have a hard time backing a company that is doing such bad things when clearly they don't need to. They are letting power corrupt them. They should have known better.

So until I see something that really convinces me, I'm back to Firefox.

UPDATE: it turns out that Firefox 4 takes a little less memory than Chrome 7.0.517.8 - nice. I'm hoping that what I've read about the performance of the JavaScript engine in Firefox 4 is as good as they say. I just would hate to have to move back to Chrome to get something that was responsive. Don't Be Evil - indeed.

Google Chrome dev 7.0.517.8 is Out

Friday, September 17th, 2010

Looks like Google Chrome dev has a few more bugs fixed, but the blog of the release notes is a little sketchy about what the exact nature of these bugs are. So I take it on faith that I'm going to like these changes, and install the update. Can't do a lot else.