What's it all about, Alfie?

This morning I was lamenting that the US Options Expiration calendar that I've been subscribed to for a few years now had a problem, and it was no longer available on iCal Exchange. I think from this note, it was lost, and the author never republished it. So I got tired of lamenting and did something about it. I went to the CBOE page and created a new US Options Expiration calendar. It took me all of about 15 mins and I'm sitting here now wondering why I didn't do this a long time ago.

Anyway... it's got the normal equity option expiration, as well as the VIX expiration and even the quarterly expirations. It's nothing fancy, but it's decent. I tried to keep the event titles as small as possible to make them look nice on my small iCal window, but it's a matter of taste, and that's the way it goes.

Use it if you want. I'll update it every year so you won't have to worry about that.

Whew! It's been a heck of a day... again. Today I was able to get a complete first cut of the lisp-like parser all done and checked-in. As with yesterday, the bulk of the work was really on the variant class and then just a little work in the functions of the parser. Today it was math operations and equality tests. I didn't think about it at the time, but my equality tests were more restrictive than I really wanted.

For example, originally, if I had the code:

  variant   v(25.0);
 
  if (v == 25) {
    ...
  }

it would fail. Why? Because the variant is a 'double', and the test is an integer. I had two different data types for these - as opposed to one "number" type. While that seems like a little thing, it's not - the rounding and the storage, not to mention the serialization size, all favor heavily the two distinct types.

So I had to go in and re-write all the equality and inequality operators to take this into account. Not horrible, but again, it was a few hours to get everything done right.

Thankfully, it's all done now, and the tests run great.

This morning I noticed that after a year of nothing, HandBrake 0.9.5 is out, and it's hardly a "bug fix" release. There are some major new features, and thankfully, it's going to keep do the job I need it to do - ripping DVDs, as long as I need it. There are new UI features, and the ability to run more than one at a time. I can't imagine doing that, but OK.

Fantastic update.

Today I've been hard at work filling in the parser I'm creating with the obvious collection of functions that any script is going to need. Things like add, subtract, multiply, divide... plus the more advanced math calculations - they all need to be done, but what makes this all slow going is the fact that to do this right, we need to implement these features in the variant class first, and then use them in the parser. In fact, using them in the parser is the easy part. It's getting them all in the variant class in the first place.

The reason it's slow going is that it's not enough to make a simple inequality set:

  bool operator==( variant & anOther ) const;
  bool operator==( const variant & anOther ) const;
  bool operator!=( variant & anOther ) const;
  bool operator!=( const variant & anOther ) const;

we're going to need the complete compliment of inequalities:

  bool operator==( variant & anOther ) const;
  bool operator==( const variant & anOther ) const;
  bool operator!=( variant & anOther ) const;
  bool operator!=( const variant & anOther ) const;
  bool operator<( variant & anOther ) const;
  bool operator<( const variant & anOther ) const;
  bool operator<=( variant & anOther ) const;
  bool operator<=( const variant & anOther ) const;
  bool operator>( variant & anOther ) const;
  bool operator>( const variant & anOther ) const;
  bool operator>=( variant & anOther ) const;
  bool operator>=( const variant & anOther ) const;

For the most part, this isn't too terribly hard, but you have to remember that I've got more than a dozen different types of values to check in a variant, and then there's the real work - all the convenience methods.

To make it easy to write:

  variant    v(25.0);
  if (v < 12) {
    ...
  }

I need to have a lot more operators as well. If I look at just the equality operator, I end up with something more like this:

  bool operator==( variant & anOther ) const;
  bool operator==( const variant & anOther ) const;
  bool operator==( varmap & anOther ) const;
  bool operator==( const varmap & anOther ) const;
  bool operator==( varlist & anOther ) const;
  bool operator==( const varlist & anOther ) const;
  bool operator==( uint8_t aValue ) const;
  bool operator==( int aValue ) const;
  bool operator==( int64_t aValue ) const;
  bool operator==( uint64_t aValue ) const;
  bool operator==( float aValue ) const;
  bool operator==( double aValue ) const;
  bool operator==( bool aValue ) const;
  bool operator==( std::string & aValue ) const;
  bool operator==( const std::string & aValue ) const;
  bool operator==( char *aValue ) const;
  bool operator==( const char *aValue ) const;
  bool operator==( uuid_t & aValue ) const;
  bool operator==( const uuid_t & aValue ) const;
  bool operator==( secID_t & aValue ) const;
  bool operator==( const secID_t & aValue ) const;
  bool operator==( error_t & aValue ) const;
  bool operator==( const error_t & aValue ) const;

So even if we are as clever as can be, this is a ton of code to write, and test. It's just brutal at times to make sure you haven't made any typos in the code.

I will say that I was very pleased with the one simplification I made. It's pretty easy to see that you can write the inequalities in terms of one another. However, that can take a while, and you might not end up with valid inequalities. For example, when you have two lists - which one is greater than the other? It's not easy.

My solution was to implement the:

  bool operator>=(...) const;
  bool operator<=(...) const;

methods, and then define the others in terms of them:

  bool variant::operator<(...) const
  {
    return !this->operator>=(...);
  }

So that I could be assured of at least having the "equality" in the code. Then, the others are logical combinations of these. It's not perfect, but it saves time and code, and it's a lot less headache than trying to implement a greater than on a list.

I'm back from a nice week off and it's nice to get back into the swing of things. I'll admit it was a little tough this morning, but it really worked out pretty well. Now that I've caught up with email, and reading, it's nice to be back in the full swing of things.

Vacation is nice, but it was also a lot of work with the house. It's nice to be creative once again. Ahhh...

I noticed (while on vacation) that there was an emergency bug fix for WordPress to make 3.0.4, and it was quickly available on HostMonster, so I took the time to make sure that everything got upgraded nicely. It's not a lot of work, and there were plenty of ways to know that there was an update in the offing, so it's not bad, but it is sad that WordPress is such a target of goofballs that it takes eternal vigilance to make sure servers aren't abused. Sad, but true.

I'm to the point that I believe all my Broker re-write issues are solved. I can use no pooling on the sockets, and it works, and I can use pooling and it should work just fine. The guys working on the Java side of things aren't having as good a time of this as I am. Don't get me wrong, they're sharp guys, and I believe they understand the Java classes they are using, but the complexity of some Java libraries is really quite astounding.

Case in point: J2EE. It couldn't be harder to get something going on J2EE. Tomcat is hard enough - there should be something like JSPs or PHP that will allow you to place the java file in any directory and have things "just work", but I understand the "Java way", and while it's not perfect, Tomcat is awfully nice, and very capable - you just have to work within it's paradigm.

But this work we're doing on the sockets... Holy Cow! I"m glad I'm not doing the Java side. The C++ side has had it's problems, but still, it's very easy to see what's going on in the code, it's just a question of what's happening in the libraries. It's not a question of smarts, it's really a question of familiarity with the libraries. If you're new to the way it does things, as I was with boost asio in the beginning, it's a pain. But once you really know how to use the libraries, and how the library wants you to formulate a solution, then things get a lot better a lot faster.

I just feel for the guys doing the work on the Java side. At least one of them is new to this library, and this is a real trial by fire. The complexity is staggering to him, and it's clear to me that a really well designed C++ library can make a far simpler solution than a lot of the Java library/frameworks I've seen out there.

In the midst of working on the changes necessitated by the Broker's re-write, I found myself in a very nasty little problem. I am trying to do things quickly, and my test cases are often far worse than any real-world use is going to be, but they have served me well, and they were pointing out a problem I was having this morning.

If I created new updater instances for each request, and deleted them after they were no longer needed, I ended up with a very fast create-use-delete lifecycle. This lead to a segmentation fault in boost's io_service - specifically, in it's run() method. The core dumps were of little to no help whatsoever, and I was left trying to diagnose the problem from my end.

If I didn't delete them right away, but threw them into a pool, and still created new ones, only clearing out the pool at the end of the application, then everything was fine. It seemed like it was just the short lifecycle connections that was the problem. Very nasty.

The seg faults weren't on anything related to boost asio, either. They were on the line right after the context switch after the closing of the socket connection. I spent hours debugging the code to find that guy.

I came to the conclusion that there was something in the io_service that wasn't getting a "chance" to handle the socket closing before I deleted it. So I changed my code ever so slightly. Originally I had:

  if (si->second.pool.size() >= MAX_UPDATERS_IN_POOL) {
    cLog.debug("[recycle] the pool is full - deleting the updater");
    delete anUpdater;
  }

to:

  if (si->second.pool.size() >= MAX_UPDATERS_IN_POOL) {
    cLog.debug("[recycle] the pool is full - deleting the updater");
    anUpdater->mChannelID.clear();
    anUpdater->disconnect();
    // we need to let the io_service have a go at it
    boost::system::error_code  err;
    mIOService.poll_one(err);
    // ...and now we can successfully delete the updater
    delete anUpdater;
  }

The difference was stunning. No more crashes, and the code was rock solid every time I ran it. Amazing. I'm going to have to remember this. It's like a little context switch for the io_service so it can detect the close of the socket before it's deleted.

Several things to finish on the re-write, but it's getting close now. Nice.

As I was doing a little testing this morning I realized that the Broker wasn't working like it should. When I pinged a few folks, it was clear that there had been several changes I hadn't been told about and so I had a very abrupt detour to update all my Broker-facing code to make sure it was all working.

Some things weren't too bad - the codec's code for a map went from 'M' to 'm'... the Dates are now passed as microseconds since epoch... simple things. But then I quickly found that there were some that weren't simple at all. Not in the least.

The protocol for service registration changed. The interaction with the registration service itself changed. The recycling of open sockets wasn't working - in short, it was a massive change to the codebase.

I've spent the bulk of the day getting things converted. It's touched a lot of the code, but I think I'm getting very close. There are still a few things I need to hammer out, and I'm hoping to get to them in the morning. The prickly one is that the 'close channel' message - that which indicates that the socket is clear to be recycled, can, if the conditions are right, close the actual socket. When that occurs, it's happening too fast, and the object is getting dumped before it's work is done.

I need to figure out how to fix this so the events happen in the right order.

This morning I'm continuing to work on the functions for my lisp-like parser. What I'm working on now is really the functions themselves and not how they fit into the other classes of the parser to compute values. It's a little slow going at first because I want to duplicate as much functionality as possible from the Java-version, and as you'd expect, there are absolutely no comments in the Java code. So I'm ending up chatting to one of the developers on the project and he's giving me the details on the functions.

Nothing hard, but it's slow going. I'm hoping once I get a few done, it'll go a lot faster.

Heh... that's optimism for you...