Archive for February, 2012

Wild Problem in Boost ASIO Async Reader

Friday, February 17th, 2012

Boost C++ Libraries

This afternoon, I ran into what appears to be a problem with boost ASIO. I'm reading a framed TCP message where the fixed-sized header includes the number of bytes in the rest of the message, and then trying to read the rest of the message with the boost::asio::transfer_all() completion code. The goal of this says to either return the full buffer or an error. What I'm seeing from time to time is that I'm asking for n bytes and I'm getting m where m is a good bit below n.

I have been able to catch this in the reader, flag it as an error, and then notify the client to re-issue the request. These retries always seem to work (after I reset the socket connection), so it's not the server or the client - it's the communication between the two. Not a lot of fun, but at least I have a semi-reliable solution with the retries. This will hold until I get back to this in a few days.

But I'm just shocked that there's a problem in the boost ASIO code. I know it's possible to just drop the connections and not face the problem, but that seems to be excessive. What I want is to track down why this is happening.

Hopefully, I'll get to it.

Refactoring the Feed Recorders

Friday, February 17th, 2012

Building Great Code

This morning my manager stopped by to talk about the problems I've been having with the transfer of the data from the feed recorders to the archive server. He talked to my ex-manager about the issue, and he came up with the idea that we could just increase the frequency that we wrote to the filesystem from the recorder, and then stay using the filesystem where things look to be stable.

It's a pretty good idea. I needed to work out a few things - like the filenames, and how to deal with that, but in general, the idea is sound: if there's an existing file being "filled", then add to it, if not, create a new one and it becomes the current file. Once the file exceeds a certain size, don't write to it again, and let the next block of data create a new file.

The last trick I added was to have the writing of the file include the renaming of the file if appending data - to include the ending timestamp. This makes it such that the files are always consistent, and if the recorders crash, we're not loosing much as they only have 5 sec of data in them. The rest is written to the filesystem, with the updating filename so that it's easy to use at any point in time.

Empty block will naturally take care of themselves, and we're looking pretty good. It's a solid plan.

So I took out all the Broker-related stuff from the feed recorders, and then cleaned things up on the archive server, so that I got back to a nearly neutral state, checked that in, and then started updating the recorders to write out their data in this manner. The server was pretty much untouched as it now functions completely on the filesystem.

I started the tests, and sure enough, about every 5 sec, I get an update and the file gets a little bigger, and the name changes. The CPU load is a little bigger, but it's not bad, and the payoff should be significant -- the archive server should just work.

I need to let it run for a bit, and then hit it with my tests to see, but I'm very optimistic about the possibilities.

Xcode 4.3 is Out!

Friday, February 17th, 2012

xcode.jpg

This morning I found out that Xcode 4.3 was released, and this time, it was a regular app on the Mac App Store! No more "installer app" that you download and then install from - this places all the files and directories right where you need them. I was a little surprised to see everything fall right into place, but the apps weren't in /Developer any longer. Maybe this is part of the plan to clean things up, but it's OK with me. I've moved all the docs I had out of that directory with my new laptop, so it's simple for me to use the new Xcode location.

Plus, it's just getting nicer and nicer. The tools and the OS for this Mac are the most amazing things I can imagine. Had I had tools like this in grad school, I'd have made my simulator run on distributed Macs and the graphics would be so nice, it'd make your eyes tear up.

Not to mention the thesis would not have been vtroff and tape for figures.

Everything gets better, and this update is no exception. I'm really glad they are pushing forward.

Different Approaches to the Same Problem

Thursday, February 16th, 2012

I primarily deal with a certain kind of recruiter - the kind that matches my personality: non-spastic, non-agressive -- not a used car salesman, but today I had a meeting with a recruiter that I first met several years ago while I was at O'Connor, and I was reminded why I didn't deal with him then, and I doubt he's going to bring me a useful lead this time.

The guy who could be a used car salesman comes across (to me) as a used something salesman, no matter what that is, and I don't think I'm alone on this point. I think most reasonable people react to this kind of salesmanship the same way - keep their distance, and doubt everything they say. I'm not at all sure I want to be represented by a guy like this, but I can also respect that if he's talking to people that respond to that kind of dealings, then he's the right guy for me.

I'm starting to look at my options, and part of that is using a recruiter. In this industry, in this day and age, I can't imagine getting a job without a recruiter - or a friend, in the company you want to work for. It's just so unusual anymore to receive good unsolicited resumes, that I'm not sure that anyone would really pay attention to one if it didn't come from a reputable source.

So I deal with the guy that might seem right at home on the used car lot. It's a necessary step to really find out what's out there. I have my fingers crossed that they look at me with a little better eye than they do him.

Struggling with Data Transfer Issues

Thursday, February 16th, 2012

bug.gif

I spent about 60% of my day today dealing with user issues related to the latest testing cycle, and the rest of my day trying to get a good handle on this data transfer issue that I'm having with my archive server and feed recorders. It appears that so long as the recorders write the files to the filesystem, the server can read them and decode them just fine. But when I go through The Broker for the in-memory buffers in the recorders, I get all kinds of junk.

It's not every time, though - it's not hard to repeat on the scale that I need it to work, but it's very tricky to repeat on the small scale that makes it easy to find the problem. It could be a boost ASIO problem -- I know I made some changes recently hoping that would clear things up, but maybe we have a problem with the io_service instances? Don't know.

I do know that when I only look at the filesystem, it's fine. Crud.

So today I spent as much time as I could gather to narrow down the problem. As of the end of the day, I'm thinking it's not in the serial transfer, but in something after that. What? I con't know, but it's looking like it's getting into the process just fine.

We'll have to work on it more tomorrow.

OS X Mountain Lion this Summer!

Thursday, February 16th, 2012

Mountain Lion

When I first heard this on Twitter, I thought it was a lie, but that's only because it didn't come through the normal channels. OS X (no 'Mac' on the front) Mountain Lion (10.8) is going to be released this Summer, and will then be on a yearly release schedule like iOS. Some of the new things I'm excited about are a more general inclusion of iCloud, the unification of Messages across iOS and OS X, and, of course, Gatekeeper. This is going to make it a lot harder for folks to slip in viruses and malware for my family. It's something I've tried to drill into them, but I'm afraid they still don't see it.

With Gatekeeper, the OS will do it for them. I like that.

I'm sure there will be a slew of other nice little updates, but the yearly updates are exciting too. The Mac is the best platform on the planet. I really believe that. Wonderful news!

Quick Checking of Listening Socket Using Boost ASIO

Wednesday, February 15th, 2012

Boost C++ Libraries

This morning a user wanted to be able to see if a Broker was alive and listening on the machine and port it was supposed to be on. This came up because Unix Admin decided to do a kernel update on all the staging machines last night, and we didn't have everything set to auto-restart. Therefore, we had no processes for people to look to. Not good. Thankfully, we have backups, but how does a user know when to hit the backup? After a failed call, sure, but with retries built into the code, that can take upwards of 10 sec. What about something a little faster?

Seems like a reasonable request, and to this morning I added a little isAlive() method to the main Broker client. It's very simple - just tries to connect to the specific host and port that it's supposed to use, and if something is there listening, it returns 'true', otherwise, it returns 'false'. Really easy.

Boost ASIO makes it a little un-easy, but still, it's not too bad:

  bool MMDClient::isAlive()
  {
    bool       success = false;
 
    // only try if they have set the URL to something useful...
    if (!mHostname.empty() && (mPort > 0)) {
      using namespace boost::system;
      using namespace boost::asio;
      using namespace boost::asio::ip;
 
      // getting the connection in boost is a painful process…
      tcp::resolver   resolver(mIOService);
      std::ostringstream  port;
      port << mPort;
      error_code              err = error::host_not_found;
      tcp::resolver::query    query(mHostname, port.str());
      tcp::resolver::iterator it = resolver.resolve(query, err);
      tcp::socket             sock(mIOService);
      if (err != error::host_not_found) {
        err = error::host_not_found;
        tcp::resolver::iterator   end;
        while (err && (it != end)) {
          sock.close();
          sock.connect(*it++, err);
        }
      }
      // if we got a connection, then something is there…
      success = !err;
      // …and close the socket regardless of anything else
      sock.close();
    }
 
    return success;
  }

While using boost isn't trivial, I've found that the pros outweigh the cons, and it's better to have something like this that can handle multi-DNS entries and find what you're looking for, than to have to implement it all yourself.

This guy tested out and works great. Another happy customer!

Dealing with Tricky End-of-Day Pricing Issues

Wednesday, February 15th, 2012

bug.gif

One of the things that has come up in recent days is the quality of the end-of-day marks that the greek engine is using. The code tries pretty hard to get the right number, and it does a good job on getting the 'close' right, but it's another thing when it comes to calculating the last value of the day. After all, the greeks are meant to calculate on the 'close' - just the 'last', and that is the real problem.

The option pricer in the engine looks for the 'spot' - the price of the underlying that is used in the calculations, and it has a rather complex logical path based on the instrument type of the underlying, and what data is defined for it. Not hard to see, but sufficiently complex. In order to make it so that we include the 'close' in this logic might make it a lot worse before it gets better.

For example, we have the logic that an option will only be updated if it's a valid trading day (no need for weekends, holidays), and if it's during market hours for that instrument. The wrinkle here is that "during market hours" is updated only every couple of seconds at best, and what we really need is to look at the exchange timestamp on the message and see if it's something to include in the calculation set.

But we don't have that data in the right place in the code. It wasn't really designed for this, and there's where I blame myself. I allowed a junior guy who had been working with the legacy pricing server to lead the charge on this part of the code. As a consequence, it's a mess. It really is. We have objects with really questionable APIs, very bad separation of duties, etc. It's a mess.

But I have to live with it at this point, because so much is done, and the time required to fix it would not be allowed. That's a real shame. I'd like to fix it, and I'm not sure it's all that much time, but it's more than will be allowed, I'm sure. Even if it too no time, they'd oppose it on risk and rewards grounds.

So I have to figure out how to make this work with the tools at hand. Not as fun a job, but it's the job for today.

[12:35] UPDATE: well… I've got two little sections of code to work with. The first I've just put in is the check to make sure that - during market hours - the print must be a "qualified trade" from the exchange to update the 'last' price on the Instrument. The old code:

  if (!isnan(aPrint.getPriceAsInt()) &&
      ((mPrint.last.price != aPrint.getPriceAsInt()) ||
       (mPrint.last.size != aPrint.getSize()) ||
       (mPrint.last.exchange != aPrint.getExchange()))) {
    // update all the components of the print
    mPrint.last.price = aPrint.getPriceAsInt();
    mPrint.last.size = aPrint.getSize();
    mPrint.last.exchange = aPrint.getExchange();
    // ...and don't forget the timestamp
    mPrintTimestamp = now;
  }

became:

  bool  isMH = isMarketHours(aPrint.getCreationTimestamp());
  bool  isLPC = aPrint.isLastPriceChanged();
  if (!isnan(aPrint.getPriceAsInt()) && ((isMH && isLPC) || !isMH) &&
      ((mPrint.last.price != aPrint.getPriceAsInt()) ||
       (mPrint.last.size != aPrint.getSize()) ||
       (mPrint.last.exchange != aPrint.getExchange()))) {
    // update all the components of the print
    mPrint.last.price = aPrint.getPriceAsInt();
    mPrint.last.size = aPrint.getSize();
    mPrint.last.exchange = aPrint.getExchange();
    // ...and don't forget the timestamp
    mPrintTimestamp = now;
  }

The addition should work just as needed - if it's during the market hours, it needs to have the last price changed flag set. If not, take it anyway. Nice to have it be so surgical.

Google Chrome dev 19.0.1041.0 is Out

Wednesday, February 15th, 2012

This morning the Google Chrome Team is at it again, and released 19.0.1041.0 which is primarily a bug fixes release. It's impressive to see these updates coming so quickly, as it's only been a few days since the last significant release. They are certainly back at it, and moving things forward. Nice to see.

Still Working on Serialization Headaches

Tuesday, February 14th, 2012

High-Tech Greek Engine

Most of today has been spent working on little updates to the greek engine based on the results of user testing. Some of it has just been on explaining what the engine does and how to get it to do what you want it to do. This last part was really focused around one of the clients of the engine that's a neat little web page that's communicating to the engine directly through The Broker. This is really kind of neat in that the web clients now don't need a back end server to maintain their state - they maintain their own state, and make calls for data as they need it. Very much the spirit of AJAX, but not at all using XML or anything like that.

Anyway, the issue was how to get the right implied vols out of the model in the engine based on what needed to be passed in. It's not all that hard, but it's detail work, and it really helps to know the business domain reasonably well. The guy doing the web coding isn't really strong in that last category, but he'd trying. We had several meetings about what to do, and he wasn't really the best candidate to drive these meetings, but because it was his code the traders see, it was assumed that he was the guy.

So we had to clear all that up in several meetings.

Other than that, I've still been wrestling with the serialization of the data from the exchange feed recorders to the archive server. For some unknown reason, we are seemingly getting scrambled data, and it's amazing that it's just a problematic issue. Most of the serializations I've done are rock solid. It's just this one.

The size isn't horrendous. The contents are bad. It's a simple std::string containing binary data. Should be a slam dunk. But it's not so simple 100% of the time. And that's the issue. It's such a difficult problem to reproduce.

Late today I thought I'd try shifting from serializing a std::string to serializing a msg::ng::bytes_t array. This is a simple byte array that is implemented as a std::vector<uint8_t> - I just typedef it to make it a little easier to use.

What I'm hoping is that this makes the binary data a little more stable in the serialization and deserialization. We'll have to see tomorrow - I don't want to restart the feed recorders now to put in this new data.