Threading Issues with Boost ASIO async_write()
This morning it's Developer Day, and I'm fighting a problem that really had me stumped for a while. When I have two independent requests hit my service, if they are close enough in time, they can cause a real problem in the service. I seem to get the requests just fine. I'm able to generate the results for each request just fine, but when I try to send back the generated responses to the client, it gets the service and the client into an unstable state.
Not good.
When I try to reproduce the test using my C++ client, I can't seem to get the requests close enough in time to trigger the problem. Or maybe I have a better client-side socket driver. Hard to say. But someone generated a test case in Python, and that does the trick. The client doesn't appear to ever get the response, but I know I sent it. Not a fun place to be when people are supposed to be hitting your stuff for fun and enjoyment.
I got the idea that it was in the boost ASIO sending because the generation was right, the C++ client was right, but the Python wasn't. The original code looked like this:
void MMDServiceHandler::asyncSendMessage( const std::string & aString ) { bool error = false; // first, make sure we have something to do if (!error && ((mSocket == NULL) || !mSocket->is_open())) { error = true; cLog.warn("[asyncSendMessage] no socket to send on! Bad news"); } /** * Now let's put the preamble and message into a buffer so that the * MMD knows what to do with it... */ if (!error) { // let's make a few pointers for the payload boost::shared_ptr<Preamble> hdr(new Preamble(aString)); boost::shared_ptr<std::string> body(new std::string(aString)); // now let's make a package of the header and buffer std::vector<boost::asio::const_buffer> pkg; pkg.push_back(boost::asio::buffer(hdr->body(), hdr->size()); pkg.push_back(boost::asio::buffer(*body)); // finally, send it out asynchronously boost::asio::async_write(*mSocket, pkg, boost::bind(&MMDServiceHandler::asyncMessageSent, this, hdr, body, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred)); } }
My theory this morning was that by placing two items in the boost package, I was allowing boost to think that they could be sent separately. With two such writes done in quick succession, it's possible to think that the four components got interleaved and so it messed up the receiver, and then the sender as it was waiting for something back from the client, and it was never going to come.
What I decided to go with was a change to a single buffer:
void MMDServiceHandler::asyncSendMessage( const std::string & aString ) { bool error = false; // first, make sure we have something to do if (!error && ((mSocket == NULL) || !mSocket->is_open())) { error = true; cLog.warn("[asyncSendMessage] no socket to send on! Bad news"); } /** * Now let's put the preamble and message into a buffer so that the * MMD knows what to do with it... */ if (!error) { // let's make a header and a pointer for the body Preamble hdr(aString); boost::shared_ptr<std::vector<char> > body( new std::vector<char>(hdr.size() + aString.size())); // now let's put everything into one contiguous piece... memcpy(&((*body)[0]), hdr.body(), hdr.size()); memcpy(&((*body)[0]) + hdr.size(), aString.data(), aString.size()); // finally, send it out asynchronously boost::asio::async_write(*mSocket, boost::asio::buffer(*body), boost::bind(&MMDServiceHandler::asyncMessageSent, this, body, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred)); } }
By placing it all into one std::vector, using jsut two memcpy() calls, I was thinking I was even making it a little faster. The data was still in a stable shared pointer that was going to have a reference until the asyncMessageSent() method was called, and then it'd be dropped. So I was thinking I had this bad boy licked.
Yeah... well... not so much.
So the next thing I went to was switching from using async_write() to just using write(). Thankfully, I already had a syncSendMessage() method for just such an occasion:
void MMDServiceHandler::syncSendMessage( const std::string & aString ) { bool error = false; // first, make sure we have something to do if (!error && ((mSocket == NULL) || !mSocket->is_open())) { error = true; cLog.warn("[syncSendMessage] no socket to send on! Bad news"); } /** * Now let's put the preamble and message into a buffer so that the * MMD knows what to do with it... */ if (!error) { // let's make a header and a pointer for the body Preamble hdr(aString); std::vector<char> body(hdr.size() + aString.size()); // now let's put everything into one contiguous piece... memcpy(&body[0], hdr.body(), hdr.size()); memcpy(&body[0] + hdr.size(), aString.data(), aString.size()); // finally, send it out synchronously boost::system::error_code err; boost::asio::write(*mSocket, boost::asio::buffer(body), boost::asio::transfer_all(), err); if (err) { error = true; cLog.warn("[syncSendMessage] unable to send the message!"); } } return !error; }
When I switched to the syncSendMessage(), everything worked out. I didn't like that I had to synchronously write out the data, so I put the entire send method in a simple thread that does the send and then dies. Sure, I could have put it in a pool, but this is good enough for today, and we'll wait and see what the real limitations are. But for now, it's been a really hard, but good, morning and I'm ready to just sit back and help the other guys with their Developer Days projects.