I'm back from a nice week off and it's nice to get back into the swing of things. I'll admit it was a little tough this morning, but it really worked out pretty well. Now that I've caught up with email, and reading, it's nice to be back in the full swing of things.
Vacation is nice, but it was also a lot of work with the house. It's nice to be creative once again. Ahhh...
I'm to the point that I believe all my Broker re-write issues are solved. I can use no pooling on the sockets, and it works, and I can use pooling and it should work just fine. The guys working on the Java side of things aren't having as good a time of this as I am. Don't get me wrong, they're sharp guys, and I believe they understand the Java classes they are using, but the complexity of some Java libraries is really quite astounding.
Case in point: J2EE. It couldn't be harder to get something going on J2EE. Tomcat is hard enough - there should be something like JSPs or PHP that will allow you to place the java file in any directory and have things "just work", but I understand the "Java way", and while it's not perfect, Tomcat is awfully nice, and very capable - you just have to work within it's paradigm.
But this work we're doing on the sockets... Holy Cow! I"m glad I'm not doing the Java side. The C++ side has had it's problems, but still, it's very easy to see what's going on in the code, it's just a question of what's happening in the libraries. It's not a question of smarts, it's really a question of familiarity with the libraries. If you're new to the way it does things, as I was with boost asio in the beginning, it's a pain. But once you really know how to use the libraries, and how the library wants you to formulate a solution, then things get a lot better a lot faster.
I just feel for the guys doing the work on the Java side. At least one of them is new to this library, and this is a real trial by fire. The complexity is staggering to him, and it's clear to me that a really well designed C++ library can make a far simpler solution than a lot of the Java library/frameworks I've seen out there.
In the midst of working on the changes necessitated by the Broker's re-write, I found myself in a very nasty little problem. I am trying to do things quickly, and my test cases are often far worse than any real-world use is going to be, but they have served me well, and they were pointing out a problem I was having this morning.
If I created new updater instances for each request, and deleted them after they were no longer needed, I ended up with a very fast create-use-delete lifecycle. This lead to a segmentation fault in boost's io_service - specifically, in it's run() method. The core dumps were of little to no help whatsoever, and I was left trying to diagnose the problem from my end.
If I didn't delete them right away, but threw them into a pool, and still created new ones, only clearing out the pool at the end of the application, then everything was fine. It seemed like it was just the short lifecycle connections that was the problem. Very nasty.
The seg faults weren't on anything related to boost asio, either. They were on the line right after the context switch after the closing of the socket connection. I spent hours debugging the code to find that guy.
I came to the conclusion that there was something in the io_service that wasn't getting a "chance" to handle the socket closing before I deleted it. So I changed my code ever so slightly. Originally I had:
if (si->second.pool.size() >= MAX_UPDATERS_IN_POOL) { cLog.debug("[recycle] the pool is full - deleting the updater"); delete anUpdater; }
to:
if (si->second.pool.size() >= MAX_UPDATERS_IN_POOL) { cLog.debug("[recycle] the pool is full - deleting the updater"); anUpdater->mChannelID.clear(); anUpdater->disconnect(); // we need to let the io_service have a go at it boost::system::error_code err; mIOService.poll_one(err); // ...and now we can successfully delete the updater delete anUpdater; }
The difference was stunning. No more crashes, and the code was rock solid every time I ran it. Amazing. I'm going to have to remember this. It's like a little context switch for the io_service so it can detect the close of the socket before it's deleted.
Several things to finish on the re-write, but it's getting close now. Nice.
As I was doing a little testing this morning I realized that the Broker wasn't working like it should. When I pinged a few folks, it was clear that there had been several changes I hadn't been told about and so I had a very abrupt detour to update all my Broker-facing code to make sure it was all working.
Some things weren't too bad - the codec's code for a map went from 'M' to 'm'... the Dates are now passed as microseconds since epoch... simple things. But then I quickly found that there were some that weren't simple at all. Not in the least.
The protocol for service registration changed. The interaction with the registration service itself changed. The recycling of open sockets wasn't working - in short, it was a massive change to the codebase.
I've spent the bulk of the day getting things converted. It's touched a lot of the code, but I think I'm getting very close. There are still a few things I need to hammer out, and I'm hoping to get to them in the morning. The prickly one is that the 'close channel' message - that which indicates that the socket is clear to be recycled, can, if the conditions are right, close the actual socket. When that occurs, it's happening too fast, and the object is getting dumped before it's work is done.
I need to figure out how to fix this so the events happen in the right order.
This morning I'm continuing to work on the functions for my lisp-like parser. What I'm working on now is really the functions themselves and not how they fit into the other classes of the parser to compute values. It's a little slow going at first because I want to duplicate as much functionality as possible from the Java-version, and as you'd expect, there are absolutely no comments in the Java code. So I'm ending up chatting to one of the developers on the project and he's giving me the details on the functions.
Nothing hard, but it's slow going. I'm hoping once I get a few done, it'll go a lot faster.
Heh... that's optimism for you...
I've been in a lot of meetings all afternoon, but in the end, the good outweighed the bad. These other groups had their own biases, of course, and no one wanted to "just give in" and use the stuff I've been working on, but that's really understandable. After all, a few lifetimes ago, it wasn't until after my ticker plant was proven for the most demanding client (my own risk engine) that other people started to see the value in it. SO it goes... I'm going to have to prove it, and I understand that.
Still... they had several good points:
Many of these aren't too bad to add, and it's reasonable to get them out quickly, but some of the others are a little trickier, and it might be better to think of the real solution to the problem as a completely different kind of answer. For example, what about having a message with all the exchange prices and sizes for a given instrument. Then, when it's shipped out, the user gets all the prices - at once. It's got possibilities.
It's late and I need to catch a train, but it's a lot to do in the coming weeks.
This morning I've been working more on the lisp-like parser for embedding in my projects, and it's coming along nicely. I realized that I needed to make my collections use pointers if I wanted to really use polymorphism, which just makes the housekeeping a little more, but other than that, it's fine. With that, I'm getting the right messages dispatched to the right instances, and things are moving right along.
I have been adding the function class to the code this morning, trying to get the evaluation of functions going. It's just a lot of little details and lots of writing. The base classes aren't bad, but every time you start another, you're looking at a good bunch of typing. I just need to get the first version hammered out and then I can look and see if I need to make it faster.
The rest of the day is in meetings, but it's nice to get it all moving forward this morning.
One of the many potential next projects for me might be a greeks server - like I've worked on in the past. The difference here is that it's targeted to me a lot more flexible than the old MarketMash system - this one is going to allow for the clients to give it "mutations" to the center state so their numbers can be as good, or bad, as they want. I've fiddled with a lot of different schemes for specifying this kind of mutation - including muParser. But those don't seem right in this context.
The choice for this seems to be a lisp-like language that is in use in some other projects in The Shop, and it would be a natural fit to have a C++ version of this for any C++-based projects. Right now, the only version is a Java-based one, and while that's nice, it's also heavily customized to the application it sits in. What I'm looking for is a simple parser that can have variables, constants, functions all added to it - augmenting a nice set of default capabilities so the user can make of it what they want.
The language isn't really hard. The expressions are really pretty simple:
(set boo 5) (+ boo 10) (* 12 5 3) (avg (min boo 5) (max boo 5))
stuff like that. Each expression has a function as the first argument, and then a series of items - which can be values or other expressions. I've just decided to make the Item a base class, subclass Value and Expression, and then start the coding.
Today was all the parsing of the input text, and it's pretty nicely compact. I'm pleased with the way it's looking. Tomorrow is the functions, etc.
Today was heavy with meetings, but I was able to get some nice work done on my ticker plant - the Java guys were interested in using my security name to 128-bit int converter as it was non-lossy and bi-directional. There are times they want to use a number but not loose the ability to pull out different aspects of the instrument. What I went with was a subclass of my UUID where I added in a lot of nice helper methods:
tInstType getType() const; bool isStock() const; bool isOption() const; std::string getUnderlying() const; uint16_t getExpiration() const; float getStrike() const; bool isPut() const; bool isCall() const;
where I've dropped the comments for the obvious nature of these methods. The point is that we aren't going to decode the entire 128-bit stream to get the one bit that's the Put/Call flag, so these guys allow you to get the one piece of data you want without having to convert it all into a string and then parse that.
I got this all in, and the tests were just fine. I even had the operator=(std::string) written so you could do something like:
msg::ng::secID_t id = "O:GOOG:110214:74550:C";
and it'd parse the string into the 128-bit number and be ready to go. Should save a lot of time.
Finally, I added this as a possible component of the variant class so that it could be put int he codec for the messages to the Broker. This will really help move things around as we'll be able to handle getting this as a data type in anything coming from, or sending to, the Broker. Very nice.
But then there were more meetings, and some were really draining. Still... it was nice to have a little bit of work under my belt for the day. Made it feel like it wasn't totally spent in meetings.
Today I was thinking about one of the problems I was having with all these ticker plants and finding the data I needed amongst all of them. It could be a nasty little task to keep them all straight. Certainly, it was going to be a problem for the support crew. And then it hit me - what if I used the channel chats as a "broadcast" mechanism and then the "correct" ticker plant could privately answer me. If there happened to be more than one that had something for me, they would do it privately. Wow... excellent idea! I just had to make sure it was going to work.
The first thing was to realize that the only difference in the IRC message was that the "source" changes from the user (drbob) to the channel (#Hoedown). The source at the start is the same, so we can still pick off the sender, we can now just see that it wasn't a private message, but one to a channel. Sweet.
Now let's see how to put this into our IRC interface easily.
The first thing is to expand the IncomingMessage struct to include the channel. If the message is private, then the channel is empty, and if it's a channel message, it'll be filled in. Simple. Then we can add the lines in the async message reader of the protocol to check the message to see if it's any of the channels we have JOINed, and if so, then slap that into the channel part of the IncomingMessage and have it "handled" on another thread.
In the handler we just have to see if there's a channel defined in the message, and if so, we can then call onChannelMessage() otherwise we'll call the old onMessage() for the private messages. The reasoning here was just to make it clear what kind of messages you are responding to. It doesn't have to be like this, but to me, it makes it very clear what you expect, and if you don't want to mess with channel messages, then simply don't implement the onChannelMessage() method and it'll be fine.
All this coding took me all of about 30 mins, and then I started testing it with a simple command - who has AAPL?
Sweet. Worked like a champ.
Then it was a simple process of migrating a few of the private chat messages to the "public" messages with private replies. Instead of errors, I just had it return nothing. That way, if the ticker plant doesn't have anything useful to add, it's simply silent. This makes a lot of sense for these types of chats.
In the end, I have a simple way to ask the System as a whole a question and have a private response from the one guy that can answer my question. I love this interaction. Very nice.