What's it all about, Alfie?

I've spent a few hours today trying to find even more speed in my exchange decoder. It's the core part of the ticker plants as it's the first component in the chain, and the exchange datagrams come into this component, are converted into our own message format, and sending them downstream. The problem I'm seeing is that the time to get through this step is far too long, in my mind. I'm seeing times in the hundreds of milliseconds - and that's not right.

So I'm trying to find the problem. It's not easy because I can only do this during exchange hours, but even then, it's not obvious where the problem lies. I clearly need to do more work.

I'm concerned that it's in the OPRA decoding - that would be a tragic problem. Messing with that code could be really dangerous.

This afternoon I've been working very hard to convert all the prices and decimal numbers in my ticker plant codebase from float values to uint32_t values with a given getDecimalMultiplier() on each message. This came up in a meeting regarding some other group's use of the codebase, and they currently don't use floating point numbers - but rather an integer and a multiplier. OK... I can fix that, and so I did.

First thing was realizing that a uint32_t was sufficient as that would give me a 10,000 multiplier and values in excess of $400,000.00, when divided out. Good enough. Then I had to go into the code, replace all the values, add constructors and methods to take either the floating point number, and convert it to the proper integer, or take the integer and just use it.

The next thing was to look at the conversion/sampling functions on the exchange data. A lot of these take an integer mantissa and a divisor code and generate a float. What I needed to do was to alter these, or make similar functional versions, where they would take the same arguments and generate the correct integer representation of the value - offset by 10000 (my new offset). Again, not really hard, but it's detail work - making sure you get all the conversions done and don't loose any precision in the process.

Next, I created getters and setters for the messages that allowed the user to get the integer or floating point value at their choice. The scheme I used was to say that getPrice() got the decimal number and getPriceAsInt() got the biased integer. Pretty simple, and I don't think I'm going to have a lot of confusion here, which is very important.

Finally, with nothing but a few float values remaining - and the getters and setters using float arguments, I decided it was better to do a complete conversion to double and get rid of any float values in the processing. It's cleaner, more easily dealt with at the chip level, better scale and accuracy -- it's just better.

With this, I have everything stored as integers, with the multiplier available to the clients, and even decimal getters if they don't want to hassle with the conversions themselves. It's about as clean as I can imagine making it.

So I'm sitting here this morning thinking about all the emails I had to respond to this morning and wondering When can I pull out a can of Whoop-Ass? or just as effectively, When will someone pull out a can of Whoop-Ass? I mean really. Please. Now.

I'm still a bit of the "new guy" here, and I realize that there are a lot of things in every corporate culture that are really evolutionary in their entrenchment, and can't realistically be changed without major moves at the top. But I'm not think about that. I'm thinking about all the people that are fighting me - really fighting me, on the new ticker plant.

Sure, there are those that see it as an untested work, and worthy of suspicion. For those, I offer tests, and testing time for them to run their tests to see if it's working. I don't mind these people at all - in fact, I welcome them because they are the converts that will become my strongest allies. They will pick up the banner and run with it because they beat the crap out of it and proved to themselves that this is a good system, and it's now something they can treat as a provided service and not have to maintain themselves.

Then there are those that see their only value to the organization is the code they wrote a few months/years ago, and I'm talking about taking all that away from them. Maybe it's the team they have built up to help maintain the code. Mine would strip all that away from them as their team would no longer be needed to maintain this code. All these threats breed fear, and fear breeds anger. I see it, but I don't even pretend to understand it.

There are also the completely clueless, often fed information by the threatened, and that's a dangerous combo. I've had several conversations and untold emails from folks like this. They say things like "our needs are different", and "if you consider the efficiency, you'll see this is needed" - as if I've never even tried to understand their needs (have - for months), and that I'm completely unaware of the engineering issues involved (am - for years). It's best when it's all sugary-sweet in the tone you'd hear a parent of a young child use. Yeah... that's right, I'm your 4 year old writing this code and need a little help from Mommy and Daddy.

It's about professionalism, folks. Let's just all treat each other professionally. We don't have to agree on everything - we don't even have to agree on a lot of things. But we do have to agree on facts and numbers. If you show me test results, and I disagree with them, it's up to me to provide proof, typically in the form of better tests, that refute the data. Otherwise, I have to accept them. That's professionalism.

Likewise, if management (whoever that is) decides that we need to go in a certain direction, then that's what we all have to do. It's not a question of whether or not we like it... it's the job. Period. Done.

But it seems that things aren't quite this way. I'm sure the founders expect their instructions to be followed without delay, grief, crud, or delay. And they should. And I'm not totally sure where things start to break down, but by the time they are at my level, it's clear that something very bad has happened to the chain of command and the idea of professionalism. I'm not hypersensitive to it, but I do think I shouldn't have to spend months dealing with people fighting me on topics that I've been asked to do my partners in this firm.

I'm not saying it's my ideas. It's their ideas, I'm just the instrument of change. Still... these people are doing all they can to stop this implementation. Now we've moved and I'm sitting right in the middle of the snake pit. I understand why the managers did this - they want me to effect the maximum change I can, but they aren't thinking that I'm getting sick and tired of this.

They're just going to burn me out if things don't chance soon. Face it - how long are you interested in fighting the fight assigned you with no real support from those that assigned it to you? I'll do it for a while, and then I'll just say "Do whatever. Don't care." and I'm all done.

Guess I'm closer to that than I thought.

I originally designed my ticker plants to fit a specific client: the systems feeding the human traders. Eyeballs. There was no need to have everything up-to-date every millisecond - the human eye can't tell, and the systems don't update faster than a few times a second. It's just a waste. But what they do care about is that when they see the change, it's the latest data available. This means don't queue it up! You have to remember the order the ticks came in, but allow for updates to the data to replace the old with the new. This is commonly called conflation. It's a good thing for systems delivering data to humans.

But automated trading systems don't want this. They want every tick. They want it all as fast as possible. It's understandable - if you can make a machine able to see everything, then you have a much better chance of seeing opportunity and therefore making a profit. While I didn't design my ticker plants for these kinds of systems, several months ago, I was asked to make it work for these kinds of systems.

I've spent a lot of time trying to speed things up so that one system is capable of meeting the needs of both kinds of clients. It's been very difficult, and in a very real sense, what I've been doing is dumbing down my system to force the clients to handle every tick. If I could have done it, it would have been fantastic. But it really isn't possible. The compromises for one client are just too far from the compromises for the other.

So I finally had another little Ah! Ha! moment - Stop trying to make one size fit all. Elementary, but true, and an important understanding of really making something good for everyone.

If I made my ticker plants the way I started - for the 'slow' trading, and then had the 'fast' trading use an embedded ticker plant, then those that needed speed wouldn't even have to deal with a network hop. That's good. No serialization or deserialization. No worries about dropping packets from the server to the client. There are a lot of things that just "go away" when you decode and use the data in the same process.

I do this in my NBBO server - I have n exchange feeds all going into one NBBOEngine, and then sending it out to the clients. I don't take in the feed, process it, and then send it out - that'd take too long. I process the feed within the process space of the consuming application.

The resources to do this aren't horrible, two threads, less than a core and some memory. All this can be dealt with very easily by adding a box or two, if necessary. These boxes could be the "servers" you turned off because you no longer need them. In any case, it's a very solvable problem.

In the end, those that need conflation get it, and those that don't want it, get the data in-process as fast as possible. It's really the best of both worlds as it doesn't make compromises for one client or another.

I've had a very successful day testing ZeroMQ in my ticker plants with the updated parameters that had been hinted to me by a co-worker. It's not something I'd have thought to try, given that we're using OpenPGM - I thought the socket buffers were going to be controlled by OpenPGM, but I guess not.

In any case, if I create a socket and then set the send and receive buffers to 64MB and the peak speed to 500Mbps with a 100 msec recovery interval:

  // set the send and receive buffers to 64MB each
  static int64_t      __sndbuf = 67108864;
  static int64_t      __rcvbuf = 67108864;
  // have the maximum sending rate be 500Mbps
  static int64_t      __rate = 500000;
  // ...and the recovery interval 100 msec
  static int64_t      __recovery = 100;
 
  // create the socket...
  try {
    mSocket = new zmq::socket_t(*mContext, ZMQ_PUB);
    if (mSocket == NULL) {
      error = true;
      cLog.error("could not create the socket!");
    } else {
      // now let's set the parameters one by one...
      mSocket->setsockopt(ZMQ_SNDBUF, &__sndbuf, sizeof(__sndbuf));
      mSocket->setsockopt(ZMQ_RCVBUF, &__rcvbuf, sizeof(__rcvbuf));
      mSocket->setsockopt(ZMQ_RATE, &__rate, sizeof(__rate));
      mSocket->setsockopt(ZMQ_RECOVERY_IVL_MSEC, &__recovery,
                          sizeof(__recovery));
      // now let's connect to the right multicast group
      mSocket->connect(aURL.c_str());
    }
  } catch (zmq::error_t & e) {
    cLog.error("while creating the socket an exception was thrown!");
    if (mSocket != NULL) {
      delete mSocket;
      mSocket = NULL;
    }
  }

I've got a lot more testing to do, but these parameters really seem to help. Very nice.

The next step is to get the latest code from the GitHub git repo and try it. There are a ton of new features and lots of fixes which hopefully will clear up the last of the problems I'm seeing.

I've said it before to my kids, and people I've worked with time and again: What makes you think you get to make rules here? It's a classic attitude - someone does something that gets considerable praise, they enjoy that, and the attention it brings, they start to exercise a little more power over their world, and it succeeds. This continues for a while, unchecked, and in the end, the person has to be smacked down. This leads to hard feelings, etc. but in the end, if they can accept that they were at fault, then things can move forward.

I suppose I really can't blame the people, they see what they want, they grab it, don't see any consequences, and grab again. Power corrupts, and absolute power corrupts absolutely. They start to think of themselves as untouchable and begin to act that way. I suppose the only thing that keeps a person in check in these situations is some internal regulator that says Hey, they may think you're great, but I know the difference - don't get too cocky! But many people don't have that. Maybe even most don't.

But I do. I guess that's age. Maybe it's a belief in something larger than myself that's looking over us all and just laughing at us when we think we're "all that". I don't know, but I do know that it's hard to work with these people.

As I write this I've realized that in additional to not making the rules here, I also don't enforce the rules. I'm sure someone is sitting up there laughing at my frustration at trying to get through to these folks. He's probably laughing harder at me than them.

I need to just see people as he sees them - funny. Liza made a cross-stitch of a fantastic fortune cookie I received:

Life is a tradegdy for those that feel.
And a comedy for those that think.

Think more, Bob. It's all pretty funny if you can get a little distance on it.

This morning I swapped out the UDP transport system I'd written over the last few days for the ZeroMQ-based one that I'd been using for months prior to that. I have the feeling that because of the bugs I fixed in the rest of the codebase, it's very likely that the ZeroMQ transport system will work just fine. Additionally, I've been able to get the master of ZeroMQ on GitHub to compile and work with OpenPGM, so it's possible to update all our installs of ZeroMQ as well.

We'll see what happens today. If we can run all day at the same levels as we did yesterday with the UDP transport, then I can stop trying to put reliability into my UDP transport and just go with ZeroMQ. That would be nice. One less thing to do.

Today has been a reasonably successful day as long as we stretch the meaning of 'today'. Everything has been running well for the next round of tests, but I was still trying to get just a little more speed out of the UDP broadcaster, or maybe the NBBO engine. When all you do is look at the logs of a process you get thinking that maybe you can cut that 200 msec to 100 msec, and if you can do that then you can speed everything up. It's a vicious cycle.

So I spent a lot of the day working up different approaches and testing them against what I've already got in the test system. In every case, what I had was faster. Darn. But maybe it's good news in that I'd done a good job already, so I didn't have any real improvement to do. Like I said, it depends on how to define 'today'.

I have a successful system, but it was already that way. I just proved that I couldn't make it any better. Well... that's something.

Well... it's been a while getting here, but I honestly think that this morning was a break-through for my ticker plants. The maximum delay was considerably less than in the past (thanks to finding the silly second consumer thread) and even far better than I remembered from the other group's tests. It's a big day for the good guys, and it's been a long-time in coming.

To be sure, a lot of the problems I came across were because of the changes in the target market. But it took me quite a while to get all the details worked out. But Holy Cow! It's a great feeling.

Well... with the last threading bug figured out for my new UDP transport, I have to wonder if ZeroMQ is really just fine. There are several reasons to use it - OK, one: reliability, but that's a biggie. Still, it might be nice to try it again. The UDP transport, if I can add in the necessary reliability, will be better, for sure, because it's less "black box" and more transparent to all involved. It's certainly already suppressed a few nagging naysayers because they feel that PGM is not reliable at all, and they'd rather have straight UDP. Giving that to them allows them to feel like they have a victory in this "battle", and so adoption will go a little easier.

In short: Politics.

Every time I come to a crossroads like this where a technical decision is given a back-seat to a political decision, I've always felt the wrong decision was made. Always. I'm hoping to give them their UDP transport, and make it technically better than what they have, and so not have to come to this decision.

Alternatively, since they have been so negative in this adoption, it would be nice to simply do all the work myself and then point out to management how we can "re-task" these "resources" to "different efforts". Problem with not being part of the solution is that after the solution is done, you've proven yourself to be unnecessary.

Right out of a job.

Anyway... for now, I have one good transport, and I think I have two. But I'll have to run with these changes for a bit to convince people it's as good as what they have. Gotta prove they aren't needed any more. Sad but true.