Archive for the ‘Coding’ Category

Putting it All Together for an NBBO Ticker Plant

Friday, November 5th, 2010

Well... it's been a heck of a week, but I'm very happy that I have an NBBO Ticker Plant in the can - ready for testing on Monday. It took a long three days, and one of those days was devoted almost entirely to the persistence of the exchange data and message cache. The testing for that proved very helpful as I caught a few nasty copy-paste bugs in the data elements.

What I'm hoping is that with the NBBO Plant being fed from more than 25 exchange feeds, it'll still be able to keep up with the flow - because none of these feeds are option feeds. We'll have to wait for Monday to see, but I'm optimistic.

If not, I'll have to figure out how to speed things up, or possibly put in conflation queues between the exchange feeds and the NBBO Engine.

Proper Unit Testing is Valuable – But Painful

Friday, November 5th, 2010

GeneralDev.jpg

One of the things I really think is important in large systems work is proper unit testing. Now I'm not talking about the "Test First" idea, I'm talking about building some code - a class in C++ or Java, and it contains something other than mindless data containers, and those methods needing testing. If it's a cache, then throwing things in, counting them, and getting them out again. That kind of testing. Stuff where you need to be certain that the code is working before you go on.

The problem with a lot of this testing is the same thing you see in testing ICs - test vectors often require additional instrumentation on the class in order to test things properly. For example, I was testing parts of my NBBO engine where I needed to save it to a persistence system, and read it back out, and I needed to make sure it was correct. This means that I have to put operator==() on everything - even the data structures, because I can't be sure I'm not missing a component in the larger picture.

Then in order to test the larger components, I have to really add a clone() method to all the objects in the data structure so that I can be assured that once these elemental objects are equal, placing them in the larger data structure is also equivalent.

Now the code is simple:

  LeafNode & LeafNode::clone()
  {
    return new LeafNode(*this);
  }

because I always have the copy constructor on all my objects (to keep the compiler from creating one for me with behaviors I did not intend). So the code is really just a few lines - but it's this added instrumentation that makes efficient testing possible that makes the process all the more tiresome.

However, It finds bugs. It benchmarks performance. It's useful. It's just not very fun. Painful, in fact, at times.

But it needs to be done.

Stepping into the Flash-Free World

Friday, November 5th, 2010

I was reading this piece by John G on Daring Fireball, and it got me to thinking - it probably was a little more honest to disable Flash completely. I mean, it's still on my box, just not in my Safari or Firefox plugins. And with Google Chrome right on my desktop, I can see Flash if I really need to.

I think I'm going to like it a lot more... I've certainly given it a lot of thought.

Google Chrome dev 9.0.572.0 is Out

Friday, November 5th, 2010

This morning I saw that Google Chrome dev 9.0.572.0 was out - quite a step for the Googlers, but it's only release note is that it includes a new version of Flash. Hmmm... how nice. It's not that I'm against Flash as a concept, it's a language - a bad one, and it's a bad build environment, but it's also that it's just not really all that useful in terms of what can be done today in HTML, JavaScript, CSS and the like. It's just a crutch. Anyway... that's the update: Flash. Yippee.

Creating a National Best Bid/Offer Engine (cont.)

Thursday, November 4th, 2010

GeneralDev.jpg

Well... it took me a little bit, and I had to re-arrange the code slightly, but in the end, I have the logic I need for the NBBO engine that works when you remove the one NBBO component. It's a little more complex, but it's still very clean.

The previous code was slick, and this is nearly as slick, but it's a little more complex because of the different cases I had to deal with in the change of the 'best values'. Still... it's straight-through code with only the one lock to control it's updating, and I think it's going to be plenty fast enough. But we'll have to see how long it takes in real loading conditions.

  1. if ((bidExch != '\0') && (bidPrice > 0.0) && (bidSize > 0)) {
  2. // get the exchange as an index into the arrays
  3. uint8_t idx = bidExch - 'A';
  4. if (bidPrice > myExchData->nbboBidPrice) {
  5. // with a new leading bid, we are all there is to consider
  6. myExchData->nbboBidPrice = bidPrice;
  7. myExchData->nbboBidSize = bidSize;
  8. newNBBO = true;
  9. } else if (bidPrice == myExchData->nbboBidPrice) {
  10. // if we had something in the old NBBO, remove it now
  11. if (myExchData->bidPrice[idx] == myExchData->nbboBidPrice) {
  12. myExchData->nbboBidSize -= myExchData->bidSize[idx];
  13. }
  14. // ...and we know we need to add in our contribution now
  15. myExchData->nbboBidSize += bidSize;
  16. newNBBO = ((myExchData->bidPrice[idx] != bidPrice) ||
  17. (myExchData->bidSize[idx] != bidSize));
  18. } else if (myExchData->bidPrice[idx] == myExchData->nbboBidPrice) {
  19. // we had something in the old NBBO, remove it now
  20. myExchData->nbboBidSize -= myExchData->bidSize[idx];
  21. newNBBO = (myExchData->bidSize[idx] > 0);
  22. // see if this was the only one in the NBBO
  23. if (myExchData->bboBidSize == 0) {
  24. // reset the bid part of the NBBO
  25. newBidPrice = 0.0;
  26. newBidSize = 0;
  27. // save what we have now for the scan
  28. myExchData->bidPrice[idx] = bidPrice;
  29. myExchData->bidSize[idx] = bidSize;
  30. // ...and scan all the data (skipping this guy)
  31. for (uint8_t e = 0; e < 26; ++e) {
  32. if (myExchData->bidPrice[e] > newBidPrice) {
  33. newBidPrice = myExchData->bidPrice[e];
  34. newBidSize = myExchData->bidSize[e];
  35. } else if (myExchData->bidPrice[e] == newBidPrice) {
  36. newBidSize += myExchData->bidSize[e];
  37. }
  38. }
  39. // make sure we have something
  40. if (newBidSize > 0) {
  41. myExchData->nbboBidPrice = newBidPrice;
  42. myExchData->nbboBidSize = newBidSize;
  43. // we KNOW this is a new one
  44. newNBBO = true;
  45. }
  46. }
  47. }
  48. // no matter what, save the bid for the next time
  49. myExchData->bidPrice[idx] = bidPrice;
  50. myExchData->bidSize[idx] = bidSize;
  51. }

There are a few neat things in this guy. First off, we look for the obvious winners - if we have a brand new best bid, or an addition to the size on the existing best bid. We needed to be careful when looking at adding the value in (line 11), and especially careful about sending out a new NBBO message out (line 16).

But the really new part is the scanning of the existing exchange data (line 31). This is done only when there was a single component to the best bid, and he's just taken himself out of the mix. For those cases, we have no choice but to start all over and see what we can come up with. The wrinkle was that we need to add back in our current data so it too was included in the scan. It's possible, after all, that it's still part of the best bid, just ad a different price. And maybe with some friends.

All told, I really liked this code. It's fun, and the test frame for exercising it was a blast to write. I have the ability to throw any number of sequences of ticks to the engine and test each subsequent value. It's nice. Good, fun, work.

Hammering Away at the NBBO Persistence

Wednesday, November 3rd, 2010

GeneralDev.jpg

It's been a hard day. I've been working on my NBBO Engine all day, and the vast majority of it on the persistence of the engine to the Broker's configuration service. It's really a very nice way of doing things - give that I already had the Broker connectivity down. You can simply send variant values to the config service, and it'll store them under a key. Simple stuff - if you can get everything organized properly.

The problem today was that the trie and it's structure. I needed to be able to pack up all the individual components of the trie, but I also needed to be able to pack up the connectivity structure. This means that I needed to be able to put a little more metadata into the packing than I had originally planned, but it all worked out - I think.

I need to get this code to compile - I'm sure I forgot a dozen namespaces here or there, and then I need to build a test frame that will take messages and record the output NBBO messages. It's not very hard, it's just going to take time.

It was a hard day... but I'm glad I pushed through it.

iTerm2 Updated to Alpha 12

Wednesday, November 3rd, 2010

iTerm2

I checked this morning and the iTerm fork called iTerm2 released a new update and it was chock full of a lot of updates and bug fixes:

Alpha 12

Featurepalooza! And bug fixes, too.

New features:

  • Instant Replay: This feature makes iTerm2 behave like TiVo. You can press cmd-opt-b to step back in time and cmd-opt-f to step forward in time. If something is erased from the screen, it's no longer lost forever. This feature can be turned off in Preferences as it does take a bit of extra memory and has a small performance impact (less than 1%).
  • Paste History: When you copy or paste text in iTerm2, it is now saved in the paste history. You can pop up a window at the cursor with cmd-shift-h which shows the last 20 strings in the clipboard. The list is searchable Quicksilver-style.
  • Mouseless Selection: Do you often copy-paste text within the same window? Now you can do it without using the mouse. Open the findbar with cmd-f and search for the text you wish to copy. When part of the text is matched, use the tab key to expand the selection to the right by a full word and shift-tab to expand it left to the previous word boundary. It's automatically copied to the clipboard, or you can use opt-Enter to paste the selection immediately.

Enhancements:

  • Findbar appearance improved.
  • Add undo/redo to edit menu.
  • Add zoom shortcut for cmd-opt-=.
  • Read larger chunks from the network for better performance.
  • Idle CPU usage reduced.
  • Refresh rate adjustment removed, dynamic refresh rate added.
  • Input Method Editor wraps at end of line, scrolls window if necessary.
  • IPV6 support for Bonjour.
  • Bug fixes:

    • Memory usage growth when Bonjour enabled (hopefully, 43).
    • Toggling Bonjour takes effect immediately (89)
    • Make resize look better (98)
    • Restore colors on exit of Vim (99)
    • Make bookmark window resize properly (114)
    • Select name field when copying/creating/editing bookmark (127, 128, 181)
    • Scroll by selection in fullscreen window (133)
    • IME improvements (163)
    • URLs with i18n chars can be selected now (174)
    • Tab lables updated when toggling compact tabs (196)
    • Better multi-monitor support (201)
    • Set localization vars better (204)
    • Fix return to default size (223)
    • Improve i-bar cursor appearance (235)
    • Numeric shortcuts displayed properly (241)
    • Prevent 100% transparency because it doesn't focus (250)
    • Fix bugs with AquaSKK (263)
    • Fix bug where page scrolling sometimes broke.
    • Fix creeping window size when toggling fullscreen.

Quite an impressive list. I'm excited to see how it plays out.

Google Chrome dev 9.0.570.0 is Out

Wednesday, November 3rd, 2010

GoogleChrome.jpg

Seems this morning that the developers at Google released Google Chrome dev 9.0.570.0 and with it a significant version number change. Maybe this bodes well for the browser. Looking at the release notes, however, leaves me wondering:

Mac

  • Make sure the dock icon is updated after closing an incognito window with an in-progress download (Issue 48391)

and then there are several "Known issues" with the build:

Known Issues

  • REGRESSION: Windows media player for Firefox doesn't load - Issue 61603
  • Regression:accelerated compositing slows down the whole machine - Issue 61520
  • google.com/wave : "Page Unresponsive" dailog box appears - Issue 61533
  • myspace.com : Cannot enter a character in Comments field - Issue 61513

which, to me, sounds like the major version number is really just a sham, and the third number - 570, is the one to look at. They aren't using the major numbers to indicate anything other than what they consider to be dev versus beta versus release.

Kind of disappointing...

Creating a National Best Bid/Offer Engine

Tuesday, November 2nd, 2010

Today I started, then stopped. Then restarted work on a National Best Bid/Offer engine for my ticker plant system. Basically, the rules are pretty simple: you keep track of the ticks for an instrument by exchange - keeping the latest bid/ask data for each exchange. What you might end up with after a few ticks is something like this:

Exchange Bid Price Bid Size Ask Price Ask Size
A 12.50 25 14.00 20
C 12.25 20 14.00 25
D 12.50 20 13.75 20

The rules say that you look at all the bid prices - pick the largest one, and then sum all the bid sizes for those exchanges where the price is the "best bid". In this case, the best bid is 12.50 and the sizes are (25 + 20) = 45. For the best ask (offer), you look at the minimum asking price, and sum up the sizes that match that value. In this case, the price is 13.75 and the size is a simple 20.

So, there are a lot of ways to accomplish this. One is just to have a cache of all the ticks - organized by instrument, by exchange, and then when a new one comes in you toss out the old, recompute the new NBBO, and then if something has changed from the last time you calculated the NBBO, you send out a new message to the downstream clients. Pretty simple.

The problem with this is that while it's easy, it's a guaranteed two-pass system because you need to first find the best price (of either kind), and then sum all those sizes with that price. Not ideal. Also, there's the possibility that a Quote message from one of the exchange services will have a bid from one exchange and an offer from another. So it's not rally easy to keep the entire message - and I don't really need it. I only really need the data.

So what if I just kept the data?

  struct ExchangeData {
    // bids and asks are separate and organized by exchange
    float      bidPrice[26];
    uint16_t   bidSize[26];
    float      askPrice[26];
    uint16_t   askSize[26];
    // ...and this is the NBBO that's generated from it
    float      nbboBidPrice;
    uint16_t   nbboBidSize;
    float      nbboAskPrice;
    uint16_t   nbboAskSize;
    // ...and a lock to make sure it's all changed atomically
    boost::detail::spinlock   dataMutex;
  };
  typedef struct ExchangeData exch_data_t;

If I have this structure - one for each instrument, in some simple data structure (like, say, a trie), then I can forego the message retention, and just retain the data from the message and update the data in the structure.

But wait, there's more.

If I'm careful about what I do, I can update the NBBO values without having to make a two-pass update. What I can realize is that there already exists, for each tick, an NBBO, and if I just update that by subtracting out the old data, and adding in the new, then the values will always stay up to date. No need to do the two-pass calculation.

The code for just the bid looks something like this:

  if ((bidExch != '\0') && (bidPrice > 0.0) && (bidSize > 0)) {
    // get the exchange as an index into the arrays
    uint8_t    idx = bidExch - 'A';
    // see if we need to remove the old bid from the NBBO
    if ((myExchData->bidPrice[idx] == myExchData->nbboBidPrice) {
      myExchData->nbboBidSize -= myExchData->bidSize[idx];
      newNBBO = (myExchData->bidSize[idx] > 0);
    }
    // see if we have a *new* NBBO bid, or maybe just an update
    if (bidPrice > myExchaData->nbboBidPrice) {
      myExchData->nbboBidPrice = bidPrice;
      myExchData->nbboBidSize = bidSize;
      newNBBO = true;
    } else if (bidPrice == myExchaData->nbboBidPrice) {
      myExchData->nbboBidSize += bidSize;
      newNBBO = true;
    }
  }

and with this code, I should be able to maintain a decent NBBO engine with very little latency through the system. I still need to write a few things and test it all out, but it's a really promising start, and I'm excited about the possibilities.

Sweet idea.

UPDATE: This logic isn't going to work. What if I fall out of the NBBO? This code will remove the contribution, but if I'm the only one, then we're left with a bad price and a zero size. I need to fix this. But I still think it's salvageable. The only case when we will have to scan is when there's only one exchange in the BBO, and that one is out. Then, we have to find the next best, and that's likely a scan.

Gearing Up for Full-Up Tests

Monday, November 1st, 2010

Today I was working on the Enrichment Server part of the Ticker Plant system and was making pretty good headway when the Chief Architect came up to me and asked me what it would take to get a complete, full-up, system going. "Well... I'm pretty close... I'd say a few days." I thought. "Good! Let's try for the end of the week." Sounds OK to me.

There were a few things I needed to get back into the source code for the tests. We'd done a significant change to the messages - ripping out the old 64-bit security ID in favor of the std::string security key, and the group that was going to use this needed to have the security ID for trading reasons. So I had to add that back in, but only in the Client code as it makes no sense to slow down the upstream systems for the few that need this feature.

So I started digging into my git repo to find the check-in where I took it out. I have to say, for a "universal" client, gitk is not bad at all. It's pretty decent for being platform neutral. I was easily able to find the check-in and then look at the diffs and see what I needed to add back in.

It took a little bit, but it was not really hard. Just a little time.

Unfortunately, the Broker and it's infrastructure is not up, so I can't test. But when I can, it'll be nice to make sure all this works. Then I just need to find memory for my box as it's got far too little to be effective for a complete SIAC feed... but that's a task for the management types who control the money.