DKit - C++ Library of Atomic and Lockless Data Structures
Introduction
In all the work we've done over the years, nothing has been quite as liberating as thread-safe containers. They are an amazing experience for a developer because they allow the multi-threaded programmer to not have to worry about locks, lock contention, deadlocks, dirty reads, and a whole host of other issues that for years have made writing multi-threaded code difficult. These containers and objects free the user from having to worry about locking because they use the compare-and-swap (CAS) operations in GCC to ensure that the data accessed is done properly, and securely.
Library Dependencies
The focus of this library is not to be a general threading library. Nor is it to attempt to write code that's already written. As such, we're going to be using boost for several components. This will include things only used in the test applications - for example, threading code to test the multi-threaded queues. Since boost is available on virtually all platforms, and even included in the next C++ standard, this should not be a significant burden to the user.
There are even instructions to build it on Mac OS X, if you don't wish to use the Homebrew version that's freely available.
Base Container Template Classes
In order to make the conversion of the code as simple as possible, we have made a base class for each type of container in this library. These base classes are most easily used as instance variables, or method arguments, and then the specific type of container can be instantiated and used as if it were just the base class.
At the current time, we have the following base classes:
- dkit::FIFO a simple first-in, first-out queue
Async I/O Package
As we get away from the low-level classes in DKit, we start to see how these components can be put together to form more significant components. One such group is the I/O package based on the boost ASIO package. The problem with the boost package is that it still takes a non-trivial amount of work to put together some of the most fundamental things you'd need in a socket I/O library. Certainly, this is for flexibility, but when you're trying to handle exchange feeds, it makes it a lot easier if you don't start from that point, but maybe a little higher up the food chain.
This is the purpose of the DKit I/O package - to bring a set of completed, high-performance socket components that makes buidling these types of systems much easier.
DKit News
Built Templated Conflation Queue in DKit
This morning I finished up work on a conflation queue for DKit. The idea is pretty simple - take a queue and a trie, and based on the key value for the elements in the trie, allow updates to the values still in the queue, but keeping their relative placement in the queue such that when they are popped off, the latest value is pulled off, and the element is considered removed from the queue. It's a very common thing to have in processing market data when you don't need every tick, but what you do need is the very latest information when you can get it. Such would be great for risk systems, but not so great for execution systems - depending on the strategy. Read More »
Adding a Flexible Sized Key Templated Trie to DKit
Today I spent a good chunk of time writing a 64-bit key trie as the next basic component of re-writing the exchange feeds in a lot better component library than I had previously written. The purpose of the trie is to have a lockless storage mechanism that has very fast insertion and access methods. The last version I'd written was based on a specific element being stored - a pointer to a Message. In this version I wanted to be able to store any template type - including pointers, and handle the disposal as needed Read More »
LKit - C++ Library of Lisp-Language Parser
Introduction
A recurring theme in a lot of the systems we've developed in the last decade is some sense of a control language or internal scripting language. There are several existing languages like Lua and Python that are designed for this purpose, but when you want to have really unique data constructs, you often need a lot less than these languages have to offer.
Specifically, what you really want is a limited set of datatypes and control structures so that you can build in the specifics you need. For this purpose, we've created LKit. It's goal is to allow for simple numbers, boolean, and timestamps. No string, no structs. Just arrays, lists, and these simple data elements.
Current Status
At the current time, the basics of the language are working, but we need to still handle the definition of functions in order to finish the basic language features. Yet it's proving to be something that we need to put in some more thought towards.
The overriding design goal is to make it fast so that it fits in nicely with other low-latency C++ projects that might be using something like DKit.