What's it all about, Alfie?

Today was spent getting the timing of events finished up. Yesterday, I took a lot of time getting the shutdown working, and then I had to make the movement of messages work. It's pretty funny to look at the code in retrospect - I had just assumed that things would be there when I needed them, and in my development case with hard-coded values, it was. But making it work with the actual timing of events was a lot harder than I had thought.

Basically another day spent on getting the start-up and shutdown right for all the components of the system. Ick.

Today I spent a lot of time working on getting rid of core dumps when I shut down the ticker plant. This seems very obvious, but after I just did all the work to fold in the configuration and then even more work today to get the authentication token into all the configuration code, it required that I change a lot of little things around. The upshot of that was that the subtle interactions on the shutdown were quite broken and I had to fix them all up again - but not revert to the old system due to the configuration and authentication changes.

Little things I ran into made this a lot harder than it sounds. We have boost's asio to contend with, and the fact that we need to cancel any pending I/O with them on the shutdown. However, that's going to generate an error in the socket communications, and from that we'll try to shutdown the socket. This is a nasty loop that makes it very difficult to unravel.

What I had to do was to be very careful about the two different conditions, and if I'm being interrupted, then I need to assume I'm being told to shutdown and not assume there's something wrong I need to attempt to recover from. Thankfully, the error messages are clear enough to make this possible, but it's still a lot of detail work about when things are happening and who's responsible for doing what.

All this took me the better part of the day - with the initial part being trying to get the timing of the initialization of everything right as well. It's like I spent two days getting nowhere as I didn't advance the codebase one feature - I just made it possible to use the configuration and authentication tools we have to use. That's important, do be sure, but it didn't feel like I got a lot done.

I've been working hard all day trying to work the configuration system into all the classes that it really should be used. It was really my fault for not putting the configuration service into place before writing all the code that should use it, but it just wasn't there, and I didn't ask enough questions about how it was going to work to get all the parameters, etc. in-place. Just means the job of getting the configuration into the system is a lot harder than it could have been.

Thankfully, and this is a really weak silver lining, I had a good start at what I wanted to use, but I hadn't taken it nearly far enough to make the process simple. It's further complicated by the fact that I need the configuration data at the lowest level components, and that means passing it down, down, down... Not horrible, but it takes a little bit of work to make sure that you put the configuration data in as few a places as possible with maximum coverage.

Not horrible, but not trivial. Just takes a lot of time.

I was reading a little of Wil Shipley's blog this morning - he tweeted that he had another post up about the potential Amazon/Microsoft merger, and ran across this post about how to code.

Solid Gold.

It's amazing to actually see that guys I believe are doing it right are, in fact, doing it exactly how I do it. It's a great affirmation of how I do my job.

Read it. Live it. Fantastic stuff.

I've been really slugging it out today. I mean really. Today has been a lot of work writing JSON configuration files for my ticker plant that will be read into a MongoDB that is the back-end for the configuration service. There's a lot of things I need to set up for this system - all the exchange feeds, all the distribution of the messages, all the mappings... it's a lot of stuff. In addition to all this, I really need to find a way to create the service coverage maps...

These things really got me down, I have to say. Not ashamed to admit it. There's something to be said about not getting bogged down, but when you're in the middle of it, the best thing you can do - I have found - is to just push through it. Taking a break isn't going to help - you have to come back to the exact same place. That's no good. There's no way to go around the problem, you have to go through it.

Kind of a drag, but it's what you have to do. And in the end, it's really what separates the good from the middle-of-the-road. If you push through when you're sick and tired of doing something, then you're going to get it done faster, and move on to something that you like to do. Let it really slow you down, and you're spending more time doing this thing you're going to hate even more.

Well... gotta get back to it. Pushing through it.

This morning I noticed that Google Chrome dev 7.0.544.0 was out, with several fixes including some Mac changes in the accelerated drawing code. Well, isn't that nice? Written proof that there might actually be accelerated drawing in the browser - that would be very interesting. But it really doesn't matter - it's a good browser, and it's nice to see progress.

I've spent most of today doing two things: weaving the authentication data into the Ticker Plant client code, and making a nice package (tarball) of the client include files and libraries for other groups at The Shop. Over the last few days, I've been doing the "second pass" on the code - looking at those supporting sub-systems and components that I've been "faking" with simple hard-coded values. For example, the original authentication system was a very skeletal 'client' of the authentication service and a simple 'token' that contains the important data, but the tests for the validity of the token was always true, and the client just created an empty one.

Like I said, very incomplete - but it worked and wasn't crucial to the functional aspects of the codebase. It's just the supporting stuff that the app had to have before it went live, but could always be done later. Well... now is that later.

Interestingly, putting the AuthToken in the code was surprising harder than I had expected. I simply hadn't exposed the ability to set the AuthToken at the highest levels of the code, which, it turns out, is exactly what I needed. Unfortunately, to find this, I had to do a lot of digging because I wasn't logging the requests that failed due to invalid AuthTokens.

First, I had to log things a little more, and then I saw the problem. Then I needed to see how often I was doing the same things in different places in the code. These weren't always done the same way in the code, so I took the time to standardized the way I was doing the authentication work. This took even more time. In the end, I was able to get it all working, but it took a lot more time than I had expected.

After that, I got pulled into a few meetings that pointed me in the direction that we might have a C++ client sooner rather than later. What I wanted to do was create a package - a tarball, that contained the libraries and include files so it's easily used by other groups. Making it was really a process of finding the minimum set of include files that I would need. Then I just had to stage them into a good directory structure and tar them up. I made it clean with some nice dependencies so that if we change one of the headers, we'll be able to know to make the package again.

Lots of good things done... but it was a lot of details to get done.

This afternoon was spent testing the ticker plant in all it's glory - that means loading data from the configuration service, checking the login service, and getting ticks moving. I realized that there were several parts of the system that weren't as polished as they should be - specifically in the configuration service client, and even a little in the Broker client code.

The configuration client didn't have a way for me to ask for a series of keys, and wait until all of them had returned. I had to do it one at a time. While it's not horrible, it's not as efficient as the system could be - after all, we created the asynchronous calls to the Broker just for this case.

So I added in the ability to get() a list of keys and have it returned in a variant list. The code actually sends out all the requests for the values asynchronously, and then waits for them all to return. There's a timeout, and a logging, if the complete set of values doesn't come back, but that's not something that's likely to happen.

In the end, I cut 40% off the time needed to get the configuration values this way. Nice.

This morning I finally was able to push through and finish the code for my persistence system for the ticker plant. We have a cache of messages in each ticker plant, and they are typically fed from the exchange ticker feed, but in the event that we need to restart the server, it'd be nice to be able to save and load the state of the messages (ticks) so if we have to restart, we can keep the cache in roughly the same state after the restart as before.

This is very important when you need to have a stable price flow. You can't have a restart take out the messages that came in the morning and haven't been seen since. Well... I'm counting on a service hanging off the Broker that is going to handle all the saving and loading for me. I'll simply send maps/lists of data (serialized messages) to the configuration service and when I restart, I'll read from the same. It's fine in that it'll work regardless of the machine, and we have all the tools at our disposal, but the problem is that we need the Broker and it's services up before we can start.

Minor point. If we can't get to the Broker and it's services, we're in a lot bigger problems.

So I got that written up, and a good bit of testing on the authentication service and the basic configuration service. I need to go back to the code now and start putting in calls to the configuration service so that these components and apps can load their configuration from the right source.

Never a dull moment.

After getting the authorization client done this morning, I took a little time to write a configuration client for the Broker. One of the things we are going to need is a way to store all the configuration data for the different applications and clients in the system. Additionally, we need a place to store the current state of the ticker plants so that if they need to be restarted mid-day, we have a place to load them and come back to very nearly the same state.

Now nothing's going to be perfect, so there's going to be some data loss in the event of a restart, but the idea is that we need something that's going to have a recent history of the state of the machine so that we can get it back to the same place after restart. And after hours, it becomes perfect backup because the data isn't changing, so restarts for new code, etc. are easy to implement.

The scheme we've adopted is that there are several 'levels' to the configuration scheme - global values, local values, and user values, which makes it easy to read the same value at different levels and have something that personalizes the value as you get closer to the user. For example, we can set a global value for "timeout", but then a user value that's greater than that - or less than, based on the user's specific needs.

We also have it in a very simple key/value store so it's easy to put complex data structures in the config system, and give that it's read/write, you don't have to make a particularly fancy GUI to maintain it as the app itself can set it's state to some defaults, and then write this out to the config service and it becomes the values to read back in.

The problem I have right now is that it's written, but the configuration service itself isn't. So I'm a little ahead of the curve. As soon as it's done I'll fix any differences and then start using it in my code. No time like the present.