What's it all about, Alfie?

I've been working hard all day trying to work the configuration system into all the classes that it really should be used. It was really my fault for not putting the configuration service into place before writing all the code that should use it, but it just wasn't there, and I didn't ask enough questions about how it was going to work to get all the parameters, etc. in-place. Just means the job of getting the configuration into the system is a lot harder than it could have been.

Thankfully, and this is a really weak silver lining, I had a good start at what I wanted to use, but I hadn't taken it nearly far enough to make the process simple. It's further complicated by the fact that I need the configuration data at the lowest level components, and that means passing it down, down, down... Not horrible, but it takes a little bit of work to make sure that you put the configuration data in as few a places as possible with maximum coverage.

Not horrible, but not trivial. Just takes a lot of time.

I was reading a little of Wil Shipley's blog this morning - he tweeted that he had another post up about the potential Amazon/Microsoft merger, and ran across this post about how to code.

Solid Gold.

It's amazing to actually see that guys I believe are doing it right are, in fact, doing it exactly how I do it. It's a great affirmation of how I do my job.

Read it. Live it. Fantastic stuff.

I've been really slugging it out today. I mean really. Today has been a lot of work writing JSON configuration files for my ticker plant that will be read into a MongoDB that is the back-end for the configuration service. There's a lot of things I need to set up for this system - all the exchange feeds, all the distribution of the messages, all the mappings... it's a lot of stuff. In addition to all this, I really need to find a way to create the service coverage maps...

These things really got me down, I have to say. Not ashamed to admit it. There's something to be said about not getting bogged down, but when you're in the middle of it, the best thing you can do - I have found - is to just push through it. Taking a break isn't going to help - you have to come back to the exact same place. That's no good. There's no way to go around the problem, you have to go through it.

Kind of a drag, but it's what you have to do. And in the end, it's really what separates the good from the middle-of-the-road. If you push through when you're sick and tired of doing something, then you're going to get it done faster, and move on to something that you like to do. Let it really slow you down, and you're spending more time doing this thing you're going to hate even more.

Well... gotta get back to it. Pushing through it.

This morning I noticed that Google Chrome dev 7.0.544.0 was out, with several fixes including some Mac changes in the accelerated drawing code. Well, isn't that nice? Written proof that there might actually be accelerated drawing in the browser - that would be very interesting. But it really doesn't matter - it's a good browser, and it's nice to see progress.

I've spent most of today doing two things: weaving the authentication data into the Ticker Plant client code, and making a nice package (tarball) of the client include files and libraries for other groups at The Shop. Over the last few days, I've been doing the "second pass" on the code - looking at those supporting sub-systems and components that I've been "faking" with simple hard-coded values. For example, the original authentication system was a very skeletal 'client' of the authentication service and a simple 'token' that contains the important data, but the tests for the validity of the token was always true, and the client just created an empty one.

Like I said, very incomplete - but it worked and wasn't crucial to the functional aspects of the codebase. It's just the supporting stuff that the app had to have before it went live, but could always be done later. Well... now is that later.

Interestingly, putting the AuthToken in the code was surprising harder than I had expected. I simply hadn't exposed the ability to set the AuthToken at the highest levels of the code, which, it turns out, is exactly what I needed. Unfortunately, to find this, I had to do a lot of digging because I wasn't logging the requests that failed due to invalid AuthTokens.

First, I had to log things a little more, and then I saw the problem. Then I needed to see how often I was doing the same things in different places in the code. These weren't always done the same way in the code, so I took the time to standardized the way I was doing the authentication work. This took even more time. In the end, I was able to get it all working, but it took a lot more time than I had expected.

After that, I got pulled into a few meetings that pointed me in the direction that we might have a C++ client sooner rather than later. What I wanted to do was create a package - a tarball, that contained the libraries and include files so it's easily used by other groups. Making it was really a process of finding the minimum set of include files that I would need. Then I just had to stage them into a good directory structure and tar them up. I made it clean with some nice dependencies so that if we change one of the headers, we'll be able to know to make the package again.

Lots of good things done... but it was a lot of details to get done.

This afternoon was spent testing the ticker plant in all it's glory - that means loading data from the configuration service, checking the login service, and getting ticks moving. I realized that there were several parts of the system that weren't as polished as they should be - specifically in the configuration service client, and even a little in the Broker client code.

The configuration client didn't have a way for me to ask for a series of keys, and wait until all of them had returned. I had to do it one at a time. While it's not horrible, it's not as efficient as the system could be - after all, we created the asynchronous calls to the Broker just for this case.

So I added in the ability to get() a list of keys and have it returned in a variant list. The code actually sends out all the requests for the values asynchronously, and then waits for them all to return. There's a timeout, and a logging, if the complete set of values doesn't come back, but that's not something that's likely to happen.

In the end, I cut 40% off the time needed to get the configuration values this way. Nice.

This morning I finally was able to push through and finish the code for my persistence system for the ticker plant. We have a cache of messages in each ticker plant, and they are typically fed from the exchange ticker feed, but in the event that we need to restart the server, it'd be nice to be able to save and load the state of the messages (ticks) so if we have to restart, we can keep the cache in roughly the same state after the restart as before.

This is very important when you need to have a stable price flow. You can't have a restart take out the messages that came in the morning and haven't been seen since. Well... I'm counting on a service hanging off the Broker that is going to handle all the saving and loading for me. I'll simply send maps/lists of data (serialized messages) to the configuration service and when I restart, I'll read from the same. It's fine in that it'll work regardless of the machine, and we have all the tools at our disposal, but the problem is that we need the Broker and it's services up before we can start.

Minor point. If we can't get to the Broker and it's services, we're in a lot bigger problems.

So I got that written up, and a good bit of testing on the authentication service and the basic configuration service. I need to go back to the code now and start putting in calls to the configuration service so that these components and apps can load their configuration from the right source.

Never a dull moment.

After getting the authorization client done this morning, I took a little time to write a configuration client for the Broker. One of the things we are going to need is a way to store all the configuration data for the different applications and clients in the system. Additionally, we need a place to store the current state of the ticker plants so that if they need to be restarted mid-day, we have a place to load them and come back to very nearly the same state.

Now nothing's going to be perfect, so there's going to be some data loss in the event of a restart, but the idea is that we need something that's going to have a recent history of the state of the machine so that we can get it back to the same place after restart. And after hours, it becomes perfect backup because the data isn't changing, so restarts for new code, etc. are easy to implement.

The scheme we've adopted is that there are several 'levels' to the configuration scheme - global values, local values, and user values, which makes it easy to read the same value at different levels and have something that personalizes the value as you get closer to the user. For example, we can set a global value for "timeout", but then a user value that's greater than that - or less than, based on the user's specific needs.

We also have it in a very simple key/value store so it's easy to put complex data structures in the config system, and give that it's read/write, you don't have to make a particularly fancy GUI to maintain it as the app itself can set it's state to some defaults, and then write this out to the config service and it becomes the values to read back in.

The problem I have right now is that it's written, but the configuration service itself isn't. So I'm a little ahead of the curve. As soon as it's done I'll fix any differences and then start using it in my code. No time like the present.

We're getting to the point that we're putting things together at a higher level and starting to look at the larger (near final) ticker plant and it's clients. Friday I started work on taking the authentication client (AuthClient) code and taking the API we created and implementing it to mate up with the Broker's loginservice which is our Gold Standard for authentication.

The exchange with the service is pretty nice, and the work we've done up to this point makes it a lot easier to work with. You basically have to pass it a map with the key/value pairs of the login name and password, and it'll return to you a uuid_t that is the authenticated value. If it's not a uuid_t, then it failed - for whatever reason. I end up logging the failed attempts, and clearing out the AuthToken. It's not really rocket science, but it's very effective.

I've been able to remove all the hard-coded values and "debug" code so it's really working. It's nice to get one more thing really hooked up. It's getting closer, day by day.

This morning I checked and iTerm2 Alpha 10 was out, and the release notes for this update are impressive:

The focus of this release is bug fixes. It also features a major performance improvement as well as better internationalization support. This is the first 64-bit release.

• Rewrote drawing code for up to 10x speedup.
• iTerm2 is now a 64-bit binary!
• Improve text rendering by making baselines always line up when mixing fonts and in IME.
• Input method editor improved.
• Bold selections have proper foreground color (bug 106).
• Fix bug where cursor remains I-bar in title (bug 88).
• Resynchornize properly after invalid utf-8 sequence (bug 73).
• Fix bug where opening a tab changes window size.
• Fix race condition when toggling full screen and a draw occurs.
• Fix drawing bugs.
• Fix some minor memory leaks.
• Fix bug where tab bar would not draw correctly after window resize (bug 167).
• Fix bug where one-word commands in a bookmark didn't work (bug 166).

I'm most pleased with the move to 64-bit and the speed-up of the rendering code. Fantastic work!