What's it all about, Alfie?

I've run into a problem with my Ticker Plant feeds, and the best solution to the problem is to create a component that can fit in-line with the feeds and "promote" and "embellish" the trade messages (prints) with the current volume traded as well as per-exchange volumes traded. The issue is that if I allow these trade messages to conflate, information about the individual trade will be lost. Sure, the most recent data will survive, but the volume change represented by the conflated trade will be gone forever.

What I needed was a very simple, very lightweight, component that would take the stream of trades and aggregate the volumes - as well as the high, low, open values, and then "attach" those values to a new trade message - one that is a subclass of the original, and so can take it's place in all the processing, but also includes this cumulative volume and limit values.

Today has been spent creating this and fixing issues associated with it. There's a lot of little things to write including the serialization and deserialization of the component, and the handling of this new message type, as well as automated tests to make sure I'm accumulating things properly. It's close, and I just need to write a few more tests, but for now they day is over and I need a break.

Most of today has been spent trying to get my simple C++ wrapper around the hiredis C library for redis working in a way that allows for a significantly more robust usage pattern than I originally had. Specifically, all was fine until I shut off the redis server, and then my client would try to recover and reconnect and end up dumping core. The problems are only made worse by the fact that I really had no support docs on the hiredis site - only the source code, which is optimistic in the extreme. No argument checks, etc. make it ripe for a problem if it's not used exactly right.

Clearly, I wasn't using it exactly right, and those misusage patterns were what was causing the code dumps. So the first thing was to track down what I was doing wrong, and that meant that I really needed to become much more familiar with the hiredis source code. To be fair, it's a decent open source library, but it's missing so much that would have added so little to the runtime load and would have made it far more robust to the kinds of misusage patterns I had in place. After all, my code worked, so it's not that it was totally wrong, it's just that when things started to go badly, the things that you needed to do become far more important than when things are going well.

For example, if I wanted to send multiple commands to the redis server at once, you can run several redisAppendCommand() calls, but each really needs to be checked for it's return value. This isn't clear in the code, but it's very important in the actual system. Then there's the calls to redisGetReply() - typically one for each call to redisAppendCommand() - but not always. Again, you need to check for the critical REDIS_ERR_IO error that indicates that the redis context (connection object) is now so far gone that it has to be abandoned.

Then there's the reconnection logic. It's not horrible, but you have to be careful that you don't pass in any NULLs. There simply is no checking on the hiredis code to ensure that NULL arguments are skipped. It's simple to do, but it's not there - not at all.

In the end, I got something working, but it was hours of code dissection and gdb work to figure out what was going wrong and what needed to be done to handle the disconnected server and then the proper reconnection. Not fun, and several times I was wondering if it just wouldn't be easier to write my own as it's all TCP/telnet based anyway… but I kept going and in the end I have something that's reliable and solid. But it was nasty to get here.

This morning I had a funny little bug in the Greek Engine that was really a hold-over from my Ticker Plants, regarding the creation of the security ID - the 128-bit version of the complete instrument description for use in maps, keys, etc. The problem was that the exchange codecs were correct in their work, it was the creation of the security ID that was at fault, and I needed to dig there to find the solution.

Basically, I pack the components of the instrument definition into a 128-bit unsigned integer, and to so this efficiently, I need to have a pointer that points to where the next component needs to be "dropped in". When I got to the strike, which was passed in as a double, I had the code:

  // finally, slam down the strike
  *((uint32_t *)ptr) = htonl((uint32_t)(aStrike * 1000));
  // swap the network order bytes to host order
  byteSwap();

and because of the (aStrike * 1000), I was getting 64.099 when I should have been getting 64.10, among other rounding errors. What I needed to do was pretty simple, but essential to get the right answer:

  // finally, slam down the strike
  *((uint32_t *)ptr) = htonl((uint32_t)(aStrike * 1000.0 + 0.5));
  // swap the network order bytes to host order
  byteSwap();

and with that, we got the right numbers. We even got the right thousandths place in the case of 8.125, for example. I'm sure there was partly the need to cast the 1000 into a double, but the additional movement by a half was the thing that brought it home.

Now we're OK.

I ran into a problem that completely reinforces my belief that coding standards are a very good thing. And I was the culprit here - but only because I missed it, not by choice. What I had was a constructor for a Dividends set - basically dates and values in a table that needs to be treated as a single unit. The copy constructor for the instance looked like this:

  Dividends::Dividends( const Dividends & anOther )
  {
    // let the '=' operator do all the heavy lifting
    this->operator=((Dividends &)anOther);
  }

and the operator=() method looked like this:

  Dividends & Dividends::operator=( Dividends & anOther )
  {
    if (this != & anOther) {
      boost::detail::spinlock::scoped_lock   lock(mDivsMutex);
      mSetName = anOther.mSetName;
      mSecurityKey = anOther.mSecurityKey;
      mDivs = anOther.mDivs;
      mUpdateTimestamp = anOther.mUpdateTimestamp;
    }
    return *this;
  }

Now the trick here is to realize that the Boost spin lock is really just an atomic integer that they wrap with CAS operations. It's good, but it really needs to be initialized properly or it's not going to function well. What I'm doing in the copy constructor isn't wrong in the sense that it's logic is wrong, but the implementation is horrible.

When I saw this, it was clear to me what the bug was: the spin lock wasn't initialized properly, and the use of it in the operator=() method was locking the thread. Bad news. The solution was simple:

  Dividends::Dividends( const Dividends & anOther ) :
      mSetName(anOther.mSetName),
      mSecurityKey(anOther.mSecurityKey),
      mDivs(anOther.mDivs),
      mDivsMutex(),
      mUpdateTimestamp(anOther.mUpdateTimestamp)
  {
    // the initialized values do everything I need
  }

and the advantages are clear: since we're constructing a new instance, locking the new instance isn't necessary - no one knows it's there. Also, we properly initialize the mutex, and that's everything in this problem.

Standards. Get them. Live with them.

This morning we realized that some of the fundamental data we are using in the Greek Engine is not fixed for each security, but in fact, is dependent on the prime broker for that trade. This means that we needed to take that piece of data out of the security definitions table, and make it a separate data table, with set names, and all that noise, so that we could define the right rate in the profiles.

It's a complicated way of saying we took what we believed to be a dependent variable in the Greek Engine and made it independent. The upshot of that is that we needed to be able to handle data source updates as well, and that means threading this new independent variable's updating scheme through out the code. Not horrible, but it takes time to do it right and update all the classes where the control logic needs to flow.

In the end, I wanted to get this done before lunch, and I just made it. Nicely done, as the last time I did this it took me all day.

Today my co-worker and I finally figured out the last, nasty bug in the Greek Engine that I've been working on for quite a while, and it's brought to a close a stage in my life that brings with it a lot more questions than answers. I've been working on this project for about six months, and while that's not very long for a project of this type, it's a long time to be getting a lot of grief from upper management about the time this project is taking, and the fact that it's not already in production.

I'm no idiot, I know that the finance industry is focused on What have you done for me today?, but when I'm asked to build something like this Greek Engine, and it's clear that it's not a four week project, then it's tough to sit here and take the accusations of incompetence and neglect when what I'm building is for the benefit of the name callers. I'm thrown back to the Little Red Hen: Who will help me back the bread?

In the end, it doesn't really matter because I did this for me, and not for them. I choose to not deliver crap, and they are the hapless beneficiaries of my morality. It's not something I'm necessarily proud of, but it's what I've come to see as the way things often are. I make a choice, and for the most part, there are detractors, and many of those detractors are in fact beneficiaries of my work. I do the work under their critical and harassing eye, word, and act, and in the end, they are the ultimate winners in this game.

Which, of course, isn't true at all, is it? They don't really win, because they don't have the ability to do what I've done, and they probably don't even have the ability to maintain the work I've started. But they benefit because they get to use the software I've written, and as long as I'm willing to allow them to play their sad, little role, they believe themselves to be my master.

But they aren't.

And it's not even close.

But today I've finished the last major problem on this project. It's now going to be a bunch of little features that are pretty easy to put in the code. Nothing major, and there's no real performance issues to deal with. It's pretty much done. I can sit back and look at the work I've done and smile. It's amazing work, really. I never thought it'd come together in six months, but it has. That's an impressive delivery schedule in my book.

And that's the only one that matters.

This morning I'm reminded of the quote from Star Trek series, the Mirror, Mirror episode where Kirk, Bones, Scotty, and Uhura are in a transporter accident and are sent to an alternate universe where the Federation is evil and plundering everything in their path:

In an insane society, the sane man must appear insane.

It was quoted in a movie as well, but the title of that movie escapes me this morning. Suffice it to say that I feel like that sane man in an insane society right now.

I was talking to a high-level manager at The Shop the other day, and they were facing a difficult decision: confront their own people with their lack of skills, or come up with a "story" to tell them that makes it not their fault, but the fault of others, and arrive at the same place. But it isn't really the same place, is it? In one case, the person has to come face-to-face with the fact that they aren't really doing the job they think they are doing. This is hard. It's emotional, and it might possibly even get ugly. But it's the truth.

In the other case, the person not doing their job is allowed to believe they are and able to blame someone else for the situation. That's not the same place at all, is it? Nope.

But it saves this manager the effort of having to confront these under-performing individuals. And for this manager, it seems, that's a win for them, and that seems to be the best thing to do. Sad, yes. For it's the manager's job to actually manage people. Not tell them stories. Not blame others. What is going to happen if some day the truth gets out? Not good, I can assure you. Not good at all.

And it's like this all over this place.

About six months ago, a co-worker was getting sick of the work, and I don't blame him. He started looking. I told my manager/partner about this several times over the next six months, and then yesterday he informed us all he was leaving. No surprise to me, but a huge surprise to those not listening to me. Significant loss to the firm, but they all acted shocked. I'd been telling them this was going to happen for months, and they have the nerve to act shocked.

Crazy.

Over and over this place seems to be making the most short-sighted, ill-informed decisions possible. It's almost like they want to pick the wrong decision. More often than not, they are successful in that goal, and their choice is a spectacular failure. But the real question that I keep coming back to is simple: What am I going to do about it?

I can sit here, complain/document/laugh at the insanity, I can leave and laugh all the way to my next position. I can stay and try to fix things - knowing full well that the odds of that are long - at best. What can I really do?

I'm not a partner here. I'm not even a senior manager. Heck, I'm not even a manager. I'm a coder. What can I really do? Really?

Well… I guess the only thing I can do, is either stay and do my job to the best of my ability in a completely insane environment, or I can leave. I've already asked - several times, to move our little group of two, off this floor - maybe even out of this building - just to get away from the insanity. But it's falling on deaf ears. I'm sure as much as they think I'm a pain in the neck, they feel far better with me "right here" as opposed to anyplace else.

And then I can leave.

Don't want to leave. Really don't want to, but I don't see a lot of options. I'm really having a hard time dealing with this insanity, and that's what it's really going to take to "fit in" here - become one of the inmates in this asylum. Not quite sure that's where I want to be. Would you?

Just don't know.

I've been having a tough day for a Monday - not that all days aren't tough, but this one is really getting on me for a lot of reasons. It's probably all about childhood - isn't everything? But this morning I was really in no mood to listen to people essentially question my honesty. If I'm new at a job, and you don't trust me - and believe me, I can certainly understand that, then don't give me responsibilities that are outside your comfort zone. Make the trust and comfort of the task and the person match.

As I do more things for you, and prove myself over the course of many projects and many months, then you can either increase that trust - or decrease it as the case may be. But at no time should you as a manager feel like you have no control over the situation. You are the manager, after all. If you're uncomfortable, ask questions until you are comfortable enough to make decisions. If you never get to that place, then you have, by default, made a decision - that you can't trust me. Then I need to go. Period.

But if you talk to me and I am able to make you feel comfortable about the work, or the time or whatever it is that's bothering you, then we both come out ahead - you for asking the questions to get the understanding necessary to extend the trust, and me for taking the time to do the same. It's a win-win every single time. If I can't answer your questions, then your fears were founded, and if I can't make you feel better, then I need to be cut back - managed more closely, or even let go.

All this I'm very comfortable with, because I've been on the other side of the management table as well. It's not easy to walk the fine line of managing and allowing creative people room to invent and create. But it's all about trust. Nothing more, nothing less. If I trust you, then whatever it is that you do for me, good or bad, I'll trust that it's the very best you could have done, and no amount of second-guessing is needed. Things happen, and this is one of those times.

But if I don't trust you, then even if it's on-time, maybe you padded the times and had some of this done already. No trust means that I can't believe a good outcome even when it's a genuinely good outcome.

What's bothering me today is the fact that I feel I'm existing in an atmosphere of a near complete lack of trust. It's not something that someone will come up and talk to me about - it's about the setting of near crazy expectations of the work I have been assigned to do, and how those expectations have made it so that I simply can't be believed. If the project I'm on should have taken two man-years, there's no way I can say that now without people thinking it's just "excuses time" by me. I made the horribly niece assumption that people would know what it is they asked of me, and adjust accordingly. But they didn't.

My mistake.

I'm going to finish this project, and when it's all done and delivered, and everyone is happy, I'm going to talk to those in power and have a frank discussion about the promises made, promises broken, and these expectations. All will be taken into account in what we choose to do going forward.

This morning, as I was getting ready, I realized that I was really worrying far too much about what was happening at work. In a technology all-hands meeting yesterday, the CTO presented the status and plans for a bunch of projects that are happening in the group. In fact, I think he hit on everyone that's more than a couple of days long - it was pretty exhaustive. So when I heard there that several projects didn't include the work I was doing - even though I was told my work was critical to these efforts, I got upset. Really upset.

If I'm working 13 hour days for this project, and it's not critical, what am I killing myself for? What's the rush? Why work so hard? No, I was told that this work I was doing was fundamentally important to the effort, and what's why I've been working this way. In short, I've been a Good Team Player - internalizing the external needs of the Team even though they are painful for me.

But in the show this morning I realized that there's no reason for any of this worry and concern. I'm giving it everything I can, and if it's not enough, then they need to replace me. If they think I'm doing a decent job, then I think things are just fine. After all, this is the place where it's expected that everyone be extraordinary (got this in my last review).

So I realized in the shower that there's no reason to get upset. I've been angry at management for quite a while, and in the end, the fact that they haven't fired me, or moved me onto a different project, means that they must think I'm "meeting expectations", and by their own definition, delivering extraordinary results.

I know they don't think this, but I'm tired that once again it seems that I've forgotten the most important lesson of working in a group - do your best, and if that's not good enough, then it's time to move on.

I'm ready to make them make that call.

I had a somewhat heated discussion with a co-worker this morning and he's been at The Shop for a number of years while I've only been here a little over one. He's adapted to how this place measures success, and I can't blame him for it. He's tried to write code to the best of his ability, and when he thought he was being successful, he's been told it was a failure. He is reacting to the measurements of success and failure that he's seen, and in that sense, it's completely understandable why he sees things the way he does.

But that's harder for me to deal with. Much harder.

All the time I was talking to this guy I was thinking How can I tell him why I write code without coming off as a jerk? Which lead me to this post. I can't really tell him, but I can tell you, and maybe you'll understand.

Writing code isn't just about a paycheck for me. In fact, I'd say that if it ever becomes just about a paycheck, I think I'd get out of it. Coding is about the creation of beautiful, functional, code that fulfills a specific need in an elegant and awe-inspiring way. I'm not niece enough to think that all code will be this wonderful an experience, but that's the goal. When code is being written, it ought to have all of these factors in mind: simplicity, clarity, style, function, and design.

So why do I go through all the headaches? Why put in all the hours? I believe it's because when I get the opportunity to create software, I will create something I'm proud to have my name on. It'll be something that has a good number of those idealistic qualities, and I'll be glad I put forth all the effort.

So when I hear a good developer talk about essentially "looking on the bright side", I get a little worried because I think I've lost a fellow Creator. He's become a worker - happy to deliver whatever it is that management asks for - no matter how outrageous, and unrealistic. It makes me sad.

Everyone has bills. Everyone needs a paycheck, and I'm no different. But I think there's a way to pay those bills and do something that's aligned with your moral compass. And as extreme as that sounds, I think it is a moral issue. Someone can ask me to deliver them something that's unreasonable, but if I let them believe that it's OK and reasonable, then that's my fault. If, after I tell them it's unreasonable, and I'll try, but won't write junk, they still want me to do this, then I'll give it a try. But we all have to live with ourselves, and thankfully, I've been given a set of talents that allows me to be a little picky in the work I choose to do.

I have to feel that my life has been something good. I'm almost 50 years old. If I don't do that now, when will I?