What's it all about, Alfie?

Last night I had to explain some very bad clojure storm code to a couple of guys in the group. The reason for this was the new assault on the latency - as opposed to trying the one thing I said to try - which was remove one of the kafka publishers. But that's here nor there because that will be the last thing tried because it's my idea.

But because of this explanation, and the horrible code, something that should have taken about an hour or two - at the outside - is now sitting at a day and a half... and one of those days - yesterday - was a completely wasted day. Nothing was done on this because of the very bad code written by the developer asked to do it, and the fact that the other guys either didn't get to it, or wasted too much time before asking for my help (arrogance or marginalization - both are bad traits), and so a day is gone.

But this morning, I'm chatting with the author of the bad code, and he's thinking it's all good... that they refactored the code, and it's all happy skippy. But it's clearly not. And I'm chatting to him via IM, and not in person - at the office.

And I realize that it's times like these that I need to leave this team, and the management is right. If they are completely satisfied with this kind of work - then I'm not right for this team - at all. There was plenty of playing around yesterday... everyone but me took the lunch time off to go watch a talk... and there was no one in before 9:00 am - except me... and a day was wasted in the solution.

Yeah, this is not the kind of work I want to do.

This afternoon I'm forced to stay late because two guys in the group needed to have some ver bad clojure code explained to them because the guy that wrote it was off for the day, and the code he wrote was bad - to say the least. First, to be fair, this is clojure code, so it can be tough for some to understand, but these two guys aren't bad, so I don't think that was the issue.

No, the issue is that you can read clojure code a lot sooner than you can write good clojure code. For example, this was in the most convoluted function:

  (defn build-a-bridge
    [start-point end-point & [lanes type width]]
    (let [lanes (or lanes (/ width 20))
      ... ))

in a few short lines, he'd done so many bad things - that actually would work - that it made understanding this code very hard for these two guys.

Start with the re-valuation:

    (let [lanes (or lanes (/ width 20))

yes, it's possible, but in the real code, there was so much else going on it was hard to see this (the guys missed it), and so they thought that the lanes argument was nil, when in fact, it was being re-valued in the let. Yes, it's legal, but it's bad form, and the reason it was done is because the author really didn't understand optional arguments:

  (defn build-a-bridge
    [start-point end-point & [lanes type width]]

Here, the type of the bridge is non-optional - and so, really, are the number of lanes. Yet he was in so much of a hurry to get this done in the day, that he slapped things wherever he needed to to get them to work. Then, when the optional args were nil, he used the re-valueing to "make them right".

To call it hacks is being generous. It was a horrible way to use functional code, and it was all because he wanted to impress. Impress management that he's capable of getting the feature out in a day. In truth, it should have been an hour, but a day for him was within reasonable limits. But the time slipped by, and he started to panic.

I think it's fair to say that a panicked rookie clojure developer is bound to make some pretty bad code. And so I had to stay late to explain this code to the others in the team that couldn't figure it out for themselves.

And I'm the one that has to leave... Insanity.

Part of the New Direction that we are embarking on is to re-examine the topology that runs the data feed. Now I had built this, and done experiments covering months to get this to the point it was. Yet one of the managers talked to another Storm developer in another division, and that developer convinced him - without any knowledge of the specifics of the topology - that what I was doing was "All wrong". So my manager told me to work with this guy to fix it.

So I followed his advice and made the changes.

It didn't help, but it didn't make things worse.

Now today I'm being asked to revisit the topology and "Scale it back" because my manager is convinced there is something else in play here. OK, so I do as I'm told - because they are clearly not interesting in my opinions or we'd still be at the topology structure we had before all this.

So I start the experiments with a baseline, and then I start halving things: half the workers, half the bolts - all to get numbers to see if I'm going in the right direction. I don't fine-tune things until I'm really close, and half is easy to work with because we're still doing coarse tuning.

After about three experiments, we're far better, far faster, and have higher peak capacities during time of load. All is looking very good. Then I look at the Storm UI to the Capacity numbers, and I fine tune a little. This guy is a little high, so add some. This guy is very low, he doesn't need as much. Not a lot, just a little tweaking.

In the end we're looking a lot better from all the metrics. Good.

And the values?

Yup... just the ones I had before all this started.

I really am amazed about the real joy in using GitHub. It's hard to imagine the vision they had... Let's take git, and then build an entire culture and eco-system around it - all in the browser! That's some vision. Yet this morning I was able to put targeted comments on the lines of a pull-request with syntax-highlighted examples, and see it all in preview mode. It made me smile. This kind of attention to detail is really inspirational to me.

It's more than source control - it's workflow... collaboration - it's a wonderful framework with which to do group development - or personal development, for that matter. I really do enjoy working with it - even on really bad code. And the style... it's not always been perfect, and they have made some changes I might not agree with, but they have done it up to the nines. There is certainly no way someone is going to say they haven't sweated the details.

So even when I have to spend 30 mins making comments on code that should not have been written, I'm happy that I'm doing it in GitHub. What a great tool!

Today I've been just trying to keep things going because that's what I've been asked to do. Interestingly enough, the new management team that's out on the West Coast for a trip pressing the flesh with users this week is now convinced that everything we have been doing - and when I say "we" - they really mean me, is totally wrong.

They now believe that shared Storm as a service is the magic bullet and will make everything work perfectly. They believe my work has been lazy, and cryptic - at best, and incompetent at worst, and they now want to implement a plan to Clean Things Up.

Never mind that I have a plan to find out the issues plaguing us, and if they just started on that plan, I feel sure we'd see the solution... but that's me again - the lazy, shiftless, good-for-nothing that got us into this mess in the first place.

To do this, let's find out where every machine is in the data center, and then map all the switches and routers - not that we can change anything, but it's possible that this will explain the problem. Let's re-tune the topology because a manager has the feeling that it's wrong - never mind that he is that - a manager.

Let's do all these things that stem from the minds of the managers, and let's all do them Right Now! Drop everything... add five people... this has to be done now.

If I weren't in it, I'd laugh. It's a WTF of epic stature. Insanity.

Well... after the shock of yesterday, I have to admit, it was a very hard thing to get going today. I'm planning on taking the rest of the week as Work From Home, but even then, the crushing fear of being without a job, and having all the responsibilities of my little house - and Liza's and the kids... well... it was one of the worst starts on a day since those first few days in the hotel. It was just horrible.

But I had a feeling that I needed to get up - I needed to run, and I needed to pretend that my life was going to be OK. That I was going to pull through this - I didn't know how, but somehow, someway, I was going to make it. As my Mom was always fond of saying: The motion becomes the emotion - and I think she stole that from her Dad.

So I got up to run, and do my regular morning routine. It wasn't easy, but I tell you, I've never enjoyed Stan and Neil on SportsCenter as much as I did that morning. They helped get the day going. Then it was time to think... what to do? During the rest of my work-out, getting cleaned-up, and dressed, I worked through what I needed to do... the emails that needed writing, the people I needed to reach out to.

So I hit the morning going - emailed the last place that made me an offer, to see if it was still open (nope, not surprising). Then it was chatting with a few friends, pinging them about what's going on, and chatting to a friend that just left The Shop for a West Coast start-up. It was nice to talk to him... it's been a tough couple of years, and he's known me through almost all of it.

Then I chatted with a friend that I worked with a while back, and asked him about anything he's heard. He's still in Finance, and it's been two years, plus, since I've been there, and I do miss it. He said they are hiring, and that I'd be great for the job(s). So we set up some time to talk on Monday.

My hope picked up.

Then I saw that my former manager at The Shop - who left a few months before for another job on the West Coast, and he'd been fired yesterday, too. Wow... I told him I was sorry it happened, and he explained that the CEO got shaken when the company came under a little fire, and my friend was jettisoned. It felt like fate. Same day. Wow.

My hope picked up a little more.

I don't know what the rest of the day will bring - I fully expect to have dark times - I have no future now, so I'm going all on faith, but I have a few glimmers of hope, and if I can just hold on to those, maybe I can weather this storm, too.

Hope so.

Today I ran into a Legacy Bug - some little bit of code that used to work, but hasn't been used in so long that when someone really did try to use it - well, it kinda worked, but not exactly. It was really a simple database column mapping. What I had was:

  (defn hit-to-array
    "Function to format the 'hit' map of data into a simple sequence of the
    values for the map - in a specific order so that they can be easily
    understood."
    [arg]
    (if arg
      (map arg [:variant :country :browser :t-src :b-cookies])))

and I needed to change it to:

  (defn hit-to-array
    "Function to format the 'hit' map of data into a simple sequence of the
    values for the map - in a specific order so that they can be easily
    understood."
    [arg]
    (if arg
      (map arg [:variant :country :browser :traffic_source :users_count])))

because we had done a database restructure, and the rows coming back were now focused on the business names, and not the internal representation. The old code was returning nil, and the new code was properly finding the fields.

Similarly, I had to change:

  (defn sale-to-array
    "Function to format the 'sale' map of data into a simple sequence of the
    values for the map - in a specific order so that they can be easily
    understood."
    [arg]
    (if arg
      (-> arg
        (util/update :billings util/to-2dp)
        (map [:variant :country :browser :t-src :d-chan :orders :qty
              :billings :consumers :est_monthly_billings
              :est_monthly_billings_pct]))))

to:

  (defn sale-to-array
    "Function to format the 'sale' map of data into a simple sequence of the
    values for the map - in a specific order so that they can be easily
    understood."
    [arg]
    (if arg
      (-> arg
        (util/update :billings util/to-2dp)
        (map [:variant :country :browser :traffic_source :demand_channel
              :orders_count :qty :billings_total :buyers_count
              :est_monthly_billings :est_monthly_billings_pct]))))

because we dropped the qty field - no one wanted it - and we again changed the names to those more user focused names. It's not a big deal, but it makes a big deal to the guys now implementing the feature.

I wrote this up quite a while ago, and it never got called, but there wasn't a really horrible error - so it didn't crash when they called it - it just didn't return the right data. Now it does.

I love these simple fixes - and it's pretty much all about the functional style of coding.

I just had one of the oddest conversations I think I've had in a very long time. The shock factor is right up there with Liza telling me she wanted a separation - and then a divorce. It's really in that league of a shock.

It started out as what I thought was a conversation about how to make the work I do - and really how I interface with the rest of the team - not that it's quickly doubled with the return of the old manager and a new Director. Since this talk was with the Director, and I'd worked with him in the past, I felt he was the right guy, and this was the right time to have a talk.

He started with some of the most flattering compliments:

You're the best engineer I've ever seen. Some are as smart, but they aren't as productive. Others are productive, but they aren't nearly as smart. And some are really good - but don't put in nearly the hours you do. You really take it to a new level.

and then went on to try and convince me that I wasn't a good fit for The Shop.

I was trying to work through ideas, but he kept coming back to these same themes - not the right fit. And then it hit me from out of the blue - he was asking me to leave the Team.

Wow... Stunned.

We were just in a meeting where something I did on the side, for another team, is responsible for $500,000 a week. Yeah, $2 million a month, and I'm the best he's seen - but I have to leave.

But wait... there's more...

He wanted me to stay through Black Friday and Cyber Monday - and for helping them out this way, they'd let me stay until January when another stock block vests, and it'd be worth about $20,000 to me.

Help them get through the busiest season... and be allowed to pick up another $20k while during that time - and for every month after that, my work is generating them millions of dollars every month.

Wow...

I was shocked, stunned, hurt, and I'm still not over it. Not by a long shot.

The divorce Liza wanted in dragging on into it's third year, and no end is in sight - no matter how I try to help things along. Then this. Wow.

I talked to a lot of folks - all were shocked. Those that have worked with me were stunned, and others were just in disbelief at this - thinking without HR in the room, this was all kinda iffy. It was all mildly reassuring, but it didn't solve the problem I faced: I needed to get out.

[11/6] UPDATE: I talked to an old friend, and his firm - back in Finance - is hiring, and so I'm talking to him on Monday. It sounds like a nice job, and that's what I need - a place where performance matters.

[11/7] UPDATE: Wow... and they're laying it at my feet. I've talked to someone in the group and management is saying this is my choice - that I'm abandoning the group. Sorry, Charlie... you need to own this. You want me off - be the responsible manager and take the hit - if there is one. Maybe it'll just be praise for the decision, but it's exceptionally cowardly to say it's all my choice.

I have to admit when I'm wrong - it's just the only decent thing to do, in my opinion. And today, I was schooled by my boss's boss's boss at The Shop saying that we did, in fact, have the user-agent in the nginx logs for the user email open messages I was just finishing up on. I was sure we didn't, and when I showed him... well... I instantly apologized for my mistake - I was looking right past it. Totally my mistake.

The reason this all came up was that the messages have an app_version field, and typically, for the other user actions, this is the browser and version for the web, or the version of the Android or iOS app - something that lets us know a little bit more about the platform it's coming from. Sadly, without this user-agent, I was stuck looking at the URL of the nginx log, and that didn't have much of anything really useful.

With this, I was able to easily parse it - already had the functions for it - and then drop that in just like all the other messages. It was a very simple fix, but it had a profound effect on the data quality. Much nicer to know this. Much.

This afternoon, before I leave to go vote, I wanted to add in the code to decode all the email opens that occur in a day. I have actually been working on decoding these messages for a while, but I've had to divert my attention to other, more pressing, needs of late. Finally, this afternoon, I was able to get back to the email opens, and it was nice to close it out.

It was a basic addition to the topology, and while I could have combined it with the other email-based data feed, I have chosen to keep it separate for now - just to be able to monitor the send traffic separate from the open traffic. I will say that I did have one logic error in the code - and that was because the email opens are simple nginx logs, and those aren't formatted as JSON - so I had to parse that first, and then process it, and I had the initial checks done before the parsing. They always failed, and that was an issue.

But a quick logical walk-through, and I found the problem, and we were off to the races. What I was surprised about was the very moderate levels of traffic at 2:00 pm in the afternoon. Now it's probably much heavier when the sends are done, so we'll have to watch it in the morning, but it's nice to see that the addition isn't a torrent that floods all processing - immediately.

I was expecting more load - maybe tomorrow will show it to me.