What's it all about, Alfie?

I've been trying to get some data validation code going, and one of the best looking libraries is Prismatic's Schema. It's got both clojure and clojurescript, as well as a very well thought out scheme. It's got capabilities to run tests on code, in-line, or as data to filter like tests. It's very nice.

Problem is, it's meant to use clojure 1.5.1 or better.

But I wanted to see if we could get it going in clojure 1.4.0 for the Storm topologies we have. Turns out, it wasn't all that easy. But it appears possible.

First, let's change a few things in the project.clj file. First, make the version something that's clear it's not the normal version:

  (defproject prismatic/schema "0.3.4G"

and then, because clojure 1.4.0 doesn't have the EDN reader, we needed to add:

    :dependencies [[potemkin "0.3.2"]
                   [org.clojure/tools.reader "0.8.12"]]

and then changed the main version of clojure:

    :profiles [{:dev {:dependencies [[org.clojure/clojure "1.4.0"]

And then in src/cljx/schema/coerce.cljx we need to reference the correct EDN reader:

    #+clj [clojure.tools.reader.end :as end]

and then in test/cljx/schema/core_test.cljx we need to use a proper defprotocol for 1.4.0:

  (defprotocol ATestProtocol (test-fn [x]))

And at this point, we can run:

  $ lein do clean, reps, test

And it will work just fine - for the clojure part. The clojure script requires 1.5.1, but that's OK, as we don't need it.

Well... it's Thanksgiving Week, and that means a light week at the office, and then a long, stressful day with the family. I suppose it could be a lot worse, and it really kicks off the very best time of the year for me. Yeah, it's cold, and it's supposed to snow today, and it's going to get a lot colder and stay that way for the next six months, but hey... that's Chicago weather, and I knew that going in.

Today is the last full day at the office for me for the week. Tomorrow I have an appointment in the afternoon, and then the day after that, my oldest is moving back to town - and moving in with me, and then there's Thanksgiving, and the day after I'll be making sure things are working well and everything is humming right along with the systems at work for Black Friday and Cyber Monday.

Things at The Shop have been kinda stressful lately - I need to leave, not for cause, but because I make the folks in the team feel uncomfortable, and they would rather have a happy team than one that gets work done. I have known there's a mis-match here for a while, but it's another thing to present it in the way they have.

But hey... it's Thanksgiving Week... let's have a decent week, and relax a little. If we can.

Last night I had to explain some very bad clojure storm code to a couple of guys in the group. The reason for this was the new assault on the latency - as opposed to trying the one thing I said to try - which was remove one of the kafka publishers. But that's here nor there because that will be the last thing tried because it's my idea.

But because of this explanation, and the horrible code, something that should have taken about an hour or two - at the outside - is now sitting at a day and a half... and one of those days - yesterday - was a completely wasted day. Nothing was done on this because of the very bad code written by the developer asked to do it, and the fact that the other guys either didn't get to it, or wasted too much time before asking for my help (arrogance or marginalization - both are bad traits), and so a day is gone.

But this morning, I'm chatting with the author of the bad code, and he's thinking it's all good... that they refactored the code, and it's all happy skippy. But it's clearly not. And I'm chatting to him via IM, and not in person - at the office.

And I realize that it's times like these that I need to leave this team, and the management is right. If they are completely satisfied with this kind of work - then I'm not right for this team - at all. There was plenty of playing around yesterday... everyone but me took the lunch time off to go watch a talk... and there was no one in before 9:00 am - except me... and a day was wasted in the solution.

Yeah, this is not the kind of work I want to do.

This afternoon I'm forced to stay late because two guys in the group needed to have some ver bad clojure code explained to them because the guy that wrote it was off for the day, and the code he wrote was bad - to say the least. First, to be fair, this is clojure code, so it can be tough for some to understand, but these two guys aren't bad, so I don't think that was the issue.

No, the issue is that you can read clojure code a lot sooner than you can write good clojure code. For example, this was in the most convoluted function:

  (defn build-a-bridge
    [start-point end-point & [lanes type width]]
    (let [lanes (or lanes (/ width 20))
      ... ))

in a few short lines, he'd done so many bad things - that actually would work - that it made understanding this code very hard for these two guys.

Start with the re-valuation:

    (let [lanes (or lanes (/ width 20))

yes, it's possible, but in the real code, there was so much else going on it was hard to see this (the guys missed it), and so they thought that the lanes argument was nil, when in fact, it was being re-valued in the let. Yes, it's legal, but it's bad form, and the reason it was done is because the author really didn't understand optional arguments:

  (defn build-a-bridge
    [start-point end-point & [lanes type width]]

Here, the type of the bridge is non-optional - and so, really, are the number of lanes. Yet he was in so much of a hurry to get this done in the day, that he slapped things wherever he needed to to get them to work. Then, when the optional args were nil, he used the re-valueing to "make them right".

To call it hacks is being generous. It was a horrible way to use functional code, and it was all because he wanted to impress. Impress management that he's capable of getting the feature out in a day. In truth, it should have been an hour, but a day for him was within reasonable limits. But the time slipped by, and he started to panic.

I think it's fair to say that a panicked rookie clojure developer is bound to make some pretty bad code. And so I had to stay late to explain this code to the others in the team that couldn't figure it out for themselves.

And I'm the one that has to leave... Insanity.

Part of the New Direction that we are embarking on is to re-examine the topology that runs the data feed. Now I had built this, and done experiments covering months to get this to the point it was. Yet one of the managers talked to another Storm developer in another division, and that developer convinced him - without any knowledge of the specifics of the topology - that what I was doing was "All wrong". So my manager told me to work with this guy to fix it.

So I followed his advice and made the changes.

It didn't help, but it didn't make things worse.

Now today I'm being asked to revisit the topology and "Scale it back" because my manager is convinced there is something else in play here. OK, so I do as I'm told - because they are clearly not interesting in my opinions or we'd still be at the topology structure we had before all this.

So I start the experiments with a baseline, and then I start halving things: half the workers, half the bolts - all to get numbers to see if I'm going in the right direction. I don't fine-tune things until I'm really close, and half is easy to work with because we're still doing coarse tuning.

After about three experiments, we're far better, far faster, and have higher peak capacities during time of load. All is looking very good. Then I look at the Storm UI to the Capacity numbers, and I fine tune a little. This guy is a little high, so add some. This guy is very low, he doesn't need as much. Not a lot, just a little tweaking.

In the end we're looking a lot better from all the metrics. Good.

And the values?

Yup... just the ones I had before all this started.

I really am amazed about the real joy in using GitHub. It's hard to imagine the vision they had... Let's take git, and then build an entire culture and eco-system around it - all in the browser! That's some vision. Yet this morning I was able to put targeted comments on the lines of a pull-request with syntax-highlighted examples, and see it all in preview mode. It made me smile. This kind of attention to detail is really inspirational to me.

It's more than source control - it's workflow... collaboration - it's a wonderful framework with which to do group development - or personal development, for that matter. I really do enjoy working with it - even on really bad code. And the style... it's not always been perfect, and they have made some changes I might not agree with, but they have done it up to the nines. There is certainly no way someone is going to say they haven't sweated the details.

So even when I have to spend 30 mins making comments on code that should not have been written, I'm happy that I'm doing it in GitHub. What a great tool!

Today I've been just trying to keep things going because that's what I've been asked to do. Interestingly enough, the new management team that's out on the West Coast for a trip pressing the flesh with users this week is now convinced that everything we have been doing - and when I say "we" - they really mean me, is totally wrong.

They now believe that shared Storm as a service is the magic bullet and will make everything work perfectly. They believe my work has been lazy, and cryptic - at best, and incompetent at worst, and they now want to implement a plan to Clean Things Up.

Never mind that I have a plan to find out the issues plaguing us, and if they just started on that plan, I feel sure we'd see the solution... but that's me again - the lazy, shiftless, good-for-nothing that got us into this mess in the first place.

To do this, let's find out where every machine is in the data center, and then map all the switches and routers - not that we can change anything, but it's possible that this will explain the problem. Let's re-tune the topology because a manager has the feeling that it's wrong - never mind that he is that - a manager.

Let's do all these things that stem from the minds of the managers, and let's all do them Right Now! Drop everything... add five people... this has to be done now.

If I weren't in it, I'd laugh. It's a WTF of epic stature. Insanity.

This morning, my old friend Bret poked me with a stick to get me moving, and I fired up Xcode 6.1 and loaded up my old CryptoQuip solver, and gave it a run. The results brought a smile to my face - again:

  2014-11-10 08:43:42.536 CryptoQuip[87664:8603115] Solving puzzle: 'Fict O
       ncc bivteclnbklzn O lcpji ukl pt vzglcddp' where t=n
  2014-11-10 08:43:42.572 CryptoQuip[87664:8603115] Solution found: 'When I
       see thunderstorms I reach for an umbrella'

and the total run was only 36 msec. Very nice!

Then I decided to convert it to ARC because I know iOS requires ARC, and it should be even faster - right? I mean it's not doing the retain and release, so it should be even faster!

So I converted it to ARC, and removed all the retain and release and autorelease, and rebuilt it and re-ran it:

  2014-11-10 12:30:32.582 CryptoQuip[89789:8710573] Solving puzzle: 'Fict O
       ncc bivteclnbklzn O lcpji ukl pt vzglcddp' where t=n
  2014-11-10 12:30:32.640 CryptoQuip[89789:8710573] Solution found: 'When I
       see thunderstorms I reach for an umbrella'

So with ARC, it's taking 58 msec - that's a real increase. I had to run it several times to make sure that I was really measuring it properly. But there it was... as plain as day - 58 msec. Wow. ARC isn't cheap.

And as I started to think about it, it made sense - at least I came up with a plausible explanation: What if the retain/release version was using the standard Autorelease Pool. Then it wouldn't really hassle with deallocation in the runtime - it'd wait until the work was done to run the collector. However, ARC would see what needed to be done and then do it as it wouldn't wait for the pool to clean up. In that case, all the creation and destruction would come at a cost: Time.

In the end, it doesn't really matter - we have to use ARC for iOS, and it's not horrible for performance, but it does mean that I'm going to want to be careful as we move forward with this to not create all the temporary objects.

This morning I was chatting with Carl (that's not his name) - the guy that used to be my manager, but went to the West Coast, and now is looking to move back to Chicago... We were chatting about an idea he had - of using Salesforce.com as the source of data for sales projection algorithms. Then I remembered that Heroku got acquired by Salesforce and there's a specialized connect platform between Heroku and Salesforce for just this kind of scalable application building.

Heroku also handles Postgres as a Service, and they support clojure very nicely. In all, it sounded like a really nice platform to build this on. I can't way to see what Carl comes up with next.

Well... after the shock of yesterday, I have to admit, it was a very hard thing to get going today. I'm planning on taking the rest of the week as Work From Home, but even then, the crushing fear of being without a job, and having all the responsibilities of my little house - and Liza's and the kids... well... it was one of the worst starts on a day since those first few days in the hotel. It was just horrible.

But I had a feeling that I needed to get up - I needed to run, and I needed to pretend that my life was going to be OK. That I was going to pull through this - I didn't know how, but somehow, someway, I was going to make it. As my Mom was always fond of saying: The motion becomes the emotion - and I think she stole that from her Dad.

So I got up to run, and do my regular morning routine. It wasn't easy, but I tell you, I've never enjoyed Stan and Neil on SportsCenter as much as I did that morning. They helped get the day going. Then it was time to think... what to do? During the rest of my work-out, getting cleaned-up, and dressed, I worked through what I needed to do... the emails that needed writing, the people I needed to reach out to.

So I hit the morning going - emailed the last place that made me an offer, to see if it was still open (nope, not surprising). Then it was chatting with a few friends, pinging them about what's going on, and chatting to a friend that just left The Shop for a West Coast start-up. It was nice to talk to him... it's been a tough couple of years, and he's known me through almost all of it.

Then I chatted with a friend that I worked with a while back, and asked him about anything he's heard. He's still in Finance, and it's been two years, plus, since I've been there, and I do miss it. He said they are hiring, and that I'd be great for the job(s). So we set up some time to talk on Monday.

My hope picked up.

Then I saw that my former manager at The Shop - who left a few months before for another job on the West Coast, and he'd been fired yesterday, too. Wow... I told him I was sorry it happened, and he explained that the CEO got shaken when the company came under a little fire, and my friend was jettisoned. It felt like fate. Same day. Wow.

My hope picked up a little more.

I don't know what the rest of the day will bring - I fully expect to have dark times - I have no future now, so I'm going all on faith, but I have a few glimmers of hope, and if I can just hold on to those, maybe I can weather this storm, too.

Hope so.