What's it all about, Alfie?

I told myself this morning that I didn't want to get upset at the folks I work with any more. I told myself it's a bad thing to do - and it never accomplishes my goal of actually getting something done. Never.

Yet here I find myself, some four hours later, fighting very hard not to be nasty to one of the Code Monkeys when he gets in. The story goes like this:

We were looking into a problem late yesterday, and I believed it to be possible that the choice of Rack web server had something to do with it. We were running 'thin' in development, but 'unicorn' in production. So this morning, I wanted to move development to 'unicorn', and see if I could reproduce the problem in development.

I look at the git repo and see that another developer had made these changes already. Well… some of the changes. He didn't remove the 'thin' gem from the gemset, and that needed to be done. And he didn't document how to start the 'unicorn' server other than the fact that there's a Procfile in the root of the project.

There is a place on the GitHub README.md for how to start this, and he hadn't changed a thing. So I figured a little googling might help out.

The only thing I could find was that a gem called 'foreman' uses a Procfile and it looks about the same, but we don't have 'foreman' in the Gemfile, so I can't see how he's using it. Unless he installed it globally and didn't update the docs to reflect that.

And, of course, he's not in and it's 8:40 am.

It's this kind of thing that really drives me crazy. They make a change - commit to master and don't update the docs. They either don't think it's important, or that it's clearly obvious, or that they are just plain forgetful. But for whatever the lack of discipline, they are now blocking me from working. That's pretty annoying.

If it were 10:00 pm, and I wanted to do a little late-night coding, I could see that. It's off-hours, and he's got time to get it ready. But they all know that we have the rules about master, and they all know when I get in. This is just having no discipline about what you're doing.

It's very hard for me to maintain a calm exterior when I see this around me all the time - every day. I don't think a one of these Code Monkeys takes this project seriously, and maybe that's OK for them. But when I'm constantly having to bust my hump for deadlines for this entire group, and getting so little help as this, it's really hard to not get angry at them.

This isn't calming me down… I had hoped that it would, but it's not. Gotta just stop writing then...

The large part of today has been spent trying to add in the closed deal adjustment to the demand to make Dark Magic on feature-parity with the code in Quantum Lead. The idea is that we needed to look at all the deals that have closed after the raw demand was generated, and then look at how much of them sold, or at least what their sales expectations are. This is then subtracted from the demand units requested as it represents fulfilled demand in the time span of the demand's life.

The code was all written in ruby, and worked fine, but was heavily dependent on the fact that (j)ruby uses references and mutable objects as it had multiple passes of the adjustment based on the individual deal options that had closed. In order to refactor this into something that works in the immutable space of essential clojure (which is how I'm calling the design philosophy of my co-worker that's a clojure purist) I spent lots and lots of time trying to figure out how to put this into that primitive functional style.

I had learned that there were many things in clojure that could make this very simple. There's transactional memory for mutability, and references, so that - in theory, it could have been a very simple port of the code. But Socrates, as I'll call my co-worker with the purist attitudes, wouldn't have any of this. Since this is being done (against my better judgement) to make him happy, that's what we're doing.

I wonder if he has any idea why this is all really happening? Maybe he just thinks we all "see the light"? Who knows. I'm half hoping that this fails and we do it again in ruby, and half succeeds so I can have guaranteed job security as Socrates will be bored with all this long before we're done with the system we're building.

In the end, I was stuck on the one problem, and I had to sit and wait for about 20 mins as Socrates came back from his walkabout - wherever that took him, and have him show me how to refactor this ruby code into something that works in clojure.

Functional languages are nice. But I'm hating this experience because I feel I'm in the middle of an ocean, and my only life raft is a guy that doesn't care about jack-diddly, and comes and goes as he pleases. It's a very uncomfortable position to be in. Maybe in six months when I know clojure well enough not to have these blocks, it'll be OK… but now, it's just exceptionally frustrating to think that this company is making a business decision to use this new language, and then forcing me to be on this project because they want it to succeed.

Very frustrating.

This morning I noticed that the crontab job I had to cleaning up the stale, unused jars that happen when you have a project in jruby or clojrue where everything is in a single jar, and they are all tagged with the GitHub SHA to tell them apart. Actually, it's about the nicest thing about jruby and clojure because it means that you don't have to hassle with directories of files, and it's super easy to have a symlink to the current version and then de-reference it in the script with a little:

  jar="$HOME/dark-magic-current.jar"
  actual_jar=`readlink $jar`

and then in the rest of the code for running the app, use $actual_jar and you can deploy on top of a running system without disturbing it. Restart, the script will pick up the new symlink, and you're in business. Pretty nice, actually.

Anyway, the bug was in the vacuum script:

line 26 used to read:

  for i in "${oldJars}"; do

and I learned that because oldJars was a variable, what I was doing by quoting it was to make that one value for the loop - and not a series like I had hoped. Simply removing the quotes took care of the problem as bash is smart enough not to have thought the space-delimited names in the variable were actually separate commands.

Bash is smarter than I thought. Nice.

Simple fix, and now we are checking all the jars for expiration.

I spent most of the day working on the docs for the API and logic for the clojure project I'm working on. This all fits in the README.md in the root of the github repo so that it's rendered nicely on the GitHub pages. It's nice to have something there, as it allows everyone to know what the goals are for the next few updates/releases and gives them talking points in case they want to ask some questions and don't want to start by asking the simple questions.

I used OmniGraph Sketcher to make the graphs, as it seemed to me to have the best toolset for the job, and it worked out quite nicely. Really came through for me. I then threw in some JSON from the actual service to make sure it was the right format, and started typing.

The biggest part of all this is keeping things straight and not overusing the terms. There just aren't that many terms for 'demand' and 'inventory', so I had to be a little creative and careful about how I wrote it all up. It's certainly not perfect, but it's a far far cry better than nothing, and I think I'm pretty decent at documentation when I get going. So there.

Glad to get that all done so that we don't get hammered by questions from project managers.

I spent the entire day doing something I'd hoped would have only taken about an hour this morning. I wanted to get the seasonal adjustment code done, and then test and be finished! But that's not how it turns out, is it? I spent a lot of time fighting with clojure, and more time fighting with how it was being used by the architect of the project I'm on.

The biggest thing about clojue today was the use of the apply function. I hadn't really used it before, but I had a sequence of 12-element sequences and I wanted to make a new sequence - call it a transpose if you want to borrow from linear algebra, but basically it's another array, but this time, it's a sequence of 12 elements - each with a size of 'n' represented by the number of sequences in the original data.

I needed this for computing the maximum factor for each of the months based on a series of yearly factor sequences.

I found something that looked like it'd work:

  (def factors '((1 2 3) (4 5 6) (7 8 9)))
  (apply map vector factors)
  => ((1 4 7) (2 5 8) (3 6 9))

so I used it and was happy that I found it. But my happiness faded when I got a clojure error message saying that somewhere in my code, the map function was getting only one argument - and that was an error. I looked and looked, and could not find the problem.

I chatted with my teammate and he pointed out that if factors is empty, then map will have no arguments, and that's the error. He used this saying "all languages" do this with varargs.

OK… let me get this right… if I have a non-nil, but empty factors, then this fails? Yup, it does. I was pretty pissed at clojure for this. First, for not handling edge conditions better than this, and secondly for the error message that could have been far far more useful than it was.

OK, to be fair, I can see his point. They don't want to have to make exceptions for things like:

  (apply + ())

and I was using something I'd never used before. My bad. I should have known better.

But the guy that's supposed to be the "clojure horse" of the group is a guy that barely works 40 hrs a week and is on a three-day trip to talk at a conference. Bully for him, but that doesn't leave me anything to do if I don't venture out on my own. And if I do, I'm going to get burned like this, and it's going to piss me off.

Period.

When I know a lot more about clojure, I'll be more comfortable and I won't mind his walkabouts to wherever. But while I'm trying to make progress, it's a real problem for me. But it's something I can't say anything about because I'm not his manager. Yippee.

The other big issue was the circular reference problem in clojure. I really can't believe there's no solution to that. I've gotta hit google for that tomorrow.

This morning I finally finished up the problem of the demand time series time-shifting: a simple left-shift of the sequence:

  (defn shift-left
    "Do a simple left-shift for the sequence of 'n' items"
    [coll n]
    (seq (into (vec (drop n coll)) (vec (take n coll)))))

and this works wonderfully:

  (def a '(1 2 3 4 5 6))
  (shift-left a 3)
  => (4 5 6 1 2 3)

Nice little function to have. Interestingly, the way this works is entirely different than a very similar function:

  (defn shift-left-no-vec
    "Do a simple left-shift for the sequence of 'n' items"
    [coll n]
    (into (drop n coll) (take n coll)))

This guy doesn't convert the drop and take sequences to vectors, and the result is interestingly different:

  (def a '(1 2 3 4 5 6))
  (shift-left-no-vec a 3)
  => (4 5 6 3 2 1)

seems the into function when operating on a sequence pulls from the end and not the beginning, or something odd. The drop and take looked good in the REPL, but there's something about these sequences that messes things up. So I used vectors and then made it back into a sequence at the end.

When I tried to test this in the larger system I realized that there were tons of problems in the code that had been written by my teammate. There were poorly thought out arty functions for updating the data - not hard to fix, but it took a little time. There were really poorly designed insert methods that wouldn't properly strip out the existing id values from a source data set for it to be inserted as a new data set without errors - again, not too hard to fix, but it was thoughtless.

Then there's the one that probably bugs me the most because it's brought about by clojure the language and the clojure guru that's really setting the design and namespaces for the project. Circular references. Clojure can't deal with them.

OK, if you're in a single namespace, you can say:

  (declare foo)

and then use it and later in the same namespace say:

  (defn foo
    [x]
    (* x x))

but if you need to have two methods in two namespaces, and there are require directives set up to have one include the other, then you're hosed. No way around it.

Now I'm no language designed - OK, I did it long ago, and it wasn't nearly this complex, but the point is this seems like a horrible oversight. Unless, the assumption I'm working under is faulty. Maybe the point is that the designers didn't expect to have 50 namespaces like we have. Maybe they expected 5. In that case, this all goes away. If I were to collapse a few namespaces, that would fix things up.

But then my clojure guru would not approve.

This is what's really beginning to bug me - good languages like ruby spoiled by the tubists culture. Ditto here. Don't be a math prude - don't try to design the namespaces before you know you need them. Arguably, in a C++ project, I'd lay out directories as I knew I'd need them - but only a minimal set. This is something that just goes to bad design, and since I'm a total newbie, this is on my guru. He didn't even see it coming.

I want to learn clojure, and ruby, and then use them as I'd use C++ - with the features I see I need, and how I'd use it. But I'm getting the feeling that that's not how this is going. I'm being shown a limited set of the features in clojure and that's all he wants to use. I get it. I'm not sure I'd want someone to be doing template meta-programming in their first C++ project, but it's still a little unsettling.

For most of the day I've been working on taking the fixed demand from a source and making it a year-long demand time series. It's not done, but it's close, and I've been testing the code along the way. The idea is pretty simple: the source of demand we're forced to work with is a single value given to us over a specific valid time window. That's it:

  { 'start_date': '2012-12-07',
    'end_date': '2013-02-07',
    'units': 1000,
    'service': 'Skiing' }

of course there are locational components and taxonomy is a little more complicated than I've made it out to be, but for what we're doing, it doesn't matter.

On top of this, we have a source of Seasonality Factors that we have been given by the city planners and regional VPs. These are how they see the demand changing over the course of a year for a given service:

  { 'cleveland': { 'Skiing': [200 150 0 0 0 0 0 0 100 100 150 200],
                   'Ziplining': [0 0 100 100 100 100 150 150 150 150 100 50] } }

Here, we have two services for Cleveland - Skiing and Ziplining. The data shows that the factors (an array of integers representing the 12 months in the year) are high for Skiing in the cold months, and opposite that for Ziplining. Not a big surprise.

What I needed to get done today - and almost made it, was to take these factors and the raw demand and make a function that converted the initial demand data to something like this:

  { 'start_date': '2012-12-07',
    'end_date': '2013-02-07',
    'units': [2000 1500 0 0 0 0 0 0 1000 1000 1500 2000],
    'service': 'Skiing' }

where the fixed 1000 units are multiplied by the factors (given in percentage) and expanded into a nice array. Interesting to note - we've got to do one more thing: when we have calculated the data like this, we need to rotate the time series of units to make sure that the current month is in the first position and the remaining 11 are in "the future" representing the next 11 months of demand.

It's this last part that got me today. I couldn't get it figured out. But I will.

This morning we had s significant production (and UAT) problem caused by a group that's not nearly as together as I'd like them to be for a critical source of data for the project I'm on. The are currently the source of demand for the system, and that's used to find matching merchants and asses the potential value of each matched merchant in order to enable us to rank them for the sales reps. It's really one of the two key data sets we need to run.

Interestingly enough, when they created this data set, they had the foresight to include a start_date and an end_date in the data set so that we could tell when the data was generated, and how long we were to consider it 'valid':

  { 'start_date': '2012-12-07',
    'end_date': '2013-01-07' }

but as it's clear from my example, today was one day past expiration! This means that when the code ran, it saw the data, and it accepted it, but it realized that it was expired, and so it didn't use it. Lovely.

The solution was pretty simple. Since we have a new system that deals with the demand from this source and puts it into a nice PostgreSQL database, we could simply go to the psql console and say:

  UPDATE demand_sets
     SET valid_to='2013-02-07'
   WHERE valid_to='2013-01-07';

and I bought them a month.

I could then re-run everything and a mere couple of hours later, everything was fine. Once again proving that Problems are solved by people that show up.

I thought I had it all figured out but then a little later in the day it hit me: When we go to reload the data from the "stale" source, we'll see that it's different, and assume that "different" means "newer" and we'd overwrite the data I just updated with something that was clearly going to fail again. Not good.

So I realized that I needed a real solution.

What I realized was that there's no reason to make the insert code more complex. I can look at the reader code from the API endpoint and see if I'm getting data that's clearly expired. It's right there I can fix things up nicely. Then I started thinking that my original solution was a nice start, I just needed to formalize it in the code. So I started with a simple function in the app's until namespace:

  (use 'clj-time.core)
 
  (defn leap-frog-date
    "Looks at a date to see if it's in the past, if so, add a number of
    months to the date until it's in the future and return that."
    [d]
    (let [ts (now)]
      (cond
        (after? ts d) (plus d (months (inc (in-months (interval d ts)))))
        :else d)))

this will get me a nice way to filter the end_date before it becomes the valid_to in the database.

But I wanted to add in a little logging as well, but it didn't belong in this general function. So in the importing namespace, I simply had a private method with a side-effect:

  (use 'clojure.tools.logging)
 
  (defn- leap-frog-date!
    [d]
    (let [d' (leap-frog-date d)]
      (if-not (= d d')
        (error "Had to move expired dates: %tF to %tF" (to-date d) (to-date d')))
      d'))

At this point I'm ready to go. Every month that they miss regenerating the data, I'll detect this, log it in my logs (for easy detection) and then update the expiration date so that we use the old data anyway.

It's not ideal, but it's such an important part of the system we can't afford to just not run because they can't get their act together. At the same time, I was pretty happy with the clojure tools in the clj-time and clojure.tools.logging packages as they really made this a lot nicer than if I had to do this all myself.

Today I struggled trying to get the same keep-alive going in out production server, but because it's running Unicorn and not Thin, and Unicorn is not an event-driven server, it was a lot harder. To be fair, a lot of this had to do with the problems I was having with the deployment of software to the boxes. The Shop has a lot of progressive tools, but the deployment of packages is not among them. So there were several hours lost there.

But the real problem was figuring out how to integrate an EventMachine into the Unicorn server. Thankfully, I was able to find a simple way to do it:

  Thread.new { EventMachine.run }

So simple, but so completely non-obvious.

With that, it started working and I just continued to struggle with the fact that Unicorn doesn't expose a logger like Sinatra/Thin.

These are not the easy tools that folks make them out to be.

Today I've been working very hard to get this ruby app (Sinatra) to have a keep-alive with requests that take a very long time to complete. Typically, you might think this was a case for streaming the data, and in some cases, that's true. But in many of the cases I'm dealing with, the server has to marshall a lot of data, piece it together, and then format the output. It's really not ready until it's all ready. So I needed a simple keep-alive.

Back in a previous job, I used Tomcat and just had a thread that sent a space every 10 seconds to the client until I stopped it. I then put this in the rendering code, and all of a sudden, everyone got this keep-alive code, and no one had to worry about a thing. I know it's not going to be that easy this time, but it's not that far off.

First, we need to make sure that we're using the 'Thin' server in Sinatra. WEBrick doesn't support an event model, and so won't be starting an instance of EventMachine. This is thankfully pretty easy to do, just require 'thin', and backup will detect it and we're good to go:

  require 'thin'

Next, we need to put the keep-alive in the individual calls. This one, for instance, takes a while to complete:

  get "/call_list_spreadsheet/:dbname/:extag.tsv" do
    content_type('text/tab-separated-values')
    attachment("call_list-#{params[:dbname]}-#{params[:extag]}.tsv")
    stream do |out|
      timer = EventMachine.add_periodic_timer(10) do
        out.write("\0")
      end
      out.write(CallListTSV.new(params[:dbname], params[:extag]).render)
      timer.cancel
      out.flush
    end
  end

the key component here is that little bit in the middle:

      timer = EventMachine.add_periodic_timer(10) do
        out.write("\0")
      end

Simply put, that's going to send a NULL to the client every 10 sec. until the timer is cancelled. This works like a charm to do what I need. Amazingly so. I was really quite pleased.

All this is in a nice little gist.