What's it all about, Alfie?

I've spent many hours today getting the seasonality data into our clojure project. And I have to come away with the fact that it's a lot harder than it looks. A lot.

The problem for me was that the way this is being done in the clojure code is to have the entities defined in one file - but the tables created in another. Then all the insert/update as well as compare and load functions are in another file. There's a lot more to this than a simple set of INSERT statements. It's a lot more.

So I spent a good chunk of the afternoon trying to understand what I needed - based on the fact that I'd done a little of this with the demands and demand sets, but it's still a lot to do.

I hope to finish it tomorrow.

I have spent far more time with this problem today than it deserves. One of the real annoyances with clojure is also a touted strength - the JVM. But that's where I'm having the problems today - in the ugly world of the java.util.Date and it's ilk. When I receive a JSON snippet like the following:

  {
    "division": "cleveland",
    "start_date": "2012-12-07",
    "end_date": "2013-01-07",
    "demand" : [ … ]
  }

and need to place it into a simple PostgreSQL table:

  (:require [clojure.java.jdbc :as sql])
 
  (sql/create_table "demand_sets"
                    [:id :uuid "PRIMARY KEY DEFAULT uuid_generate_v4()"]
                    [:division_id :uuid "references divisions (id)"]
                    [:source_id :uuid "references sources (id)"]
                    [:valid_from :timestamp]
                    [:valid_to :timestamp]
                    [:loaded_at :timestamp "DEFAULT now()"])

One would expect that we'd convert the string into a valid Java Timestamp with something like clj-time:

  (defn get-demand-set
    [division]
    (let [demand-set (client/extract division)
          start-date (:start_date demand_set)
          end-date (:end_date demand_set)
          demands (:demand demand_set)]
      {:valid_from (to-date-time start-date)
       :valid_to (to-date-time end-date)
       :demand (map transform demands)}))

and, in fact, we get nice Java Date objects that are the correct time in the UTC (Zulu) time zone. This all makes sense.

Where it gets squirrley is the saving and reloading from a PostgreSQL database. The database table that was created in the above statement has the timezone in the fields, and there's a defined timezone for each instance of PostgreSQL - right there in the config file. I had my local database set to my local box - US/Central, and that was part of the problem. I wanted to see the dates in my database as I was seeing them in the JSON, and that's not realistic. I needed to understand that I'd really be looking at "2012-12-06 18:00:00" for the start-date, as that's the US/Central time of 2012-12-07 00:00:00 UTC.

Then I had to hassle with getting them out and comparing them correctly. This is really where this whole problem started - comparing dates where they formatted correctly, but were just off by a timezone or offset of some kind.

To do that, it turns out I needed to have something that would get me a single demand_set record:

  (defn find-demand-set
    [demand-set-id]
    (select-1 demand-sets
              (where {:id [= demand-set-id]})))

and here, we're using korma for the entity mapping and selecting. With this we can then get a demand set and extract out the Timestamp fields, but I need to make sure that I convert them properly:

  (defn fetch-demand-set
    [demand-set-id]
    (let [demand-set-rec (db/find-demand-set demand-set-id)
          ;; …more here
         ]
      {:valid_from (from-date (:valid_from demand-set-red))
       :valid_to (from-date (:valid_to demand-set-rec))
       ;; …more here
      }))

And then we can safely do something like this:

  (defn demand-sets-equal?
    [ds1 ds2]
    (and (= (:valid_from ds1) (:valid_from ds2))
         (= (:valid_to ds1) (:valid_to ds2))
         ;; …more stuff here
         ))

and the timestamps will properly compare.

But don't expect them to look like they did in the JSON unless you have the PostgreSQL databases set up on UTC time. Thankfully, that's what we have in the datacenter, so I don't have to worry about that. The only place I have to worry is on my laptop, and that's something I can live with for now.

There is one alternative I could have chosen. That's to have used the TIMESTAMP WITHOUT TIME ZONE field in the PostgreSQL table. I think that would have worked as well, and it would have shown the same value in all databases, regardless of the individual time zone setting. But then we would loose an important point in the time, and I wasn't ready to do that yet.

Still, to have spent several hours on this was exhausting

I want to like clojure - I really do. Erlang was interesting, and it taught me a lot about functional programming - not that I'm really all that good now, but in Erlang, there were a lot of things that I found very hard. One was state - that was all in the arguments. The other was error messages. Man, those error dumps could go on for pages, and finding the first one that really was the source of the problem was a trick. But once you knew what to look for, and found it, then you had a fighting chance to figure out what you did wrong.

I'm learning that with clojure, it's not always that easy. After all, clojure is just java byte code, right? And with all the functional components written in classes in the compiler, it's easy to understand why they have classes that make no sense to a human reader.

OK, maybe to some human readers, but not to beginning clojure coders.

I came across this today and it's really pretty shocking.

Stack traces 50, 60, 100 lines long and not a clue in the lot of it where the problem is. Now I'm not saying it was hard to find… I've looked in more Java stack traces than I can count. I'm saying that every level in that stack trace was nothing in my code. Not a single thing. Certainly not by the unknown name munging that clojure uses.

So I'm left just commenting out sections of code hoping to find the ones that are causing the problem. In functional code, that's not easy as it's all nested within itself in the source file, and commenting out a section is really quite a little adventure to make sure that you get all the parens in the right places.

This is a serious problem in my mind. How long do you have to work with clojure to understand the name munging? Six months? A year? If that's the case, then there's no way a new developer on a team can do any fire-fighting, it's too stressful, and the only people to do it are the senior debs on the team.

That might not be bad, but our senior clojure dev is not a workaholic, so having him fire-fight anything before 9:00 am and after 5:00 pm is a dicey proposition. It makes me very nervous.

It makes me want to drop it and use something else.

One of the problems with Couch is that when you change or create a view in Couch, it has to rebuild/reindex the entire view by running the map function on all the documents in the database. This sounds very reasonable because like any database, it needs to maintain it's indexes, and this is how it does this.

The problem is that when you're doing this, the view is completely unavailable unless you want stale data. Not really ideal, but again, you can see why it's implemented this way. It's possible to see the old view, but that's stale, or you can wait for the new one. Your pick.

In order to make this easier on our environments, one of my co-workers came up with the idea that if you deploy the new view in a different document, and then after it's done being built, you rename it to the one you want, there's no second rebuild. The rename is nearly instant, and everything is OK. He built something so that when we deploy to the UAT and Production Couch DBs, we deploy in these "temp" spaces, and then there's a crontab job that sees if the rebuilds are done, and moves things in.

Well… that's great for UAT and Prod, but for dev, I don't want the cron job - I just want to have a direct-deploy scheme where I can wait the two minutes to rebuild my (much smaller) database. So I added that into the rake task, and was then able to deploy my changes to dev first, and see that they were working just fine, and then to deploy them to UAT and Prod and let them wait.

The reason for all this was that the views in the Pinnings design document were out of date - people had changed the code and not updated the views, so that they weren't picking up the right documents as they were supposed to. Just not disciplined about what they were doing, I suppose.

This afternoon the Google Chrome team released 25.0.1354.0 to the dev track. Unfortunately, the release notes are more than terse, and of no help to someone at all. I wonder why they even bother?

I needed to find out how to list all the active queries in PostgreSQL, so I did a little googling, and found the following:

  SELECT * FROM pg_stat_activity;

and the response comes back with all the activity you can imagine. I'm pretty sure it's by the connection, as a lot of my connections are 'idle', but that's great! This gives me the knowledge of what's going on in the database in case a query it taking forever.

This morning I noticed that Google Chrome dev is now up to 25.0.1349.2 with another nearly useless set of release notes. While I like that they provide a link to the SVN log, that's not really release notes by any stretch of the imagination. I write very nice commit messages, but even I know those aren't anything like release notes. They're far too detailed. I wish they'd write release notes and not just reference the SVN logs.

This morning, once again, I had about 500+ error messages from the production run last night. It all pointed to the JSON decoding - again, but this time I was ready: the fail-fast nature of the script now didn't try to do anything else, and I could retry them this morning. So I did.

Interestingly, just as with the tests yesterday, when I run it from the login, it all works just fine. So I fired off the complete nightly run and then set about trying to see what about the crontab setup on these new boxes was messed up and didn't allow the code to run properly. Thankfully, based on yesterday's runs, I know I could get them all done before the start of the day.

So when I started digging, I noticed this in the logs:

  Input length = 1 (Encoding::UndefinedConversionError)
    org.jruby.RubyString:7508:in 'encode'
    json/ext/Parser.java:175:in 'initialize'
    json/ext/Parser.java:151:in 'new'
    ...

so I did a little googling and it brought me back to encodings - what I expected. Which reminded me of this issue I had with reading the seasonality data in the first place. Then I looked at our code, and we are using a standard reader method to get data for both CSV and JSON:

  def self.read_file(filename)
    contents = ''
    what = project_root + '/' + filename
    File.open(what) do |file|
      contents = file.read
    end
    contents
  end

which is all very standard stuff.

What the hits on google were saying was that I needed to think about the encodings, and so I changed the code to read in iso-8859-1 and then transcode it to utf-8:

  def self.read_file(filename)
    contents = ''
    what = project_root + '/' + filename
    File.open(what, 'r:iso-8859-1') do |file|
      contents = file.read
    end
    contents.encode('utf-8', 'iso-8859-1')
  end

Then I saw in another post about encodings in ruby, that I could collapse this into one step:

  def self.read_file(filename)
    contents = ''
    what = project_root + '/' + filename
    File.open(what, 'r:iso-8859-1:utf-8') do |file|
      contents = file.read
    end
    contents
  end

which simplifies the code as well as the understanding: The file is iso-8859-1, but I want utf-8. Perfect! I put this in and I should be good to go.

But the question is really then: Why does the login shell work? After all, if they both failed, that would make sense. But they both don't. That got me looking in the direction of what's defined in the login shell that's not in the crontab pseudo-shell. As soon as I scanned the output, it was clear:

  LANG=en_US.UTF-8

and that explained everything.

The crontab 'shell' doesn't define this, and you can't put it in the crontab file like you can the SHELL and MAILTO variables. So the solution was simple: put it in my main script right after the PATH specification:

  export LANG="en_US.UTF-8"

and all the problems should just go away! That would be nice. I'll have to check when the runs are finished this morning.

In the beginning of this project, we created a very simple bash script to run the jobs we needed run in a crontab. It's just a lot easier, I've found, to run things out of a simple bash script than try and put them in the crontab itself. The crontab just looks cleaner, and it's not a real shell, so it's just better all-around.

But as the project got more complex, it was clear that I was beginning to test the limits of what could be easily done in bash. The problem then, was the fact that a vocal contingent of the guys on this project don't really know bash - and have no desire to learn it. Interestingly enough, their argument for using complex and difficult things like meta-programming in ruby is that there's a "floor full of people that understand it". But when bash comes up, it's not even really checked against that same "floor full of people" to see if they know it as well.

It's Code Monkeys, what can you say?

Anyway, as things progressed, I needed to have a way to simply run many jobs at the same time, but ensure that all jobs of a single kind are done before moving on to the next phase of processing. The solution I came up with was pretty straightforward, but not exactly very efficient. The idea was to have a loop and start n background processes, and then wait for all them to finish before continuing the looping and starting more.

This is a pretty simple but it means that the speed with which we can process things is determined by the slowest (or longest running) job in the batch. Therefore, a relatively small number of well placed jobs in the queue can really spread things out.

While this has been OK for a few weeks, we really needed something cleaner, so I came up with this far simpler plan:

  function worker {
    for i in $list; do
      if [ lock($i) ]; then
        do_work($i)
      fi
    done
  }

The idea being that if I launch n of these with a simple:

  for (( i=0; i<n; i++ )); do
    ( worker ) &
  done
  wait

then we'll have these workers running through the list of things to do - each picking up the next available job, and doing it, but skipping those that have been locked by the other workers.

Really pretty slick. The trick was finding out that mkdir is atomic, so it's simple to use that to make the directory tagging the process, and if we are able to make it, then we have to do the work, and if we can't, then someone else is, or has, done the work.

This is super cool!

I was thinking I'd need a queue, or something, and all I really needed was a list and a filesystem. That's sweet. Really. That's one of the coolest things I've seen in a long time.

Interestingly enough, the code is now a lot simpler:

I still need to test it all, and that will be Monday, but there's no reason to think it won't work, and this way, we have n workers all doing the best they can until all the work that's needed to be done is done, and then everything stops and we can move on to the next phase.

Very cool!

This morning I was able to finish up the re-write of the summary script and I was very pleased with the results: the processing of the pipeline log dropped from 4+ min to less than 5 sec - even with jruby, and the other two are in the sub-2 sec range. The latter are really dominated by the jruby startup time, and if we can move to an RMI ruby in deployment, that will help here too.

In short - fantastic success! Now I need to come up with a better queueing and processing scheme in bash - or re-write that in ruby as well…