What's it all about, Alfie?

As I was trying to test the demand from Dark Magic in the Quantum Lead app, I noticed that I was getting a nasty parsing error - the dreaded UTF-8 encoding. Argh! I looked up what I might be able to do, and the solution seemed pretty simple. In the clojure code where we make the response from Dark Magic, we needed to make sure to set the charset in the headers:

  (defn return-json
    [ob]
    {:status 200
     :headers {"Content-Type" "application/json; charset=UTF-8"}
     :body (json/generate-string ob)})

But still, when I ran this code, I didn't get the JSON decoding error in Ruby, but I also didn't get any data. There must be something wrong! And sure, enough, there was.

No dates.

The pipeline code expected to see a start_date and end_date in the demand set to know if this data is valid for the current running date/time. If not, then it doesn't use it. I looked at the code in Dark Magic, and sure enough - we were throwing away the dates we were getting from the original source. I guess my co-worker who wrote this didn't hear me say those were important. (Hint: he did, he's just not disciplined)

So today I figured out enough clojure to change the code and get all these things into the system. That means new fields in the database, new insert methods, extract methods, comparison changes. Lots of things had to change, but it was worth it, because when I was done, it was working with the encoding and all the dates.

It started out very frustrating, but ended up being a great learning experience.

I want to like clojure - I really do. Erlang was interesting, and it taught me a lot about functional programming - not that I'm really all that good now, but in Erlang, there were a lot of things that I found very hard. One was state - that was all in the arguments. The other was error messages. Man, those error dumps could go on for pages, and finding the first one that really was the source of the problem was a trick. But once you knew what to look for, and found it, then you had a fighting chance to figure out what you did wrong.

I'm learning that with clojure, it's not always that easy. After all, clojure is just java byte code, right? And with all the functional components written in classes in the compiler, it's easy to understand why they have classes that make no sense to a human reader.

OK, maybe to some human readers, but not to beginning clojure coders.

I came across this today and it's really pretty shocking.

Stack traces 50, 60, 100 lines long and not a clue in the lot of it where the problem is. Now I'm not saying it was hard to find… I've looked in more Java stack traces than I can count. I'm saying that every level in that stack trace was nothing in my code. Not a single thing. Certainly not by the unknown name munging that clojure uses.

So I'm left just commenting out sections of code hoping to find the ones that are causing the problem. In functional code, that's not easy as it's all nested within itself in the source file, and commenting out a section is really quite a little adventure to make sure that you get all the parens in the right places.

This is a serious problem in my mind. How long do you have to work with clojure to understand the name munging? Six months? A year? If that's the case, then there's no way a new developer on a team can do any fire-fighting, it's too stressful, and the only people to do it are the senior debs on the team.

That might not be bad, but our senior clojure dev is not a workaholic, so having him fire-fight anything before 9:00 am and after 5:00 pm is a dicey proposition. It makes me very nervous.

It makes me want to drop it and use something else.

Today I was able to get Dark Magic, our new Demand Machine (DM) that will be the place where our system goes to get the varied demands for the services that we'll mate up with the merchant info data from Salesforce. It's not a lot, right now. It's really only cacheing the data from our current only source of Demand, but it's now in our control. We don't have to worry about their system going down, or not having data, or having bad dates, etc. It's all in our hands.

True, it's only production now. I need to make sure my co-worker is done with the UAT machines, and then I can spin it up there as well. But hey… it's a big step! I'm glad it's running.

Today I did more work with the Demand Service - trying to get it to the point that it's reading in data from the external source and vending it out. Most of that time has been spent with the database schema trying to get it all worked out. For the most part, we're using korma, and that defines the tables and other DDE. So there was a lot of running the migration, checking the tables, making sure it'll hold what we need, talking about the effects, and repeating.

Not a bad day, and it needed to be done, but not exactly something you write a lot about, either.

I'm pretty sure I've ranted about this already, but once again today, it has been thrust in my face by the Code Monkeys that I work with. We start out the application design with a Hash. It's a wonderfully simple, flexible, data storage tool, and since we don't know all that we need, it makes far more sense to use it than enumerated ivars. So far, so good.

Then we pull in Hashie. This gives us "dotted methods" for all the keys in the hash. This includes the key? methods - one per key, that we can use as opposed to saying: m['key'].nil?. It's a win, and I love it. It does this without any restrictions or set-up on our part. I love it. So far, even better.

Then some in the group decide that we need to switch to a formatting/declarative version of the Hashie. Now if you want to use a value in the hash, you have to enumerate it in the class with a property keyword. All of a sudden, I'm not happy, and I say why I think this is a bad idea.

Up comes the ugly spectra of weak management and strong willed co-workers and Agile. They decide to re-write it even though we didn't all agree to it. Kinda bites, in my book. But I guess someday that'll work for me when I make unilateral decisions, and since I'm willing to work more hours, I'll get more of those opportunities. If I were a jerk.

So now, we have error messages and the code doesn't work because the incoming data source added a field and the code didn't. Had we left it alone, it'd be fine and we'd be working just as you expect. But because of this decision, we have production issues.

Where's the win, Boys?!

So I had to add some properties to the class just to make it not error. Yeah, this is good work guys.

One of the problems with Couch is that when you change or create a view in Couch, it has to rebuild/reindex the entire view by running the map function on all the documents in the database. This sounds very reasonable because like any database, it needs to maintain it's indexes, and this is how it does this.

The problem is that when you're doing this, the view is completely unavailable unless you want stale data. Not really ideal, but again, you can see why it's implemented this way. It's possible to see the old view, but that's stale, or you can wait for the new one. Your pick.

In order to make this easier on our environments, one of my co-workers came up with the idea that if you deploy the new view in a different document, and then after it's done being built, you rename it to the one you want, there's no second rebuild. The rename is nearly instant, and everything is OK. He built something so that when we deploy to the UAT and Production Couch DBs, we deploy in these "temp" spaces, and then there's a crontab job that sees if the rebuilds are done, and moves things in.

Well… that's great for UAT and Prod, but for dev, I don't want the cron job - I just want to have a direct-deploy scheme where I can wait the two minutes to rebuild my (much smaller) database. So I added that into the rake task, and was then able to deploy my changes to dev first, and see that they were working just fine, and then to deploy them to UAT and Prod and let them wait.

The reason for all this was that the views in the Pinnings design document were out of date - people had changed the code and not updated the views, so that they weren't picking up the right documents as they were supposed to. Just not disciplined about what they were doing, I suppose.

One of my co-workers brought up a bug that I hadn't noticed up to now - one of the test methods - that is, a method that returns a boolean about the Merchant argument, was not working, and there was a far simpler way to implement it. Basically, when we were dealing with Hashes in the data structures, it was as efficient a way to handle the problem - modulo the bug, as we could have. But now that we have the Merchant and within it, the Opportunity objects, with their own boolean methods, this implementation really became - ask the Merchant to ask it's Opportunity if it's a live deal.

Far simpler in the code.

Plus, at the same time, we're doing the right test because when I coded that one up, I did it right. Go figure.

In any case, a simple fix, and we reduced the lines of code. Not bad.

For the second morning in a row, I had issues with production. Thankfully, this bug didn't effect any of the production data - only that data that was supposed to be written to Couch. So in a sense, it was bad, but not worthy of a re-run. The bug was pretty simple, and for once, I'm glad it wasn't me. The original code was:

  def to_augmented_hash
    to_hash.delete('accounts').merge({
      :category_counts        => category_counts,
      :number_of_new_accounts => number_of_new_accounts
    })
  end

where we were getting a NilClass error on the merge call. I had remembered reading that the delete() method would return a nil under some cases, so I looked it up again, and sure enough - it only returns the value of the key removed. What needed to happen was:

  def to_augmented_hash
    to_hash.delete_if { |k,v| k == 'accounts' }.merge({
      :category_counts        => category_counts,
      :number_of_new_accounts => number_of_new_accounts
    })
  end

With this in, it all worked just fine. When I went to check it in, I saw that a co-worker also fixed this, but split it out on several lines, and mutated the value with the delete(). I decided to leave mine in as it was cleaner, and more to the way it would have been re-written by someone else anyway.

Today I've spent a good deal of time reading up on clojure and getting up to speed with my co-worker that's got several years of clojure experience on me. I knew going in it was going to be like this, and the learning curve is about what I expected. It's Java, under the hood, and that's got all kinds of pros and cons with it, but there are decent build tools, and I'm able to spend a lot of time getting the build/deployment environment up.

Today I also got the init.d scripts going so that we can start and stop it from a Makefile. Basically, all the things we can do from the rake files of the sister project in Ruby.

There were also issues with building PoestgreSQL 9.2.2, but those weren't hard to solve. Lots of nice progress today.

This morning we had a lot of warnings in the summaries with regards to the Merchant Status Reasons (MSRs). Salesforce exists on string fields, and while they make drop-downs that limit the applicable values, it's still a mess if something goes amiss, and we have unusual string values in fields. Such is the case this morning.

It turns out that our Salesforce team added several new merchant status reasons to the drop-down, and so now we were unable to know where in the sales cycle these merchants were. This is not good because it means that the only thing we can assume is that they are at the beginning, and that's clearly not right for all of them.

The solution was to learn what additions had been made, how they translate into our internal status codes, and put in the mappings so that tomorrow we will not fail. So that's what I had to do this morning.