Archive for January, 2013

Cool Sub-Selects in korma

Thursday, January 31st, 2013

Clojure.jpg

I was doing some selects from a postgres database into clojure using korma, and they were pretty straight-forward for a master/detail scheme:

  (defn locations-for-demand-set
    [demand-set-id]
    (select locations
            (with demands)
            (fields "locations.*")
            (where {:demands.demand_set_id demand-set-id})))

and it was working pretty well. The data was coming back from the database, and everything was OK. But as the database got bigger and bigger, we started to see a real performance penalty. Specifically, the pulling of the locations was taking on the order of 30 seconds for a given demand set. That's too long.

The problem, of course, is that this is implemented as a join, and that's not going to be very fast. What's faster, is a sub-select where we can get all the demand ids for a given demand-set, and then use that with an IN clause in SQL. Thankfully, I noticed that korma had just that capability:

  (select locations
    (where {:demand_id [in (subselect demands
                                      (fields :id)
                                      (where {:demand_set_id demand-set-id})]}))

Unfortunately, this didn't really give me the kind of speed boost I was hoping for. In fact, it only cut off about a half-second of the 31 sec runtime. Kinda disappointing. But the fact had to be related to the size of the sub-select. It was likely 25,000 elements, and doing an IN on that was clearly an expensive operation.

I like that korma supports this feature, but I need a faster way.

Horribly Slow Time Machine Backups

Wednesday, January 30th, 2013

TimeMachine.jpg

Yesterday evening - as I was packing up to leave work, I noticed that my Time Machine backup hadn't completed. True, there was a new Xcode release yesterday, and that was bound to make the first backup after that large, but still… to have it take a full day and not be done? Well… I had to pack up, but this morning I wanted to get to the bottom of this and figure it out.

What I found was that the backup was working, but horribly slowly. I started googling, and it appeared that the convict is between Spotlight's indexing and Time Machine's backing up. Since I'm not a huge fan of Spotlight (I use LaunchBar, and always have) - I was ready to turn it off. But as I read on, I learned that wasn't the best idea either.

What it seems was the best alternative was to start fresh with Spotlight. SImple wipe out it's cache, reboot, and let it churn for about an hour rebuilding the indexes, and then the two would peaceably co-exist. So to that end, I said:

  $ sudo rm -rf /.Spotlight-V100/*

then a quick reboot, and about an hour of indexing later, Spotlight was back in the saddle. Time Machine then backed up it's 6+ GB very quickly after that, and things were fine.

Sometimes, it takes a little help, but this technology is about the most wonderfully designed stuff I've ever had the pleasure to work with. Amazing.

Xcode 4.6 is Out

Tuesday, January 29th, 2013

xcode.jpg

This morning I saw that Xcode 4.6 was out, and so I had to get it for my laptop. It included an update to the command line tools, as you'd expect, and the release notes indicate that they have pulled in all the Cx11 additions for C++, which - to me, is a great win. This includes the move semantics which makes a lot of the code I've written just a while lot more memory efficient - sweet!

There are also updated to the rest of the tools, and I was thinking, as I was updating just now, that Xcode has been my very favorite development environment of all time. It's just wonderful. Complex, yes. Hard to find things, certainly. But in the end, an amazingly powerful tool.

Hitting Teradata from Clojure

Monday, January 28th, 2013

Clojure.jpg

Today I worked on hitting Teradata from within clojure using clojure.java.jdbc, and I have to say it wasn't that bad. There are plenty of places that a few paragraphs of documentation could have saved me 30 mins or so, but all told, the delays due to googling weren't all that bad, and in the end I was able to get the requests working, and that's the most important part. I wanted to write it down because it's hard enough that it's not something I'll keep in memory, but it's not horrible.

First, set up the config for the parameters for the Teradata JDBC connection. I have a resources/ directory with a config.clj file in it that's read on startup. The contents of it are: (at least in part)

  {:teradata {:classname "com.teradata.jdbc.TeraDriver"
              :subprotocol "teradata"
              :subname "//tdwa"
              :user "me"
              :password "secret"}}

Then, because we're using Leiningen, the jars are loaded in with the following added to the project.clj file:

    [com.teradata/terajdbc4 "14.00.00.13"]
    [com.teradata/tdgssconfig "14.00.00.13"]

so that the next time we run leon, we'll get the jars, and they will know how to connect to the datasource.

Then I can simply make a function that hits the source:

  (defn hit-teradata
    ""
    [arg1 arg2]
    (let [info (cfg/config :teradata)]
      (sql/with-connection info
        (sql/with-query-results rows
          ["select one, two from table where arg1 = ? and arg2 = ?" arg1 arg2]
          rows))))

Sure, the example is simplistic, but it works, and you get the idea. It's really in the config and jar referencing that I spent the most time. Once I had that working, the rest was simple JDBC within clojure.

Learning to Code Like a Rubist

Thursday, January 24th, 2013

Code Monkeys

Today I spent some time fixing a bug, and while I was in the code, I wanted to see if my manager was right - that I was writing code that no one on the team liked working with. After all, I'm new to writing in ruby, and there's a unique style, I've come to learn, and I wanted to see if I've been doing it right, or maybe not right enough. I had something like this in my code:

  if meetings.empty?
    ru = fills * misses / factor
    fu = (1.0 / ru) * fill[0]
  else
    ru = fills.map { |f| f * misses / factor }
    fu = ru.map { |r| 1.0 * fill[0] / r }
  end
 
  all_zips = location.map { |l| l['zip_code'] }.compact.uniq
  {
    :ru => ru,
    :fu => fu
  }

Now, in truth, this makes no sense and I had about 10 lines in both branches of the 'if' statement, but the gist was that if the data was a constant, do one set of calculations, and if it was a vector, then do a slightly different set, but along the same lines as the single-valued calculations.

Nothing complex.

But it seems that The Ruby Way is to make anything that's more than a line or two into a new function - even if it's significantly expanding the lines-of-code in the file. So contrary to my prevailing understanding of the reasons for this extreme composition, it's really about the complexity of the code.

I was talking to a consultant new to the group, and his statement was that he trusted the code to do what it said it was doing, and by making the methods very small, and well-defined names, it was easy to understand what they were doing.

I asked him about debugging, and his statement was again about trust.

I was shocked. On so many levels.

First, that trust had anything to do with debugging. Second, that they felt this code was too complex, and required refactoring.

It's a few lines - say 10, and it's a series of calculations. There's no savings in making it functions. It's adding stack calls, tests on the input values, and re-testing the values that caused the branch in the first place. It was horribly less efficient than before, but that's what they wanted.

It's back to the "CPU cycles are Cheap - Dev cycles are Expensive" - Hogwash that I've heard before, and don't agree with in the slightest bit. Nothing is cheap. Everything is expensive, and it's all about balancing the costs of different factors into the final design. There's no free lunch, but it seems that fact hasn't found the Rubists yet.

Secondly, you never trust code in the debugging. You 'single-step' all the code when you have a bug to find out where it lies. If you trust code, you're going to get burned. Guaranteed. Debugging is a thoughtful exercise. You have to think what should happen, and then watch and see exactly what does. This means you can't take anything for granted. Not a single thing. That they would makes me very nervous that "fixes" on their part aren't going to be fixes at all. They'll patch or "refactor" until the bug goes away, but that's not the same as really fixing it. It might just pop up somewhere else.

Finally, I can't imagine that I'm going to be able to write code like this long-term. I don't see it as overly-complex, so I'm not going to think that it needs refactoring. If I take the absurd approach that every calculation is a method, I know I'll be wrong on the other extreme, so I can't just pretend to know what to do.

I'm frustrated, and I feel like giving up. It's been a hard day, and by all measures I seem to be incapable of getting the right mindset to write this code like the rest of the team. They aren't happy with my code, and they aren't willing to put in the hours to write it before I do. So they are going to complain about it, and I'm going to be powerless to understand their mindset properly to write code like they would.

I wish I knew what I could do to fix this problem…

Adding More Metadata to the Demand Service

Thursday, January 24th, 2013

Dark Magic Demand

This morning I was asked to add something to the demand service so that it'd be easier debugging what's happening within that service. It wasn't horrible, and given that we already had the provision for the meta data associated with the demand, it was simply a matter of collecting the data and then placing it in the map at the right time.

I was really pretty pleasantly surprised to see how easy this was in clojure. Since everything is a simple data structure (as we're using it), it's pretty easy to change a single integer to a list, and add a value to the end. Then it's just important to remember what's what when you start to use this.

Placed it into the functions, deployed the code and we should be collecting the data on the next update. Good news.

Things are Getting Complicated it Seems

Wednesday, January 23rd, 2013

I was in a meeting today - talking about an interview that was coming up today and the guy looking to make this hire was very explicit about the type of person he was looking to get: Type A. Serious go-getter. You know… me. OK… people like me - and he specifically talked about hiring from the finance industry here in Chicago. I smiled - maybe even giggled a little bit, and he asked me what that was about.

I explained what had been on my mind for a while: Was this place really ready for the kind of change that these kinds of hires was going to represent? I was very specific about the fact that I honestly had no idea what the long-term effects were going to be, but just based on my experience, I know that there was certainly going to be a division based solely on physical things like hours worked, and hours not worked while playing ping-pong.

These simple things, and the drive, intensity and others are going to make it clear to everyone that these new developers are not at all like the others here now. My fear is that the critical developers we have now will see this, sense the winds of change, and decide that this is no longer the place they know and love, and simply leave. It won't have to be all at once, but if the right few leave, then it'll be very hard to continue the development of the existing web apps, and then more significant changes will have to take place.

It could easily turn into an avalanche where a few Type A groups make enough of the old guard uncomfortable, and that forces even more of the new to replace/retool/re-work the old apps, and that makes even more old guard jittery, etc. It can be a nasty feedback cycle that causes a massive change in the landscape in a very short time.

So I was talking with my manager about this, and it quickly got to the point that we needed to talk about what's happening with me and this group.

They love the work I do. He said he's never seen anyone work anything like what I do. But he also sees that the other guys in the group are complaining about the fact that I'm taking ownership of too much - making it hard for them to feel involved in the project. I explained myself, and it's clear that we had the same situation that I've had several times before - I work to the point that others feel threatened, marginalized, and then they resent me even though there's nothing to resent. Not really.

I'm working on the things we, as a group, need to do. I'm doing my best to do this in the same way that I'm seeing them set by example. After all, I'm no great ruby coder - or clojure coder. I'm looking at what they do - how they do it, and following along as best I can.

I just happen to do it about four times their speed.

In the beginning, I was "cute" and interesting. Now that I've gotten up to speed, I'm not so cute, and I'm becoming a real annoyance to them. I get it. I really do. But I'm not doing any of this to be a trouble-maker. I'm doing this because I think this is what I should be doing. This is just how I work.

I'm 51, for Pete's sake! Aren't they the 20-somethings supposed to be showing this "Old Man" how things are done? Why are they crying "Uncle!"?

So I have to now work harder at including them. I do. I don't take things over. I don't change their code without serious reason, and when I do I document the heck out of 'why'. I am trying to fit in, but it's not turning out that way.

My manager is talking about finding something for me to do that's outside this, and building a team around me. But I know the key factor is that what I'm on now is the most important project he's got, and to take me off that is just crazy in management's mind. I'm the one that's making everyone breathe easy on this one. So to take me off now just isn't happening.

But I'm a problem for the guys in the group, it seems.

I wish I knew what to do - other than the obvious. The thing that pops to mind is a simple group chat. Plain, simple, clear the air. I don't expect anyone to change their minds, but it will at least let them know that I mean them no harm, and that it's all up to them. They get to choose how hard to work. They get to choose when to work. They choose.

Not me. Them.

And when they realize that - that it's less about me than them, then maybe - just maybe, they'll start to see that it's not me that's the problem.

iTerm2 is Good, but Terminal.app is Better

Wednesday, January 23rd, 2013

Terminal.gif

I switched from Terminal.app to iTerm2 about a week ago because I liked the way it's double-click selection was controllable in preferences, and each tab in a window was selectable with the Cmd-1, Cmd-2, etc. whereas Terminal.app uses this to switch between windows. Not typically how that hotkey is used in other apps I use.

Plus, How different could it be - really? kept running through my mind. It's a shell. Should be pretty simple. Well… it's not that simple, and I've finally had it with iTerm2.

In the past week, large outputs to the terminal have crashed the entire iTerm2 app. I'm not saying that if it were like Chrome or Safari, and only the one shell session crashed, I'd be "ok" with it, but to have multiple windows with multiple tabs crash on you due to one tab is more than a little annoying to me.

So I'm heading back, and I'll probably never switch off it simply because Terminal.app never crashes on me. Just never. The convenience of the selection and tab switching would be nice to have, but I just can't have a crashing terminal program. Nope.

Added Investigative Tool to Metrics App

Wednesday, January 23rd, 2013

WebDevel.jpg

This morning, with the right help, I was able to put up a slick little javascript JSON viewer on the main 'metrics' web page for the project I'm on at The Shop. The goal of this is really a quick way to allow folks - both developers and support folks, to look at the details of the Demand objects in the demand service. Since each demand has a UUID, it's possible to look at the series of the demand objects as it "flows" through the service and is adjusted, combined, and delivered to the client.

I have to say that I'm very impressed with the speed of the clojure service. Then again, it's java, and there's not all that much to it - there's the indexed postgres table, a select statement, and then formatting. Sure, that's a simplification, but it's not like we're delivering streaming video or something. But wow! It's fast.

And the regex-based JSON classifier in javascript is pretty impressive as well. It's fast and clean and the CSS is a lot of fun to play with in the syntax highlighting. I can really see how much fun this can be. Big time sink, if I played with it too much.

But it's nice to punch it out and make it available on the much improved unicorn/nginx platform. Those timeouts are gone, and the code is much simpler, and that's a wonderful thing.

Slick Little JSON Viewer

Wednesday, January 23rd, 2013

WebDevel.jpg

Late yesterday I found a slick little javascript-based JSON formatter and syntax highlighter. What I was looking for was a simple and easy way to get the JSON data out of the demand service and display it to the user. The point was to have a clean way for the users to be able to see the low-level data for a specific demand in a nice way so that they can look into problems without having to go to a developer or support person.

The project manager said that we didn't need to have anything fancy - just display the JSON data for the object as it's reasonably self-documenting, and most of the folks could figure out what they needed from that. For the more serious use-cases, we're still going to need to have a more comprehensive dashboard, but for now, this would be something we could whip up pretty easily and solve 90% of the need.

The code was really amazingly simple:

I have the classifier and then the CSS for the individual classes. Then I just have to hit the server based on the demand ID and show the results. Very nice and clean.

Pretty stylish, too! I'm impressed.