Code Coverage Tools and Clojure

August 18th, 2015

Clojure.jpg

I was asked recently by my manager to look into code coverage tools for clojure because some of the other Senior Devs at The Shop wanted to have code (test) coverage numbers automatically generated for all projects as a part of continuous integration (TeamCity), and then available for all to see and track. I can certainly understand what they are trying to achieve - testable code that allows them to feel comfortable changing the code after the original author is long gone.

With complete test coverage, the theory goes, you can make a change, and then run these tests and prove to yourself that you haven't broken something because you didn't take the time to really understand the codebase.

It's an understanable goal from a management perspective... but even there, I have to think that this has never worked in my experience - and there's no way to have really foolproof tests. Yet today I read a nice post, which contained a quote from Rich H. about testing:

A bad design with a complete test suite is still a bad design. -- Rich H.

And the author, Alex, also verbalizes a lot of the concerns I've had over the years about tests. I can remember adding a feature in 15 mins and then spending several hours updating the unit tests because there were so many that had double-coverage, and yet none could (politically) be removed. Tests, it seems, are a lot like gold bricks - once people get some, they are very reluctant to get rid of any.

Yet they are code, and cost to maintain just like code. You can't think that tests are "free" once they are written... at least not if you're honest with yourself. And to Rich's point, it's better to think first and then attack the problem than it is to believe that tests are everything you need.

It was a very illuminating article. I'm glad a have a link to it now. I'll be sending it to a lot of folks who talk about tests in the future.

Slow Starts but Strong Finishes

July 8th, 2015

cubeLifeView.gifOver the course of the last week or so I've noticed that I'm having a really hard time getting motivated in the morning. I'm running well - and that's nice, but it's after that - when I leave for the train, and then work. It's just hard feeling that I'm heading out to do something worthwhile and good.

More like just do what I needed to collect a paycheck. And I hate that feeling.

Yet at about 4:00 pm, when the workday is almost over, I'm looking back at what I accomplished during the day and it's simply amazing! It's like I'm able to forget about all the things that had me down at the start of the day, and the things I'm able to do with the code I'm working on make me smile and laugh -- all the time!

So I'm trying to figure out what it's all about. Why the slow start, and strong finish? Why can't I look forward to the work so that my day starts with excitement? If I know I'm heading for a day that I'm going to do great, fun things - then why can't I use that as motivation at the start of the day?

I have a feeling it's related to The Three-Martini Lunch.

Specifically, I'm not really sure what's happening at The Shop right now. Yes, there's work, for now, but it could all end on Aug 1 - when I deliver this app I'm working on. They have plans, but it's all still awaiting approval by upper-management, and that, to me, means that it's still up for debate as to what will be done.

If that's the case, then I could find myself maintaining one little clojure app - and some .NET code (shudder) in just no time at all.

Yet when I'm able to ignore all that... forget the place I'm working, and just focus on the problems at the keyboard, then I can start solving problems, and delivering value. That makes me happy. It's like work - the real act of solving probelems, using clojure and it's amazing toolset, allows me to forget the realities of my current situation.

Clojure is my mid-afternoon drinking.

It doesn't solve anything. It's just getting drunk to avoid the problems you're facing. Tomorrow you'll be sober, and it'll all come back to you. So you dive into the "bottle" (work) and don't come up for air.

I'm not sure this is the correct analogy, but I have a feeling it's close. I love the work, and the work is what I find enjoyment from. But the place seems to be having a hard time with focus, and that concerns me.

Not much I can do about it, so it's likely to continue for a while.

SSL in Jetty (Compojure)

July 6th, 2015

Clojure.jpgI've been working on a project at The Shop and it's a simple-ish clojure back-end with a Javascript/jQuery/Bootstrap front-end, and it's not too bad. Getting there. But recently, it was time to add in the user authentication code, and that meant passwords. That meant security, and simple HTTP requests weren't going to do. We had to have security. HTTPS. So I dove down the rabbit hole of getting Jetty 7 to run as SSL only within a compojure/ring application.

What I found was that there really wasn't a wealth of information on the subject. Most folks expected a front-end service like nginx to handle the SSL, and then forward on to the clojure app requests on simple port 8080. While this is possible, it also means that port 8080 is there listening, and it's possible to have an intercept on that port and all of a sudden we're not so secure.

Nope. It needed to be Jetty on SSL on port 8443. Period.

Get the Certificate File

No doubt, getting the initial certificate file is the hardest part for most folks, as it's all about money. But I suppose self-signed certificates are useful - to a point, but being in a commercial organization, it's nice to see that there's a real certificate file, and all I need to do is to convert it to a Java Key Store file. In this case, it started as a pox file - a PKCS formatted certificate. Then:

  $ keytool -importkeystore -srckeystore my_cert.pfx -srcstoretype pkcs12 \
      -destkeystore my_cert.jks -deststoretype jks -deststorepass MY_PASSWORD

where it's clear what the file names are, and passwords. This gets you a my_cert.jks file that's what you are going to need. The MY_PASSWORD is also important as it will have to be in the code as well to read the file.

Put the KeyStore File in resources/certs

Like everything additional in a Meiningen project, this needs to go into the resources directory, and for convention, I'm throwing it into resources/certs as it's not the only thing in my resources directory for this project, and a little separation is a good thing.

Convert the Code to Dual-Mode

Working with the following dependencies in the project.clj file:

    ;; command line option processing
    [org.clojure/tools.cli "0.2.2"]
    ;; web server
    [compojure "1.3.4"]
    [ring/ring-core "1.3.2"]
    [ring/ring-jetty-adapter "1.3.2"]
    [ring.middleware.jsonp "0.1.6"]
    [ring/ring-defaults "0.1.5"]

and assuming you have a it running on port 8080, it might look a little like this:

(ns myapp.main
  (:require [clojure.java.io :refer [resource input-stream]]
            [clojure.tools.cli :refer [cli]]
            [clojure.tools.logging :refer [error info infof]]
            [myapp.server :refer [app]]
            [ring.adapter.jetty :as jt])
  (:import java.security.KeyStore)
  (:gen-class))
 
  (defn handle-args
    "Function to parse the arguments to the main entry point of this
    project and do what it's asking. By the time we return, it's all
    done and over."
    [args]
    (let [[params [action]] (cli args
               ["-p" "--port" "Listen on this port"
                 :default 8080 :parse-fn #(Integer. %)]
               ["-v" "--verbose" :flag true])]
      (cond
        (= "web" action)
          (jt/run-jetty app { :port (:port params) }))
        :else
          (do
            (info "Welcome to My App!")
            (println "Welcome to My App!")))))
 
  (defn -main
    "Function to kick off everything and clean up afterwards"
    [& args]
    (with-error-handling (handle-args args)))

this will assume port 8080, and yet allow the command-line args to override this. In order to add the SSL component to this, we simply have to add a few options to Jetty:

  (defn handle-args
    "Function to parse the arguments to the main entry point of this
    project and do what it's asking. By the time we return, it's all
    done and over."
    [args]
    (let [[params [action]] (cli args
               ["-p" "--port" "Listen on this port"
                 :default 8080 :parse-fn #(Integer. %)]
               ["-s" "--ssl-port" "Listen on this port"
                 :default 8443 :parse-fn #(Integer. %)]
               ["-v" "--verbose" :flag true])]
      (cond
        (= "web" action)
          (jt/run-jetty app { :port (:port params)
                              :ssl? true
                              :ssl-port (:ssl-port params)
                              :keystore "resources/certs/my_cert.jks"
                              :key-password "MY_PASSWORD" }))
        :else
          (do
            (info "Welcome to My App!")
            (println "Welcome to My App!")))))

Start this guy now, and it'll answer on port 8080 for normal HTTP traffic, and port 8443 for HTTPS traffic. So far, so good. But there's a catch here, and we'll get to it soon.

Make it SSL Only

The real request was to make is SSL-only, so we to remove the port 8080 traffic, but we can't just tell Jetty not to run that one - we have to actively remove it. Thankfully, composure allows us this flexibility. If we make a function:

  (defn remove-non-ssl-connectors
    "Function to configure the Jetty instance to remove all non-SSL
    connectors so that there's **no way** to get into this service BUT
    by SSL (https)."
    [server]
    (doseq [c (.getConnectors server)]
      (when-not (or (nil? c)
                    (instance? org.eclipse.jetty.server.ssl.
                                    SslSelectChannelConnector c))
        (.removeConnector server c)))
    server)

then we can add that as an option to Jetty to tell it to run on it's configuration:

  (defn handle-args
    "Function to parse the arguments to the main entry point of this
    project and do what it's asking. By the time we return, it's all
    done and over."
    [args]
    (let [[params [action]] (cli args
               ["-p" "--port" "Listen on this port"
                 :default 8080 :parse-fn #(Integer. %)]
               ["-s" "--ssl-port" "Listen on this port"
                 :default 8443 :parse-fn #(Integer. %)]
               ["-v" "--verbose" :flag true])]
      (cond
        (= "web" action)
          (jt/run-jetty app { :configurator remove-non-ssl-connectors
                              :port (:port params)
                              :ssl? true
                              :ssl-port (:ssl-port params)
                              :keystore "resources/certs/my_cert.jks"
                              :key-password "MY_PASSWORD" }))
        :else
          (do
            (info "Welcome to My App!")
            (println "Welcome to My App!")))))

and now if you restart the app, it won't answer on port 8080, but it will still answer on port 8443 with HTTPS. Getting very close.

Making it Deployable in an Uberjar

The final wrinkle is that the :keystone option is a location in the filesystem of the key store file. That's not good for deployments because it means that while the keystore file will be packaged up in the uberjar, it's not going to be referenced that way - and it will have to also exist in the filesystem in the same relative location.

This stinks.

So I did some more digging, and ring-jetty had what I needed - the ability to pass it a java.security.KeyStore instance. Now I needed to read the keystone file from the jar, into an instance of that object, and pass it in.

Start with:

  (defn load-keystore
    "Function to load the SSL KeyStore from the resources so that it's ready
    to be used to run the SSL connections for Jetty. This is a preferred method
    to having just a path for locating the certificate, as this allows the cert
    to be _included_ in the uberjar itself."
    [loc pw]
    (if (and (string? loc) (string? pw))
      (doto (KeyStore/getInstance (KeyStore/getDefaultType))
        (.load (input-stream (resource loc)) (char-array pw)))))

which will load a keystone file from the uberjar - placed in the resources directory. This was a major find for me, and made it all possible. That it only takes a few lines of clojure and java is just amazing.

Now we can put it all together:

  (defn handle-args
    "Function to parse the arguments to the main entry point of this
    project and do what it's asking. By the time we return, it's all
    done and over."
    [args]
    (let [[params [action]] (cli args
               ["-p" "--port" "Listen on this port"
                 :default 8080 :parse-fn #(Integer. %)]
               ["-s" "--ssl-port" "Listen on this port"
                 :default 8443 :parse-fn #(Integer. %)]
               ["-v" "--verbose" :flag true])]
      (cond
        (= "web" action)
          (let [loc "certs/my_cert.jks"   ;; SSL keystore location
                pw "MY_PASSWORD"]         ;; ...and password
            (jt/run-jetty app { :configurator remove-non-ssl-connectors
                                :port (:port params)
                                :ssl? true
                                :ssl-port (:ssl-port params)
                                :keystore (load-keystore loc pw)
                                :key-password pw }))
        :else
          (do
            (info "Welcome to My App!")
            (println "Welcome to My App!")))))

Pulling Query Params from URL in Javascript

June 24th, 2015

SquirrelFish.jpg

As part of this continuing project at The Shop, one of the things we're going to need is to be able to put URLs in emails and have the user click on them, and have it take them right to the document in question. This means we need to have the index.html page accept a query param of the ID of the thing to load. Then, if we get to the page, and this query param is there, then we load the requested data, if not, we put up a blank page, and then let the user search for the document they want.

Seems simple. If we can get the query params from the URL in Javascript.

Thankfully, someone posted this lovely little function:

  /*
   * This function extracts the provided query string from the URL for
   * the page we're on, and it's a nice and simple way to get the parts
   * of the URL that we're looking to see if they provided.
   */
  function qs(key) {
    // escape RegEx meta chars
    key = key.replace(/[*+?^$.\[\]{}()|\\\/]/g, "\\$&");
    var match = location.search.match(new RegExp("[?&]"+key+"=([^&]+)(&|$)"));
    return match && decodeURIComponent(match[1].replace(/\+/g, " "));
  }

Then we can have a little script block at the bottom of the index.html page, after loading all the javascript libraries, that checks to see if this parameter is being provided:

  // when the page is done loading, then load up the form
  var lid = qs('loan');
  $( document ).ready(loadLoan(($.isNumeric(lid) ? lid : null)));

This snip does everything we want - checks for the numeric value, and if it's there, uses it, if not, shows the blank page. Very nice.

Nice Bootstrap Trick for Clean Search Box

June 24th, 2015

SquirrelFish.jpg

I've been working to build a little collaboration tool at The Shop with a clojure back-end and a simple Bootstrap front-end. I have to admit that Bootstrap and Handsontable are some amazing tools, and I could not imagine doing the project without them.

Part of this was to have a simple 'Search' feature in the top menu bar where the users could input the known ID of a document to pull up, and the system would do it. Thankfully, Bootstrap supports this capability nicely:

better colors

But the problem with this is that it's put in the page as an HTML form:

  <form id="search_loan" class="navbar-form navbar-right" role="search">
    <div class="form-group">
      <input id="loan_id" type="text" class="form-control" placeholder="Search">
    </div>
    <button id="find_loan" class="btn btn-default">Go!</button>
  </form>

so that when you hit 'Enter' in the text box, or click the 'Go!' button, it performs the POST, and your code either has to be ready for the POST, or you have two refresh of the data - and that's not simple or clean. The solution is to intercept the form submission and hijack the event to do your bidding.

At the bottom of your HTML page, where you load all the javascript libraries, you can put a little script block, last of all, and it'll do some cool stuff:

  // set up the form components to work as we need them
  $("#search_loan").submit( function(e) {
    e.preventDefault();
    loadLoan(document.getElementById('loan_id').value);
  });

this little bit of code will capture the form submission event, prevent it's default behavior from occurring, and then call the loadLoan function with the contents of the text box in the search field.

Given that this function is what you want to have happen, this will make the search box work just like you want. All from one page, no redirections, no calls to refresh the page. Just load up the data on the search. Very cool.

Built a Nice Simple Clojure/Bootstrap/Google App

June 2nd, 2015

Clojure.jpg

Over the course of the last day and a half, I've been putting together a nice little test app using clojure and compojure on the back-end, and Bootstrap with Handsontable as the UI. It's simple, and not all that complicated, but it's nice in that it demonstrates a lot of the things that I often want to do in a simple application.

Then today one of the guys I was putting this together for suggested OAuth2 for authentication and then something internal for authorization. This turned out to be not all that bad with Google's Identity Service. They have a nice little JavaScript client that gets the auth token that you then feed to the calls to the service, and it, in turn, hits Google to make sure that this is a person that they say they are. You end up with an email address, and that can be used as the "username" for any authorization scheme you want.

In short, it's very slick. Works like a dream. I'm having a pretty good time with this.

better colors

Excellent JDBC4 Fix for ?-operators in Postgres

May 28th, 2015

PostgreSQL.jpg

In Postgres 9.4, there are several JSONB operators that include a ? as part of the operator. This is nasty for the JDBC usage because the JDBC driver uses the question-mark as the substitute parameter place-holder for arguments. And there's no escaping of values in a JDBC PreparedStatement so what's a guy to do?

Sure, you can make a simple custom function to do this, but that's kinda wasteful, and it's not very transportable. Thankfully, I was reading the release notes for the 9.4 JDBC driver and saw this.

Using the latest, released JDBC driver in your project.clj file:

  [org.postgres/postgres "9.4-1201-jdbc41"]

you can pick up this pull request, and feature.

By simply doubling the question mark, it'll now reduce this to one ? and the operator will work. Very slick. So now in a clojure SQL statement, just put ?? when you need ? and you're good to go!

Fantastic! I've checked it and it works perfectly.

Slick Way to Add Array and JSON Support to Postgres Clojure JDBC

May 28th, 2015

Clojure.jpg

The default JDBC behavior in clojure.java.jdbc is not to handle the additional Postgres data types like Array and JSON. But you do have the protocols that you can use to override the behavior of how to store and read things - and this article looks to set this up for both these data types.

The idea is to look at the data type on both the write and read, and do the JSON mapping and we're done. The code is very simple:

  (ns our-app.jdbc.json
    "Inspired by http://www.niwi.be/2014/04/13/
     postgresql-json-field-with-clojure-and-jdbc/"
    (:require [clojure.java.jdbc :as jdbc]
              [cheshire.core :as json])
    (:import org.postgresql.util.PGobject))
 
  (defn value-to-json-pgobject [value]
    (doto (PGobject.)
      (.setType "json")
      (.setValue (json/generate-string value))))
 
  (extend-protocol jdbc/ISQLValue
    clojure.lang.IPersistentMap
    (sql-value [value] (value-to-json-pgobject value))
 
    clojure.lang.IPersistentVector
    (sql-value [value] (value-to-json-pgobject value)))
 
    clojure.lang.IPersistentList
    (sql-value [value] (value-to-json-pgobject value)))
 
    clojure.lang.LazySeq
    (sql-value [value] (value-to-json-pgobject value)))
 
  (extend-protocol jdbc/IResultSetReadColumn
    PGobject
    (result-set-read-column [pgobj metadata idx]
      (let [type  (.getType pgobj)
            value (.getValue pgobj)]
        (case type
          "json" (json/parse-string value true)
          :else value))))

Where I have chosen to use Cheshire, as opposed to clojure.data.json. It's a matter of taste, I'll agree, but we're using Cheshire all the time anyway, so the cost is nothing, and it's consistently parsed in the same way.

Very cool.

Twitter Ad Service API

May 14th, 2015

Twitterrific.jpg

Today I spent a good bit of the day trying to figure out how to authenticate with Twitter's OAuth 1.0 system, and I think I'm getting close, but I'm still a bit away because I don't control these accounts, and the sheer volume of ways to authenticate on Twitter is daunting. Let allne the different APIs.

There is the client-facing Tweets API, and then there's the Ad Server API, and it's not at all clear that there needs to be different authentication schemes for these APIs. But it should be clear that access to one set of APIs probably should not guarantee access to another set - and maybe they handle that in the authorization, but it's not clear from the docs I'm reading.

And speaking of docs, wow... these are really something else. There are at least four ways to authenticate, but they ask people to use libraries - that they don't provide. Sadly, I don't see one that does 100% of what I need, but I do see an OAuth 1.0 library, but the Client ID and Secret are nowhere to be found on their site.

So clearly, I'm missing something.

What I believe is that you have to create an App that then gets you the redirect URL and ID and Secret. There were none defined to base a new one on, and so I sent off an email to the Twitter representative to see if this was, indeed the preferred way.

While I was waiting, I decided to try and make an app. Yet in order to do that, you need to assign a mobile phone number to the Twitter account, and I can't really do that because the account is not mine. SO I sent another email to the relationship folks in The Shop about that.

In short, it's just a waiting game. But it's also so much more of a mess than the other systems I've been integrating with. Wow...

Got a Call… from a Guy… about Clojure…

May 13th, 2015

Clojure.jpg

Yesterday I got a quick note from an ex-co-worker about how he might go about writing a micro-service in Clojure. Seems his CTO is a big functional programmer, and as such, he wanted my friend to write something in Clojure. So he reached out to the only person he knew that had actually done that, and so I got the call.

I was happy to help him along. I sent him links to Gists about how a server.clj file might look, as well as how to handle JSONP - assuming that the calls were coming from jQuery, and basically, just make sure that things got off on the right foot.

I also gave him some pointers about the standard libraries that I've been using for clojure work - cli-time, composure, cheshire, carmine, etc. They make it a lot easier, and a lot of folks are using them - so it's good to get started using the standard stuff right off the bat.

We talked a lot about getting started, and how the RESTful stuff was going to look, and then he said he'd dig into it when he had time. But the most impressive part was the recognition that by simply saying "Clojure Developers Wanted" - the CTO knew he'd get all kinds of hits, and all kinds of street cred to make the most simpler.

Smart man... deserves to be in that position.