What's it all about, Alfie?

One of the things that I find I need to do every now and then is to have a case-insensitive regex pattern match in clojure, and I always have to spend 15 mins looking for it on the net. Not that it's hard to find, but it's 15 mins, and if I know I wrote about it here, then I can do a much quicker search, and get the same answer.

The key is that the Java regex parser recognizes two interesting options to the source for the regex: (?i) and (?iu). The first, (?i), is the flag to indicate that the entire regex is to be considered case-insensitive. The second, (?iu) is for unicode support.

So how's it work?

  (re-find #"Cook" "This is how to cook.")
  => nil
 
  (re-find #"(?i)Cook" "This is how to cook.")
  => "cook"

It's simple and easy, and typically saves you from calling lcase on the string before the regex test.

This morning I noticed an upgrade notice in Adium, and while I wasn't going to go back to 1.5.11b2 - due to the Sparkle issues and security, I read it and the maintainers had created 1.5.11b3! While I know it's not making a lot of progress, it's something to see these updates shipping - it's not dead. And that's great news in my book.

Sure, Microsoft has killed Hotmail's IM support - and that's a shame, I don't understand why they wouldn't want to have users on their service, but that's their choice and not mine. I still hope that one day libpurple will support it, but that's a wish, not something I'm really holding my breath for.

But today it's nice to have Adium updated with the latest Sparkle and a few other fixes. Nice!

This morning I saw that Adium 1.5.10.1 had been released, and while I've been running 1.5.11b2 for quite a while, and while I was reluctant to go back to 1.5.10.1, because there had to be something in 1.5.11b2 I liked, I realized that there really wasn't. Not a thing. And as long as I had the ability to get back to 1.5.11b2, I could download it and go back to it again.

No downside.

Then again, it's important to note that Adium hasn't been getting a lot of love from the maintainers lately. I have been toying with the idea of getting the code and building it, and then seeing if I could fix the Microsoft Live connection so that I could once again get my Hotmail IM again.

Then word spread of the Sparkle security issue. I didn't know the depth of the issue on the software I have - but certainly a lot of the OS X software uses Sparkle, and the HTTP vs HTTPS endpoints is a big issue, and I'm glad they have fixed it, but then Adium reported a new version with the new Sparkle, and I decided that maybe 1.5.10.1 isn't so bad.

Maybe it's stable, and not abandoned. Maybe I just need to realize it's working great for me - save the Hotmail issue, and that's good enough.

So back I am. Happy to be here.

Postgres is an amazing database. With 9.4's JSONB datatype, and a little clojure code, I can save data structures to and from Postgres, and have them query-able by any JDBC client. It's just amazing. But recently we had a problem at The Shop where we needed to index a field in a JSONB element, but it was a date, and so we ran into the problem of Immutable Functions.

Simply trying to create an index on the field:

  CREATE INDEX idx_d ON my_table ((fields->>'DateStarted'))::DATE;

didn't work with Postgres saying you can't have a mutable function in the creation of the index. So it was time to dig out the browser and look up what's going on. Sure enough, this was something well-known, and all involving the conversion of dates by locale, and why that was not allowed for an index.

Makes sense, but I still needed an index because the queries using this field in the WHERE clause were taking 6+ min, and we needed to really bring them down - like way down. Then I found that you could make functions that were tagged as IMMUTABLE and so made these:

  CREATE OR REPLACE FUNCTION mk_date(src VARCHAR)
  RETURNS DATE
  immutable AS $body$
    SELECT src::DATE;
  $body$ LANGUAGE SQL;
 
  CREATE OR REPLACE FUNCTION mk_datetime(src VARCHAR)
  RETURNS TIMESTAMP
  immutable AS $body$
    SELECT src::TIMESTAMP;
  $body$ LANGUAGE SQL;

the reason I chose to use ::date and ::timestamp is that they handle NULL values very well, and the corresponding functions don't do that so well.

With these functions, I was then able to create the index:

  CREATE INDEX idx_d ON my_table (mk_date((fields->>'DateStarted')));

and then as long as I used the same functions in my WHERE clause, everything was great.

The result was a 1000x increase in speed - 319,000 msec to 317 msec. Amazing.

I do love Postgres.

The Shop is big into Datadog, and it's not a bad metrics collection tool, which I've used very successfully from clojure. Based on the use of the local Datadog Agent (freely available from Datadog) and how easily it's placed on linux hosts - AWS or otherwise, it's a clear win for collecting metrics from your code and shipping them to a nice graphing/alerting platform like Datadog.

The code I've set up for this is pretty simple, and based on the com.codahale.metrics java libraries. With a simple inclusion into your project.clj file:

  [io.dropwizard.metrics/metrics-core "3.1.0"]
  [org.coursera/dropwizard-metrics-datadog "1.0.2"]

you can then write a very nice metrics namespace:

  (ns ns-toolkit.metrics
    "This is the code that handles the metrics and events through the Dropwizard
    Metrics core library, which, in turn, will ship it over UDP to the DataDog
    Agent running on localhost."
    (:require [clojure.tools.logging :refer [infof debugf warnf errorf]])
    (:import [com.codahale.metrics MetricRegistry]
             [org.coursera.metrics.datadog DatadogReporter]
             [org.coursera.metrics.datadog.transport UdpTransportFactory
                                                     UdpTransport]
             [java.util.concurrent TimeUnit]))
 
  ;; Create a simple MetricRegistry - but make it only when it's needed
  (defonce def-registry
    (delay
      (let [reg (MetricRegistry.)
            udp (.build (UdpTransportFactory.))
            rpt (-> (DatadogReporter/forRegistry reg)
                  (.withTransport udp)
                  (.withHost "localhost")
                  (.convertDurationsTo TimeUnit/MILLISECONDS)
                  (.convertRatesTo TimeUnit/SECONDS)
                  (.build))]
        (.start rpt 5 TimeUnit/SECONDS)
        reg)))
 
  ;; Somewhat faking java.jdbc's original *connection* behavior so that
  ;; we don't have to pass one around.
  (def ^:dynamic *registry* nil)
 
  (defn registry
    "Function to return either the externally provided MetricRegistry, or the
    default one that's constructed when it's needed, above. This allows the user
    the flexibility to live with the default - or make one just for their needs."
    []
    (or *registry* @def-registry))

And then we can define the simple instrumentation types from this:

  ;;
  ;; Functions to create/locate the different Metrics instruments available
  ;;
 
  (defn meter
    "Function to return a Meter for the registry with the provided tag
    (a String)."
    [tag]
    (if (string? tag)
      (.meter (registry) tag)))
 
  (defn counter
    "Function to return a Counter for the registry with the provided tag
    (a String)."
    [tag]
    (if (string? tag)
      (.counter (registry) tag)))
 
  (defn histogram
    "Function to return a Histogram for the registry with the provided tag
    (a String)."
    [tag]
    (if (string? tag)
      (.histogram (registry) tag)))
 
  (defn timer
    "Function to return a Timer for the registry with the provided tag
    (a String)."
    [tag]
    (if (string? tag)
      (.timer (registry) tag)))

These can then be held in maps or used for any reason at all. They automatically send their data to the local Datadog Agent over UDP so there's no delay to the logger, and since it's on the same box, the likelihood that something will be dropped is very small. It's a wonderful scheme.

But one of the things that's not covered in these metrics is the Gauge. And there's a really good reason for that - the Gauge for Datadog is something that is read from the Datadog Agent, and so has to be held onto by the code so that subsequent calls can be made against it for it's value.

In it's simplest form, the Gauge is just a value that's read by the agent on some interval and sent to the Datadog service. This callback functionality is done with a simple anonymous inner class in Java, but that's hard to do in clojure - or is it?

With Clojure 1.6, we have something that makes this quite easy - reify. If we simply add an import:

  (:import [com.codahale.metrics Gauge])

and then we can write the code to create an instance of Gauge with a custom getValue() method where we can put any clojure code in there we want. Like:

  ;;
  ;; Java functions for the Metrics library (DataDog) so that we can
  ;; constantly monitor the breakdown of the active docs in the system
  ;; by these functions.
  ;;
  (defn cnt-status
    "Function that takes a status value and finds the count of loans
    in the `laggy-counts` response that has that status. This is used
    in all the metrics findings - as it's the exact same code - just
    different status values."
    [s]
    (reify
      Gauge
      (getValue [this]
        (let [sm (first (filter #(= s (:status %)) (laggy-counts)))]
          (parse-int (:count sm))))))
 
  (defn register-breakdown
    "Function to register all the breakdowns of the loan status counts
    with the local Datadog agent to be sent to Datadog for plotting. This
    is a little interesting because Datadog will call *these* functions
    as needed to get the data to send, and we will control the load by
    using memoized functions."
    []
    (.register (met/registry)
      "trident.loan_breakdown.unset"
      (cnt-status nil))
    (.register (met/registry)
      "trident.loan_breakdown.submit_to_agent"
      (cnt-status "Submit to Agent"))
    (.register (met/registry)
      "trident.loan_breakdown.submit_to_lender"
      (cnt-status "Submit to Lender"))
    (.register (met/registry)
      "trident.loan_breakdown.submit_to_lender_approved"
      (cnt-status "Submit to Lender - Agent Approved"))
    (.register (met/registry)
      "trident.loan_breakdown.lender_approved"
      (cnt-status "Lender Approved")))

What I like about this is that I can allow the Datadog Agent to hit this code as often as it wants, and don't have to worry about the freshness of the data - or an excessive loan on the server resources for being hit too much. I can simply memoize the functions I'm using and then control the load on my end. It's very clean, and very nice.

I was asked recently by my manager to look into code coverage tools for clojure because some of the other Senior Devs at The Shop wanted to have code (test) coverage numbers automatically generated for all projects as a part of continuous integration (TeamCity), and then available for all to see and track. I can certainly understand what they are trying to achieve - testable code that allows them to feel comfortable changing the code after the original author is long gone.

With complete test coverage, the theory goes, you can make a change, and then run these tests and prove to yourself that you haven't broken something because you didn't take the time to really understand the codebase.

It's an understanable goal from a management perspective... but even there, I have to think that this has never worked in my experience - and there's no way to have really foolproof tests. Yet today I read a nice post, which contained a quote from Rich H. about testing:

A bad design with a complete test suite is still a bad design. -- Rich H.

And the author, Alex, also verbalizes a lot of the concerns I've had over the years about tests. I can remember adding a feature in 15 mins and then spending several hours updating the unit tests because there were so many that had double-coverage, and yet none could (politically) be removed. Tests, it seems, are a lot like gold bricks - once people get some, they are very reluctant to get rid of any.

Yet they are code, and cost to maintain just like code. You can't think that tests are "free" once they are written... at least not if you're honest with yourself. And to Rich's point, it's better to think first and then attack the problem than it is to believe that tests are everything you need.

It was a very illuminating article. I'm glad a have a link to it now. I'll be sending it to a lot of folks who talk about tests in the future.

I've been working on a project at The Shop and it's a simple-ish clojure back-end with a Javascript/jQuery/Bootstrap front-end, and it's not too bad. Getting there. But recently, it was time to add in the user authentication code, and that meant passwords. That meant security, and simple HTTP requests weren't going to do. We had to have security. HTTPS. So I dove down the rabbit hole of getting Jetty 7 to run as SSL only within a compojure/ring application.

What I found was that there really wasn't a wealth of information on the subject. Most folks expected a front-end service like nginx to handle the SSL, and then forward on to the clojure app requests on simple port 8080. While this is possible, it also means that port 8080 is there listening, and it's possible to have an intercept on that port and all of a sudden we're not so secure.

Nope. It needed to be Jetty on SSL on port 8443. Period.

Get the Certificate File

No doubt, getting the initial certificate file is the hardest part for most folks, as it's all about money. But I suppose self-signed certificates are useful - to a point, but being in a commercial organization, it's nice to see that there's a real certificate file, and all I need to do is to convert it to a Java Key Store file. In this case, it started as a pox file - a PKCS formatted certificate. Then:

  $ keytool -importkeystore -srckeystore my_cert.pfx -srcstoretype pkcs12 \
      -destkeystore my_cert.jks -deststoretype jks -deststorepass MY_PASSWORD

where it's clear what the file names are, and passwords. This gets you a my_cert.jks file that's what you are going to need. The MY_PASSWORD is also important as it will have to be in the code as well to read the file.

Put the KeyStore File in resources/certs

Like everything additional in a Meiningen project, this needs to go into the resources directory, and for convention, I'm throwing it into resources/certs as it's not the only thing in my resources directory for this project, and a little separation is a good thing.

Convert the Code to Dual-Mode

Working with the following dependencies in the project.clj file:

    ;; command line option processing
    [org.clojure/tools.cli "0.2.2"]
    ;; web server
    [compojure "1.3.4"]
    [ring/ring-core "1.3.2"]
    [ring/ring-jetty-adapter "1.3.2"]
    [ring.middleware.jsonp "0.1.6"]
    [ring/ring-defaults "0.1.5"]

and assuming you have a it running on port 8080, it might look a little like this:

(ns myapp.main
  (:require [clojure.java.io :refer [resource input-stream]]
            [clojure.tools.cli :refer [cli]]
            [clojure.tools.logging :refer [error info infof]]
            [myapp.server :refer [app]]
            [ring.adapter.jetty :as jt])
  (:import java.security.KeyStore)
  (:gen-class))
 
  (defn handle-args
    "Function to parse the arguments to the main entry point of this
    project and do what it's asking. By the time we return, it's all
    done and over."
    [args]
    (let [[params [action]] (cli args
               ["-p" "--port" "Listen on this port"
                 :default 8080 :parse-fn #(Integer. %)]
               ["-v" "--verbose" :flag true])]
      (cond
        (= "web" action)
          (jt/run-jetty app { :port (:port params) }))
        :else
          (do
            (info "Welcome to My App!")
            (println "Welcome to My App!")))))
 
  (defn -main
    "Function to kick off everything and clean up afterwards"
    [& args]
    (with-error-handling (handle-args args)))

this will assume port 8080, and yet allow the command-line args to override this. In order to add the SSL component to this, we simply have to add a few options to Jetty:

  (defn handle-args
    "Function to parse the arguments to the main entry point of this
    project and do what it's asking. By the time we return, it's all
    done and over."
    [args]
    (let [[params [action]] (cli args
               ["-p" "--port" "Listen on this port"
                 :default 8080 :parse-fn #(Integer. %)]
               ["-s" "--ssl-port" "Listen on this port"
                 :default 8443 :parse-fn #(Integer. %)]
               ["-v" "--verbose" :flag true])]
      (cond
        (= "web" action)
          (jt/run-jetty app { :port (:port params)
                              :ssl? true
                              :ssl-port (:ssl-port params)
                              :keystore "resources/certs/my_cert.jks"
                              :key-password "MY_PASSWORD" }))
        :else
          (do
            (info "Welcome to My App!")
            (println "Welcome to My App!")))))

Start this guy now, and it'll answer on port 8080 for normal HTTP traffic, and port 8443 for HTTPS traffic. So far, so good. But there's a catch here, and we'll get to it soon.

Make it SSL Only

The real request was to make is SSL-only, so we to remove the port 8080 traffic, but we can't just tell Jetty not to run that one - we have to actively remove it. Thankfully, composure allows us this flexibility. If we make a function:

  (defn remove-non-ssl-connectors
    "Function to configure the Jetty instance to remove all non-SSL
    connectors so that there's **no way** to get into this service BUT
    by SSL (https)."
    [server]
    (doseq [c (.getConnectors server)]
      (when-not (or (nil? c)
                    (instance? org.eclipse.jetty.server.ssl.
                                    SslSelectChannelConnector c))
        (.removeConnector server c)))
    server)

then we can add that as an option to Jetty to tell it to run on it's configuration:

  (defn handle-args
    "Function to parse the arguments to the main entry point of this
    project and do what it's asking. By the time we return, it's all
    done and over."
    [args]
    (let [[params [action]] (cli args
               ["-p" "--port" "Listen on this port"
                 :default 8080 :parse-fn #(Integer. %)]
               ["-s" "--ssl-port" "Listen on this port"
                 :default 8443 :parse-fn #(Integer. %)]
               ["-v" "--verbose" :flag true])]
      (cond
        (= "web" action)
          (jt/run-jetty app { :configurator remove-non-ssl-connectors
                              :port (:port params)
                              :ssl? true
                              :ssl-port (:ssl-port params)
                              :keystore "resources/certs/my_cert.jks"
                              :key-password "MY_PASSWORD" }))
        :else
          (do
            (info "Welcome to My App!")
            (println "Welcome to My App!")))))

and now if you restart the app, it won't answer on port 8080, but it will still answer on port 8443 with HTTPS. Getting very close.

Making it Deployable in an Uberjar

The final wrinkle is that the :keystone option is a location in the filesystem of the key store file. That's not good for deployments because it means that while the keystore file will be packaged up in the uberjar, it's not going to be referenced that way - and it will have to also exist in the filesystem in the same relative location.

This stinks.

So I did some more digging, and ring-jetty had what I needed - the ability to pass it a java.security.KeyStore instance. Now I needed to read the keystone file from the jar, into an instance of that object, and pass it in.

Start with:

  (defn load-keystore
    "Function to load the SSL KeyStore from the resources so that it's ready
    to be used to run the SSL connections for Jetty. This is a preferred method
    to having just a path for locating the certificate, as this allows the cert
    to be _included_ in the uberjar itself."
    [loc pw]
    (if (and (string? loc) (string? pw))
      (doto (KeyStore/getInstance (KeyStore/getDefaultType))
        (.load (input-stream (resource loc)) (char-array pw)))))

which will load a keystone file from the uberjar - placed in the resources directory. This was a major find for me, and made it all possible. That it only takes a few lines of clojure and java is just amazing.

Now we can put it all together:

  (defn handle-args
    "Function to parse the arguments to the main entry point of this
    project and do what it's asking. By the time we return, it's all
    done and over."
    [args]
    (let [[params [action]] (cli args
               ["-p" "--port" "Listen on this port"
                 :default 8080 :parse-fn #(Integer. %)]
               ["-s" "--ssl-port" "Listen on this port"
                 :default 8443 :parse-fn #(Integer. %)]
               ["-v" "--verbose" :flag true])]
      (cond
        (= "web" action)
          (let [loc "certs/my_cert.jks"   ;; SSL keystore location
                pw "MY_PASSWORD"]         ;; ...and password
            (jt/run-jetty app { :configurator remove-non-ssl-connectors
                                :port (:port params)
                                :ssl? true
                                :ssl-port (:ssl-port params)
                                :keystore (load-keystore loc pw)
                                :key-password pw }))
        :else
          (do
            (info "Welcome to My App!")
            (println "Welcome to My App!")))))

As part of this continuing project at The Shop, one of the things we're going to need is to be able to put URLs in emails and have the user click on them, and have it take them right to the document in question. This means we need to have the index.html page accept a query param of the ID of the thing to load. Then, if we get to the page, and this query param is there, then we load the requested data, if not, we put up a blank page, and then let the user search for the document they want.

Seems simple. If we can get the query params from the URL in Javascript.

Thankfully, someone posted this lovely little function:

  /*
   * This function extracts the provided query string from the URL for
   * the page we're on, and it's a nice and simple way to get the parts
   * of the URL that we're looking to see if they provided.
   */
  function qs(key) {
    // escape RegEx meta chars
    key = key.replace(/[*+?^$.\[\]{}()|\\\/]/g, "\\$&");
    var match = location.search.match(new RegExp("[?&]"+key+"=([^&]+)(&|$)"));
    return match && decodeURIComponent(match[1].replace(/\+/g, " "));
  }

Then we can have a little script block at the bottom of the index.html page, after loading all the javascript libraries, that checks to see if this parameter is being provided:

  // when the page is done loading, then load up the form
  var lid = qs('loan');
  $( document ).ready(loadLoan(($.isNumeric(lid) ? lid : null)));

This snip does everything we want - checks for the numeric value, and if it's there, uses it, if not, shows the blank page. Very nice.

I've been working to build a little collaboration tool at The Shop with a clojure back-end and a simple Bootstrap front-end. I have to admit that Bootstrap and Handsontable are some amazing tools, and I could not imagine doing the project without them.

Part of this was to have a simple 'Search' feature in the top menu bar where the users could input the known ID of a document to pull up, and the system would do it. Thankfully, Bootstrap supports this capability nicely:

But the problem with this is that it's put in the page as an HTML form:

  <form id="search_loan" class="navbar-form navbar-right" role="search">
    <div class="form-group">
      <input id="loan_id" type="text" class="form-control" placeholder="Search">
    </div>
    <button id="find_loan" class="btn btn-default">Go!</button>
  </form>

so that when you hit 'Enter' in the text box, or click the 'Go!' button, it performs the POST, and your code either has to be ready for the POST, or you have two refresh of the data - and that's not simple or clean. The solution is to intercept the form submission and hijack the event to do your bidding.

At the bottom of your HTML page, where you load all the javascript libraries, you can put a little script block, last of all, and it'll do some cool stuff:

  // set up the form components to work as we need them
  $("#search_loan").submit( function(e) {
    e.preventDefault();
    loadLoan(document.getElementById('loan_id').value);
  });

this little bit of code will capture the form submission event, prevent it's default behavior from occurring, and then call the loadLoan function with the contents of the text box in the search field.

Given that this function is what you want to have happen, this will make the search box work just like you want. All from one page, no redirections, no calls to refresh the page. Just load up the data on the search. Very cool.

Over the course of the last day and a half, I've been putting together a nice little test app using clojure and compojure on the back-end, and Bootstrap with Handsontable as the UI. It's simple, and not all that complicated, but it's nice in that it demonstrates a lot of the things that I often want to do in a simple application.

Then today one of the guys I was putting this together for suggested OAuth2 for authentication and then something internal for authorization. This turned out to be not all that bad with Google's Identity Service. They have a nice little JavaScript client that gets the auth token that you then feed to the calls to the service, and it, in turn, hits Google to make sure that this is a person that they say they are. You end up with an email address, and that can be used as the "username" for any authorization scheme you want.

In short, it's very slick. Works like a dream. I'm having a pretty good time with this.