Archive for the ‘Clojure Coding’ Category

Slow Starts but Strong Finishes

Wednesday, July 8th, 2015

cubeLifeView.gifOver the course of the last week or so I've noticed that I'm having a really hard time getting motivated in the morning. I'm running well - and that's nice, but it's after that - when I leave for the train, and then work. It's just hard feeling that I'm heading out to do something worthwhile and good.

More like just do what I needed to collect a paycheck. And I hate that feeling.

Yet at about 4:00 pm, when the workday is almost over, I'm looking back at what I accomplished during the day and it's simply amazing! It's like I'm able to forget about all the things that had me down at the start of the day, and the things I'm able to do with the code I'm working on make me smile and laugh -- all the time!

So I'm trying to figure out what it's all about. Why the slow start, and strong finish? Why can't I look forward to the work so that my day starts with excitement? If I know I'm heading for a day that I'm going to do great, fun things - then why can't I use that as motivation at the start of the day?

I have a feeling it's related to The Three-Martini Lunch.

Specifically, I'm not really sure what's happening at The Shop right now. Yes, there's work, for now, but it could all end on Aug 1 - when I deliver this app I'm working on. They have plans, but it's all still awaiting approval by upper-management, and that, to me, means that it's still up for debate as to what will be done.

If that's the case, then I could find myself maintaining one little clojure app - and some .NET code (shudder) in just no time at all.

Yet when I'm able to ignore all that... forget the place I'm working, and just focus on the problems at the keyboard, then I can start solving problems, and delivering value. That makes me happy. It's like work - the real act of solving probelems, using clojure and it's amazing toolset, allows me to forget the realities of my current situation.

Clojure is my mid-afternoon drinking.

It doesn't solve anything. It's just getting drunk to avoid the problems you're facing. Tomorrow you'll be sober, and it'll all come back to you. So you dive into the "bottle" (work) and don't come up for air.

I'm not sure this is the correct analogy, but I have a feeling it's close. I love the work, and the work is what I find enjoyment from. But the place seems to be having a hard time with focus, and that concerns me.

Not much I can do about it, so it's likely to continue for a while.

SSL in Jetty (Compojure)

Monday, July 6th, 2015

Clojure.jpgI've been working on a project at The Shop and it's a simple-ish clojure back-end with a Javascript/jQuery/Bootstrap front-end, and it's not too bad. Getting there. But recently, it was time to add in the user authentication code, and that meant passwords. That meant security, and simple HTTP requests weren't going to do. We had to have security. HTTPS. So I dove down the rabbit hole of getting Jetty 7 to run as SSL only within a compojure/ring application.

What I found was that there really wasn't a wealth of information on the subject. Most folks expected a front-end service like nginx to handle the SSL, and then forward on to the clojure app requests on simple port 8080. While this is possible, it also means that port 8080 is there listening, and it's possible to have an intercept on that port and all of a sudden we're not so secure.

Nope. It needed to be Jetty on SSL on port 8443. Period.

Get the Certificate File

No doubt, getting the initial certificate file is the hardest part for most folks, as it's all about money. But I suppose self-signed certificates are useful - to a point, but being in a commercial organization, it's nice to see that there's a real certificate file, and all I need to do is to convert it to a Java Key Store file. In this case, it started as a pox file - a PKCS formatted certificate. Then:

  $ keytool -importkeystore -srckeystore my_cert.pfx -srcstoretype pkcs12 \
      -destkeystore my_cert.jks -deststoretype jks -deststorepass MY_PASSWORD

where it's clear what the file names are, and passwords. This gets you a my_cert.jks file that's what you are going to need. The MY_PASSWORD is also important as it will have to be in the code as well to read the file.

Put the KeyStore File in resources/certs

Like everything additional in a Meiningen project, this needs to go into the resources directory, and for convention, I'm throwing it into resources/certs as it's not the only thing in my resources directory for this project, and a little separation is a good thing.

Convert the Code to Dual-Mode

Working with the following dependencies in the project.clj file:

    ;; command line option processing
    [org.clojure/tools.cli "0.2.2"]
    ;; web server
    [compojure "1.3.4"]
    [ring/ring-core "1.3.2"]
    [ring/ring-jetty-adapter "1.3.2"]
    [ring.middleware.jsonp "0.1.6"]
    [ring/ring-defaults "0.1.5"]

and assuming you have a it running on port 8080, it might look a little like this:

(ns myapp.main
  (:require [clojure.java.io :refer [resource input-stream]]
            [clojure.tools.cli :refer [cli]]
            [clojure.tools.logging :refer [error info infof]]
            [myapp.server :refer [app]]
            [ring.adapter.jetty :as jt])
  (:import java.security.KeyStore)
  (:gen-class))
 
  (defn handle-args
    "Function to parse the arguments to the main entry point of this
    project and do what it's asking. By the time we return, it's all
    done and over."
    [args]
    (let [[params [action]] (cli args
               ["-p" "--port" "Listen on this port"
                 :default 8080 :parse-fn #(Integer. %)]
               ["-v" "--verbose" :flag true])]
      (cond
        (= "web" action)
          (jt/run-jetty app { :port (:port params) }))
        :else
          (do
            (info "Welcome to My App!")
            (println "Welcome to My App!")))))
 
  (defn -main
    "Function to kick off everything and clean up afterwards"
    [& args]
    (with-error-handling (handle-args args)))

this will assume port 8080, and yet allow the command-line args to override this. In order to add the SSL component to this, we simply have to add a few options to Jetty:

  (defn handle-args
    "Function to parse the arguments to the main entry point of this
    project and do what it's asking. By the time we return, it's all
    done and over."
    [args]
    (let [[params [action]] (cli args
               ["-p" "--port" "Listen on this port"
                 :default 8080 :parse-fn #(Integer. %)]
               ["-s" "--ssl-port" "Listen on this port"
                 :default 8443 :parse-fn #(Integer. %)]
               ["-v" "--verbose" :flag true])]
      (cond
        (= "web" action)
          (jt/run-jetty app { :port (:port params)
                              :ssl? true
                              :ssl-port (:ssl-port params)
                              :keystore "resources/certs/my_cert.jks"
                              :key-password "MY_PASSWORD" }))
        :else
          (do
            (info "Welcome to My App!")
            (println "Welcome to My App!")))))

Start this guy now, and it'll answer on port 8080 for normal HTTP traffic, and port 8443 for HTTPS traffic. So far, so good. But there's a catch here, and we'll get to it soon.

Make it SSL Only

The real request was to make is SSL-only, so we to remove the port 8080 traffic, but we can't just tell Jetty not to run that one - we have to actively remove it. Thankfully, composure allows us this flexibility. If we make a function:

  (defn remove-non-ssl-connectors
    "Function to configure the Jetty instance to remove all non-SSL
    connectors so that there's **no way** to get into this service BUT
    by SSL (https)."
    [server]
    (doseq [c (.getConnectors server)]
      (when-not (or (nil? c)
                    (instance? org.eclipse.jetty.server.ssl.
                                    SslSelectChannelConnector c))
        (.removeConnector server c)))
    server)

then we can add that as an option to Jetty to tell it to run on it's configuration:

  (defn handle-args
    "Function to parse the arguments to the main entry point of this
    project and do what it's asking. By the time we return, it's all
    done and over."
    [args]
    (let [[params [action]] (cli args
               ["-p" "--port" "Listen on this port"
                 :default 8080 :parse-fn #(Integer. %)]
               ["-s" "--ssl-port" "Listen on this port"
                 :default 8443 :parse-fn #(Integer. %)]
               ["-v" "--verbose" :flag true])]
      (cond
        (= "web" action)
          (jt/run-jetty app { :configurator remove-non-ssl-connectors
                              :port (:port params)
                              :ssl? true
                              :ssl-port (:ssl-port params)
                              :keystore "resources/certs/my_cert.jks"
                              :key-password "MY_PASSWORD" }))
        :else
          (do
            (info "Welcome to My App!")
            (println "Welcome to My App!")))))

and now if you restart the app, it won't answer on port 8080, but it will still answer on port 8443 with HTTPS. Getting very close.

Making it Deployable in an Uberjar

The final wrinkle is that the :keystone option is a location in the filesystem of the key store file. That's not good for deployments because it means that while the keystore file will be packaged up in the uberjar, it's not going to be referenced that way - and it will have to also exist in the filesystem in the same relative location.

This stinks.

So I did some more digging, and ring-jetty had what I needed - the ability to pass it a java.security.KeyStore instance. Now I needed to read the keystone file from the jar, into an instance of that object, and pass it in.

Start with:

  (defn load-keystore
    "Function to load the SSL KeyStore from the resources so that it's ready
    to be used to run the SSL connections for Jetty. This is a preferred method
    to having just a path for locating the certificate, as this allows the cert
    to be _included_ in the uberjar itself."
    [loc pw]
    (if (and (string? loc) (string? pw))
      (doto (KeyStore/getInstance (KeyStore/getDefaultType))
        (.load (input-stream (resource loc)) (char-array pw)))))

which will load a keystone file from the uberjar - placed in the resources directory. This was a major find for me, and made it all possible. That it only takes a few lines of clojure and java is just amazing.

Now we can put it all together:

  (defn handle-args
    "Function to parse the arguments to the main entry point of this
    project and do what it's asking. By the time we return, it's all
    done and over."
    [args]
    (let [[params [action]] (cli args
               ["-p" "--port" "Listen on this port"
                 :default 8080 :parse-fn #(Integer. %)]
               ["-s" "--ssl-port" "Listen on this port"
                 :default 8443 :parse-fn #(Integer. %)]
               ["-v" "--verbose" :flag true])]
      (cond
        (= "web" action)
          (let [loc "certs/my_cert.jks"   ;; SSL keystore location
                pw "MY_PASSWORD"]         ;; ...and password
            (jt/run-jetty app { :configurator remove-non-ssl-connectors
                                :port (:port params)
                                :ssl? true
                                :ssl-port (:ssl-port params)
                                :keystore (load-keystore loc pw)
                                :key-password pw }))
        :else
          (do
            (info "Welcome to My App!")
            (println "Welcome to My App!")))))

Built a Nice Simple Clojure/Bootstrap/Google App

Tuesday, June 2nd, 2015

Clojure.jpg

Over the course of the last day and a half, I've been putting together a nice little test app using clojure and compojure on the back-end, and Bootstrap with Handsontable as the UI. It's simple, and not all that complicated, but it's nice in that it demonstrates a lot of the things that I often want to do in a simple application.

Then today one of the guys I was putting this together for suggested OAuth2 for authentication and then something internal for authorization. This turned out to be not all that bad with Google's Identity Service. They have a nice little JavaScript client that gets the auth token that you then feed to the calls to the service, and it, in turn, hits Google to make sure that this is a person that they say they are. You end up with an email address, and that can be used as the "username" for any authorization scheme you want.

In short, it's very slick. Works like a dream. I'm having a pretty good time with this.

better colors

Excellent JDBC4 Fix for ?-operators in Postgres

Thursday, May 28th, 2015

PostgreSQL.jpg

In Postgres 9.4, there are several JSONB operators that include a ? as part of the operator. This is nasty for the JDBC usage because the JDBC driver uses the question-mark as the substitute parameter place-holder for arguments. And there's no escaping of values in a JDBC PreparedStatement so what's a guy to do?

Sure, you can make a simple custom function to do this, but that's kinda wasteful, and it's not very transportable. Thankfully, I was reading the release notes for the 9.4 JDBC driver and saw this.

Using the latest, released JDBC driver in your project.clj file:

  [org.postgres/postgres "9.4-1201-jdbc41"]

you can pick up this pull request, and feature.

By simply doubling the question mark, it'll now reduce this to one ? and the operator will work. Very slick. So now in a clojure SQL statement, just put ?? when you need ? and you're good to go!

Fantastic! I've checked it and it works perfectly.

Slick Way to Add Array and JSON Support to Postgres Clojure JDBC

Thursday, May 28th, 2015

Clojure.jpg

The default JDBC behavior in clojure.java.jdbc is not to handle the additional Postgres data types like Array and JSON. But you do have the protocols that you can use to override the behavior of how to store and read things - and this article looks to set this up for both these data types.

The idea is to look at the data type on both the write and read, and do the JSON mapping and we're done. The code is very simple:

  (ns our-app.jdbc.json
    "Inspired by http://www.niwi.be/2014/04/13/
     postgresql-json-field-with-clojure-and-jdbc/"
    (:require [clojure.java.jdbc :as jdbc]
              [cheshire.core :as json])
    (:import org.postgresql.util.PGobject))
 
  (defn value-to-json-pgobject [value]
    (doto (PGobject.)
      (.setType "json")
      (.setValue (json/generate-string value))))
 
  (extend-protocol jdbc/ISQLValue
    clojure.lang.IPersistentMap
    (sql-value [value] (value-to-json-pgobject value))
 
    clojure.lang.IPersistentVector
    (sql-value [value] (value-to-json-pgobject value)))
 
    clojure.lang.IPersistentList
    (sql-value [value] (value-to-json-pgobject value)))
 
    clojure.lang.LazySeq
    (sql-value [value] (value-to-json-pgobject value)))
 
  (extend-protocol jdbc/IResultSetReadColumn
    PGobject
    (result-set-read-column [pgobj metadata idx]
      (let [type  (.getType pgobj)
            value (.getValue pgobj)]
        (case type
          "json" (json/parse-string value true)
          :else value))))

Where I have chosen to use Cheshire, as opposed to clojure.data.json. It's a matter of taste, I'll agree, but we're using Cheshire all the time anyway, so the cost is nothing, and it's consistently parsed in the same way.

Very cool.

Got a Call… from a Guy… about Clojure…

Wednesday, May 13th, 2015

Clojure.jpg

Yesterday I got a quick note from an ex-co-worker about how he might go about writing a micro-service in Clojure. Seems his CTO is a big functional programmer, and as such, he wanted my friend to write something in Clojure. So he reached out to the only person he knew that had actually done that, and so I got the call.

I was happy to help him along. I sent him links to Gists about how a server.clj file might look, as well as how to handle JSONP - assuming that the calls were coming from jQuery, and basically, just make sure that things got off on the right foot.

I also gave him some pointers about the standard libraries that I've been using for clojure work - cli-time, composure, cheshire, carmine, etc. They make it a lot easier, and a lot of folks are using them - so it's good to get started using the standard stuff right off the bat.

We talked a lot about getting started, and how the RESTful stuff was going to look, and then he said he'd dig into it when he had time. But the most impressive part was the recognition that by simply saying "Clojure Developers Wanted" - the CTO knew he'd get all kinds of hits, and all kinds of street cred to make the most simpler.

Smart man... deserves to be in that position.

Heroku Adds Redis

Tuesday, May 12th, 2015

Heroku

This afternoon I saw a tweet from Heroku about them adding Redis to the add-ons for their service. This just a few days after their announcement that Postgres was available for the free tier, and the new "Free" tier for apps. They are getting aggressive with what services they are providing. This makes a ton of good sense to me, as I'm a huge fan of redis from clojure, and this makes all the experience I've got in building apps directly transferable.

While I know the Free tier isn't all that great, the idea that there is a Free tier is amazing, and it means that I can write something and throw it up there, and as it's needed, I can scale it up. Very cool. They also have a hobbyist tier that's only something like $8/mo. - similar to GitHub.

If I needed to be firing up a web service, it'd be clojure, redis, and postgres - all on Heroku. What an amazing service.

Interesting Test System for Clojure: test.check

Tuesday, May 12th, 2015

Clojure.jpg

A really nice co-worker of mine has been doing some interesting work with random number generators for the test.check clojure library, and I decided this morning to have a look and see what it's all about. After all, I can't imagine something a lot better than the standard clojure.test as it's got everything I've ever needed. But Bret has used it, and Gary has to have started using it for a reason, so I wanted to find out what that was.

I'm very glad I did.

The problem that I've run into with simple unit test frameworks is that they allow you to test specific inputs and outputs. If you want something to run 100 times with random inputs, you could do it, but it wasn't easy. Certainly, the question as to Why? would arise unless you had a testing framework that thought not in terms of specific inputs and outputs - but in the structure of the data.

If we wanted to test a sorting function, then we could check to see if the act of operating on a sorted sequence was the same as the input sequence, then we'd need to be able to make arbitrary sequences, and then verify the structure of the data as well. This is then more like generators and properties - which is what test.check calls them.

In this regard, it's easy to see how you'd use this to test the limits of a function without worrying about making sure you got all the edge conditions in the unit tests. It's the shotgun approach, and in some cases, it can really make short work of finding the edge cases if it's not immediately easy to see them from the code.

There's even a simple macro to allow these tests to run alongside the clojure.test unit tests - which is exactly how I can see using it. There are some things that the unit testing system really works well for, and there are others that a shotgun approach would really help for as well. So a hybrid approach seems to be the best move.

Certainly interesting, and I'll put it into a project soon.

On a Roll with jQuery and Bootstrap

Wednesday, May 6th, 2015

JQuery Framework

OK, since I was on a roll with the work I was doing at The Shop, it made sense to keep going and apply it to putting a web front-end on the CryptoQuip solver a friend and I have been working on. The solver is all Clojure, and so it took just a few tweaks to get all the static files in-place and ready, and then getting the POST working was really the challenge.

Turns out, the way to get it to work was:

  function solveThePuzzle() {
    console.log("attempting to solve the puzzle");
    // get what we need for the call...
    var inp = {};
    inp.cyphertext = $("#plaintext").val();
    inp.clue = {};
    inp.clue[$("#key1").val()] = $("#key2").val();
    // make the call to solve the puzzle
    $.ajax({type: "POST",
            url: "/solve",
            processData: false,
            contentType: 'application/json',
            data: JSON.stringify(inp),
            success: function(resp) {
              var cont = '<div class="alert alert-success" role="alert">';
              cont += '<strong>Solved:</strong> ' + resp.plaintext;
              cont += '</div>';
              $("#status").replaceWith(cont);
            }
    })
  }

and the results are very nice:

better colors

Moved to JDK 8 for Clojure Work

Friday, April 24th, 2015

java-logo-thumb.png

It had to happen... JDK 7 is end-of-life by Oracle, and while I imagine that this is not what is really going to happen to all the JDK 7 code out in the wild, it's enough of a pressure that it's time to move off it and onto JDK 8 for real.

I've got a nice switcher tool for the active version of the JDK for each shell, and I just switched it to 1.8, and restarted the REPL and web server to pick up the changes. It's not a ton of work, but it needed to be done, and I need to stay on JDK 8 moving forward.

It's funny... in writing this post, I realized that I can remember when JDK 1.5 was "new" and untested... and 1.6... and 1.7... it's just how things go. Marching onward.