Archive for the ‘Open Source Software’ Category

Excellent JDBC4 Fix for ?-operators in Postgres

Thursday, May 28th, 2015

PostgreSQL.jpg

In Postgres 9.4, there are several JSONB operators that include a ? as part of the operator. This is nasty for the JDBC usage because the JDBC driver uses the question-mark as the substitute parameter place-holder for arguments. And there's no escaping of values in a JDBC PreparedStatement so what's a guy to do?

Sure, you can make a simple custom function to do this, but that's kinda wasteful, and it's not very transportable. Thankfully, I was reading the release notes for the 9.4 JDBC driver and saw this.

Using the latest, released JDBC driver in your project.clj file:

  [org.postgres/postgres "9.4-1201-jdbc41"]

you can pick up this pull request, and feature.

By simply doubling the question mark, it'll now reduce this to one ? and the operator will work. Very slick. So now in a clojure SQL statement, just put ?? when you need ? and you're good to go!

Fantastic! I've checked it and it works perfectly.

Slick Way to Add Array and JSON Support to Postgres Clojure JDBC

Thursday, May 28th, 2015

Clojure.jpg

The default JDBC behavior in clojure.java.jdbc is not to handle the additional Postgres data types like Array and JSON. But you do have the protocols that you can use to override the behavior of how to store and read things - and this article looks to set this up for both these data types.

The idea is to look at the data type on both the write and read, and do the JSON mapping and we're done. The code is very simple:

  (ns our-app.jdbc.json
    "Inspired by http://www.niwi.be/2014/04/13/
     postgresql-json-field-with-clojure-and-jdbc/"
    (:require [clojure.java.jdbc :as jdbc]
              [cheshire.core :as json])
    (:import org.postgresql.util.PGobject))
 
  (defn value-to-json-pgobject [value]
    (doto (PGobject.)
      (.setType "json")
      (.setValue (json/generate-string value))))
 
  (extend-protocol jdbc/ISQLValue
    clojure.lang.IPersistentMap
    (sql-value [value] (value-to-json-pgobject value))
 
    clojure.lang.IPersistentVector
    (sql-value [value] (value-to-json-pgobject value)))
 
    clojure.lang.IPersistentList
    (sql-value [value] (value-to-json-pgobject value)))
 
    clojure.lang.LazySeq
    (sql-value [value] (value-to-json-pgobject value)))
 
  (extend-protocol jdbc/IResultSetReadColumn
    PGobject
    (result-set-read-column [pgobj metadata idx]
      (let [type  (.getType pgobj)
            value (.getValue pgobj)]
        (case type
          "json" (json/parse-string value true)
          :else value))))

Where I have chosen to use Cheshire, as opposed to clojure.data.json. It's a matter of taste, I'll agree, but we're using Cheshire all the time anyway, so the cost is nothing, and it's consistently parsed in the same way.

Very cool.

Heroku Adds Redis

Tuesday, May 12th, 2015

Heroku

This afternoon I saw a tweet from Heroku about them adding Redis to the add-ons for their service. This just a few days after their announcement that Postgres was available for the free tier, and the new "Free" tier for apps. They are getting aggressive with what services they are providing. This makes a ton of good sense to me, as I'm a huge fan of redis from clojure, and this makes all the experience I've got in building apps directly transferable.

While I know the Free tier isn't all that great, the idea that there is a Free tier is amazing, and it means that I can write something and throw it up there, and as it's needed, I can scale it up. Very cool. They also have a hobbyist tier that's only something like $8/mo. - similar to GitHub.

If I needed to be firing up a web service, it'd be clojure, redis, and postgres - all on Heroku. What an amazing service.

Interesting Test System for Clojure: test.check

Tuesday, May 12th, 2015

Clojure.jpg

A really nice co-worker of mine has been doing some interesting work with random number generators for the test.check clojure library, and I decided this morning to have a look and see what it's all about. After all, I can't imagine something a lot better than the standard clojure.test as it's got everything I've ever needed. But Bret has used it, and Gary has to have started using it for a reason, so I wanted to find out what that was.

I'm very glad I did.

The problem that I've run into with simple unit test frameworks is that they allow you to test specific inputs and outputs. If you want something to run 100 times with random inputs, you could do it, but it wasn't easy. Certainly, the question as to Why? would arise unless you had a testing framework that thought not in terms of specific inputs and outputs - but in the structure of the data.

If we wanted to test a sorting function, then we could check to see if the act of operating on a sorted sequence was the same as the input sequence, then we'd need to be able to make arbitrary sequences, and then verify the structure of the data as well. This is then more like generators and properties - which is what test.check calls them.

In this regard, it's easy to see how you'd use this to test the limits of a function without worrying about making sure you got all the edge conditions in the unit tests. It's the shotgun approach, and in some cases, it can really make short work of finding the edge cases if it's not immediately easy to see them from the code.

There's even a simple macro to allow these tests to run alongside the clojure.test unit tests - which is exactly how I can see using it. There are some things that the unit testing system really works well for, and there are others that a shotgun approach would really help for as well. So a hybrid approach seems to be the best move.

Certainly interesting, and I'll put it into a project soon.

Wild Bug in ZooKeeper

Friday, May 8th, 2015

ZooKeeper

I read on twitter about a bug in ZooKeeper found by the folks at PagerDuty. The story is quite remarkable, and reminds me that some companies still invest the time to get to the bottom of things - as opposed to just putting it off once a work-around is found. The level of detail and investigation they did is simply... inspiring. I'm stunned.

I'd like to work at a place where that kind of stuff is done. Not all the time, of course, as I'm sure they were all glad that it was over when it was over, but to be able to devote the time to solving the problem as opposed to stopping it - that's nice.

I don't imagine anything I'd do would hit this series of bugs. Too many components that simply aren't something I'd ever want to use. But it's nice to know someone is there digging deep.

Some days things work out…

Friday, May 8th, 2015

Great News

This morning I saw on HipChat messages from two folks at The Shop:

Chris M. said that you helped him to setup our new hardware. Guess what? I just ran a test ETL in new hardware, it is 5 times faster. The full MMS ETL cycle takes about 2-2.5 hours. In new server, it takes 0.5 hour. THANK YOU for whatever you helped Chris M. 🙂
-- Okji

and from Chris:

So, you were 100% right on the hardware specs for the Pentaho stuff.

Okji is running initial stuff now and it's insanely fast.

thank you for dealing with a stubborn asshat me through the ordeal and lighting a fire under my rear.

tbh I'd probably be flogging the dead virtualization horse at this point w/o that back and forth we had.

So yeah, thanks 🙂
-- Chris

I don't often have people sending me these kinds of notes for work I did for them. The problem was simple and obvious - to me, but if you have never seen the other way, you often think your way is the only way. I've seen it a million times. The point is to get them started, let them see, and then be very gracious when they thank you.

That last part is key.

You want to build up everyone - not just yourself. Help others feel good about what they did, and they will want to work with - or for - you again. It's simple. Who wants to be around someone that makes them feel bad about themselves? No one I know.

Interestingly enough, this is going to make things work a lot better for the short-term goals. There's a consultant at the shop, and this is going to make his Uber Plan much less attractive, and necessary.

Postgres has Added UPSERT

Friday, May 8th, 2015

PostgreSQL.jpg

One of the things I've always wanted in Postgres is the UPSERT - an INSERT that would update certain fields if the row (defined by the primary key) already existed. In the past, I've had to implement this as a custom function (stored procedure) in pl/sql, by checking for the existence of the row and then doing an UPDATE, or failing that, do an INSERT. It's workable, and it's not horrible, but it's also something that's in several other databases, and I wanted it in my database. 🙂

This morning I read a tweet that said:

better colors

and read the commit log message that described it. I love it! This is exactly what I've been hoping for.

The only question now is - When is it released?

On a Roll with jQuery and Bootstrap

Wednesday, May 6th, 2015

JQuery Framework

OK, since I was on a roll with the work I was doing at The Shop, it made sense to keep going and apply it to putting a web front-end on the CryptoQuip solver a friend and I have been working on. The solver is all Clojure, and so it took just a few tweaks to get all the static files in-place and ready, and then getting the POST working was really the challenge.

Turns out, the way to get it to work was:

  function solveThePuzzle() {
    console.log("attempting to solve the puzzle");
    // get what we need for the call...
    var inp = {};
    inp.cyphertext = $("#plaintext").val();
    inp.clue = {};
    inp.clue[$("#key1").val()] = $("#key2").val();
    // make the call to solve the puzzle
    $.ajax({type: "POST",
            url: "/solve",
            processData: false,
            contentType: 'application/json',
            data: JSON.stringify(inp),
            success: function(resp) {
              var cont = '<div class="alert alert-success" role="alert">';
              cont += '<strong>Solved:</strong> ' + resp.plaintext;
              cont += '</div>';
              $("#status").replaceWith(cont);
            }
    })
  }

and the results are very nice:

better colors

Building with jQuery and Bootstrap

Wednesday, May 6th, 2015

JQuery Framework

This morning I decided that I wanted to be able to start doing decent web site design. This means getting a lot better at jQuery - even though it's old, it's still useful and it's out there, and learning Bootstrap. Now the latter was really the influence of the folks at Groupon, because if they used it, then it had to be a good tool. Sure, they improved a lot on it, and they were really fond of their whitespace, but I want to get to the point that I can make a decent web site with decent tools so that it's not like I'm just HTML/CSS and nothing else.

So this morning I did this:

better colors

it's not amazing, but for about 3 hrs, it's not bad, and it's backed by the data, and I've got a good start on the way to make it all work in a Clojure app. So that's what I'm going to write up, so I don't forget all the issues.

Get The Tools

First off, go get the jQuery and Bootstrap downloads. I'm not going to be fiddling with the code, so I don't need to build it, just get it and use it. One important thing I noted was that Bootstrap 3.3.4 was very particular about the version of jQuery that needed to be used. Specifically, 1.11.2.

Start in the resources/ directory of the Clojure project and make a public directory where all the static assets will be served from. Anything in resources/ will be packaged up into the 'uberjar', so once there, it's save to develop and deploy on these files. When you are done, make sure the filesystem looks like this:

  resources
      +- public
          +- index.html
          +- css
          |   +- bootstrap.min.css
          |   +- bootstrap-theme.min.css
          |   +- theme.css
          +- js
              +- bootstrap.min.js
              +- jquery-1.11.2.min.js

where the contents of the theme.css is very simple and yet needs to be there to make Bootstrap look decent:

  body: {
    padding-top: 70px;
    padding-bottom: 30px;
  }
 
  .theme-dropdown .dropdown-menu {
    position: static;
    display: block;
    margin-bottom: 20px;
  }
 
  .theme-showcase > p > .btn {
    margin: 5px 0;
  }
 
  .theme-showcase .navbar.container {
    width: auto;
  }

In the index.html there will be a few places where you need to match the names of these -showcase - and possibly make the names unique to your project.

Setting Up the Home Page

Now that you have the tools copied into the right places, we need to throw together the boilerplate for the home page. This is pretty easy, and Bootstrap examples all have it as clear as day for you:

  <!DOCTYPE html>
  <html lang="en">
    <head>
      <meta charset="utf-8">
      <meta http-equiv="X-UA-Compatible" content="IE=edge">
      <meta name="viewport" content="width=device-width, initial-scale=1">
      <!-- The above 3 meta tags *must* come first in the head;
           any other head content must come *after* these tags -->
      <title>Bartender</title>
      <!-- Bootstrap core CSS -->
      <link href="/css/bootstrap.min.css" rel="stylesheet">
      <!-- Bootstrap theme -->
      <link href="/css/bootstrap-theme.min.css" rel="stylesheet">
      <link href="/css/theme.css" rel="stylesheet">
    </head>
 
    <body role="document">
 
 
 
      <!-- Bootstrap core JavaScript
      ================================================== -->
      <!-- Placed at the end of the document so the pages load faster -->
      <script src="/js/jquery-1.11.2.min.js"></script>
      <script src="/js/bootstrap.min.js"></script>
      <script src="/js/bartender.js"></script>
      <script type="text/javascript">
        // when we are all done loading the page, hit the data...
        $( document ).ready(reloadAllData());
        // $( window ).load(reloadAllData());
      </script>
    </body>
  </html>

In the middle there, you can put in all the things you want. For me it was a simple menu and a table in a container. All this is on the Bootstrap examples and it's easy to pick-n-choose.

Wiring it in with Ring

The other big thing was to have it all work from the Clojure web project. There are a few things that needed to be done here, but in the end, they really aren't all that hard, and the results are quite nice.

First, make sure you have the necessary ring components in the project.clj:

  [ring/ring-core "1.3.2"]
  [ring/ring-jetty-adapter "1.3.2"]
  [ring.middleware.jsonp "0.1.6"]

and then make sure to require:

  [compojure.route :as route]
  [ring.util.response :as resp]

so that you can add the route:

  (GET "/" []
    (resp/redirect "/index.html"))

and then at the end of all the routes, add:

  (route/resources "/")
  (route/not-found "<h1>Page not Found!</h1>")

where the resources function specifies where to find the static resources, like the index.html.

At this point, restart the web server, and things should just work. Not very interesting, but the log will show the static resources being requested, and that verifies that they are found, and loaded, and you can see this all from the javascript console as well.

Adding the Database Access

At this point, we need to add the useful stuff. To keep things separate, I created a separate bartender.js to hold all the javascript I was going to write, and for now, that is simply:

  /*
   * Function to add a single row to the 'last-import-times' table on the
   * main page. This will be called for each key/value pair returned from
   * the RESTful API on the server, and it's just a simple way to make it
   * easy to see what's loaded up in the service.
   */
  function addLastImportRow(src, vals) {
    var dates = vals.report_date
    var tr = '<tr>';
    tr += '<td>' + src + '</td>';
    tr += '<td>' + vals.generated_at + '</td>';
    tr += '<td>' + dates[dates.length - 1].substring(0,10) + '</td>';
    tr += '<td>' + dates[0].substring(0,10) + '</td>';
    tr += '</tr>';
    $("#last-import-times").append(tr);
  }
 
  /*
   * Function to reload the data in the 'last-import-times' table on the
   * main page. This will clear out what's there, and hit the endpoint that
   * will bring back all the data it needs to properly make the page.
   */
  function refreshLastImportTable() {
    console.log("refreshing the last-import data");
    // remove all the non-header rows in the table
    $("#last-import-times tr").remove();
    // make the call to get the data from Bartender
    $.getJSON("/v1/latest-imports", function(data) {
      $.each(data, function(k, v) {
        addLastImportRow(k, v);
      })
    })
  }
 
  /*
   * Simple function to be called when the page is reloaded and we need to
   * update ALL the data on the page.
   */
  function reloadAllData() {
    refreshLastImportTable();
  }

There are some nice things here that I learned. First, jQuery may be old, but it's awfully nice for simple things like I'm building. Sure, if you want to make a single-page app on the web, it might be weak, but for me, it's very nice. And stable.

It's easy to ask for JSON from the server, and then have a function that simply takes each key/value pair and puts it in the emptied table. In the HTML I put an id on the table to make it easy to find, and that's all that I needed. I cleared it out, then one by one, I added the new rows back in. Works like a charm.

With this, I should be able to start making decent looking web sites. Much more to learn.

Moved to Adium 1.5.11b2

Wednesday, May 6th, 2015

Adium.jpg

The nightly builds of Adium 1.5.11 are nice, but they are getting a little unstable with the new OAuth2 GTalk credentials - like having to re-do them when awaking from sleep. So I found that the latest Adium 1.5.11 beta is pretty recent (2015-04-16), and it works with all the services I need, and I've kinda given up on MSN coming back to life... so why go through the pain of the bleeding-edge?

None that I can think of.

So it's back to the straight beta for now. Nothing against the nightlies, and I may go back if they get MSN and GTalk stable, but it's just not worth it today.