What's it all about, Alfie?

In Postgres 9.4, there are several JSONB operators that include a ? as part of the operator. This is nasty for the JDBC usage because the JDBC driver uses the question-mark as the substitute parameter place-holder for arguments. And there's no escaping of values in a JDBC PreparedStatement so what's a guy to do?

Sure, you can make a simple custom function to do this, but that's kinda wasteful, and it's not very transportable. Thankfully, I was reading the release notes for the 9.4 JDBC driver and saw this.

Using the latest, released JDBC driver in your project.clj file:

  [org.postgres/postgres "9.4-1201-jdbc41"]

you can pick up this pull request, and feature.

By simply doubling the question mark, it'll now reduce this to one ? and the operator will work. Very slick. So now in a clojure SQL statement, just put ?? when you need ? and you're good to go!

Fantastic! I've checked it and it works perfectly.

The default JDBC behavior in clojure.java.jdbc is not to handle the additional Postgres data types like Array and JSON. But you do have the protocols that you can use to override the behavior of how to store and read things - and this article looks to set this up for both these data types.

The idea is to look at the data type on both the write and read, and do the JSON mapping and we're done. The code is very simple:

  (ns our-app.jdbc.json
    "Inspired by http://www.niwi.be/2014/04/13/
     postgresql-json-field-with-clojure-and-jdbc/"
    (:require [clojure.java.jdbc :as jdbc]
              [cheshire.core :as json])
    (:import org.postgresql.util.PGobject))
 
  (defn value-to-json-pgobject [value]
    (doto (PGobject.)
      (.setType "json")
      (.setValue (json/generate-string value))))
 
  (extend-protocol jdbc/ISQLValue
    clojure.lang.IPersistentMap
    (sql-value [value] (value-to-json-pgobject value))
 
    clojure.lang.IPersistentVector
    (sql-value [value] (value-to-json-pgobject value)))
 
    clojure.lang.IPersistentList
    (sql-value [value] (value-to-json-pgobject value)))
 
    clojure.lang.LazySeq
    (sql-value [value] (value-to-json-pgobject value)))
 
  (extend-protocol jdbc/IResultSetReadColumn
    PGobject
    (result-set-read-column [pgobj metadata idx]
      (let [type  (.getType pgobj)
            value (.getValue pgobj)]
        (case type
          "json" (json/parse-string value true)
          :else value))))

Where I have chosen to use Cheshire, as opposed to clojure.data.json. It's a matter of taste, I'll agree, but we're using Cheshire all the time anyway, so the cost is nothing, and it's consistently parsed in the same way.

Very cool.

Today I spent a good bit of the day trying to figure out how to authenticate with Twitter's OAuth 1.0 system, and I think I'm getting close, but I'm still a bit away because I don't control these accounts, and the sheer volume of ways to authenticate on Twitter is daunting. Let allne the different APIs.

There is the client-facing Tweets API, and then there's the Ad Server API, and it's not at all clear that there needs to be different authentication schemes for these APIs. But it should be clear that access to one set of APIs probably should not guarantee access to another set - and maybe they handle that in the authorization, but it's not clear from the docs I'm reading.

And speaking of docs, wow... these are really something else. There are at least four ways to authenticate, but they ask people to use libraries - that they don't provide. Sadly, I don't see one that does 100% of what I need, but I do see an OAuth 1.0 library, but the Client ID and Secret are nowhere to be found on their site.

So clearly, I'm missing something.

What I believe is that you have to create an App that then gets you the redirect URL and ID and Secret. There were none defined to base a new one on, and so I sent off an email to the Twitter representative to see if this was, indeed the preferred way.

While I was waiting, I decided to try and make an app. Yet in order to do that, you need to assign a mobile phone number to the Twitter account, and I can't really do that because the account is not mine. SO I sent another email to the relationship folks in The Shop about that.

In short, it's just a waiting game. But it's also so much more of a mess than the other systems I've been integrating with. Wow...

Yesterday I got a quick note from an ex-co-worker about how he might go about writing a micro-service in Clojure. Seems his CTO is a big functional programmer, and as such, he wanted my friend to write something in Clojure. So he reached out to the only person he knew that had actually done that, and so I got the call.

I was happy to help him along. I sent him links to Gists about how a server.clj file might look, as well as how to handle JSONP - assuming that the calls were coming from jQuery, and basically, just make sure that things got off on the right foot.

I also gave him some pointers about the standard libraries that I've been using for clojure work - cli-time, composure, cheshire, carmine, etc. They make it a lot easier, and a lot of folks are using them - so it's good to get started using the standard stuff right off the bat.

We talked a lot about getting started, and how the RESTful stuff was going to look, and then he said he'd dig into it when he had time. But the most impressive part was the recognition that by simply saying "Clojure Developers Wanted" - the CTO knew he'd get all kinds of hits, and all kinds of street cred to make the most simpler.

Smart man... deserves to be in that position.

So I just got out of a meeting with my manager's manager. My manager is trying to rebuild an existing system - as opposed to simply augmenting it, and the way he's going about doing this is by using AWS services and a flock of 11 consultants. They are skipping what is, and jumping right to what he wants, and in so doing, leaving all the employees of the company that might offer help, support, and certainly ongoing maintenance - completely in the dark.

I've asked for details about the project I'm supposed to be on, and the response was always "None of your concern." OK... you're the manager. After repeated attempts to talk to HR, and seeing nothing done about it, I've simply taken the stand that I'll be very up-front about the fact that I'm doing nothing, as long as I'm given nothing to do. That way people can choose to see that as an anomaly - or not. Their choice.

I'm being paid either way. I'm very transparent with HR so that should it come back to me that I was wasting company resources (my time), that would be documented as not true. So I'm covered.

I don't think that using AWS is wrong. It's just a different approach than what I know the CTO has expressly stated: It just costs too much. It's more, but if you factor in the cost of having a datacenter, and racks, and machines, and people to maintain them, and then people to install, upgrade, monitor and fix the services like AWS offers, well... I'm not certain that all cases are clearly cheaper - one way or the other. I really think that the costs are coming down, and the service levels are to the point that you need to consider it on a case-by-case basis to be 100% certain.

But that's just my opinion, and not the CTO's. And he has made himself very clear on the matter. So for my manager (a consultant) to come in and make the plan, and hire the consultants to execute this plan that is 100% AWS... well... I knew it'd be a point of contention, and told my manager this when he first showed me his plans.

"Not your concern. That's something I'll make happen."

OK, boss... I tried to tell you. As my GrandPa Bowen said:

Those who don't listen, must feel.
-- Don Bowen

so he's going to have to feel it.

Part of that seems to be happening in the meeting I just got out of. The CTO said that AWS was not going to happen. That the enhancement of the existing system was the job, and that I was going to be brought in - as well as the other employees that have a solid understanding of the existing system. Makes good sense to me.

But it's 180 degrees from what my manager has been doing, planning, and moving towards.

So starting tomorrow I can see a new round in the fight. There's no way that the current plan is going to be allowed. So what value are the AWS consultants? My manager is being forced to bring in people he didn't want to bring in - and actively excluded and rejected. How hard are we going to work for this guy? Why should we? He was ready to drop us all - and told me as much less than a week ago. So how is he going to make this happen?

At the same time, the first round went to him in hiring the consultants, and starting this work... but this last round went to the CTO who shut down a $2 million deployment because we already had 90% of it in-house, paid for, and supported. It would be a colossal waste of money.

I'm no fool... this fight isn't over, just a temporary lull in the action. Soon there will be more slings and arrows, and we that are caught in the middle will no doubt soon tire of this, and let the (non-)Titans battle it out amongst themselves.

But is this any way to run a company?

This afternoon I saw a tweet from Heroku about them adding Redis to the add-ons for their service. This just a few days after their announcement that Postgres was available for the free tier, and the new "Free" tier for apps. They are getting aggressive with what services they are providing. This makes a ton of good sense to me, as I'm a huge fan of redis from clojure, and this makes all the experience I've got in building apps directly transferable.

While I know the Free tier isn't all that great, the idea that there is a Free tier is amazing, and it means that I can write something and throw it up there, and as it's needed, I can scale it up. Very cool. They also have a hobbyist tier that's only something like $8/mo. - similar to GitHub.

If I needed to be firing up a web service, it'd be clojure, redis, and postgres - all on Heroku. What an amazing service.

A really nice co-worker of mine has been doing some interesting work with random number generators for the test.check clojure library, and I decided this morning to have a look and see what it's all about. After all, I can't imagine something a lot better than the standard clojure.test as it's got everything I've ever needed. But Bret has used it, and Gary has to have started using it for a reason, so I wanted to find out what that was.

I'm very glad I did.

The problem that I've run into with simple unit test frameworks is that they allow you to test specific inputs and outputs. If you want something to run 100 times with random inputs, you could do it, but it wasn't easy. Certainly, the question as to Why? would arise unless you had a testing framework that thought not in terms of specific inputs and outputs - but in the structure of the data.

If we wanted to test a sorting function, then we could check to see if the act of operating on a sorted sequence was the same as the input sequence, then we'd need to be able to make arbitrary sequences, and then verify the structure of the data as well. This is then more like generators and properties - which is what test.check calls them.

In this regard, it's easy to see how you'd use this to test the limits of a function without worrying about making sure you got all the edge conditions in the unit tests. It's the shotgun approach, and in some cases, it can really make short work of finding the edge cases if it's not immediately easy to see them from the code.

There's even a simple macro to allow these tests to run alongside the clojure.test unit tests - which is exactly how I can see using it. There are some things that the unit testing system really works well for, and there are others that a shotgun approach would really help for as well. So a hybrid approach seems to be the best move.

Certainly interesting, and I'll put it into a project soon.

This morning I have nothing to do at The Shop, and so I decided to catch up on all the Faces work I needed to do in Apple's Photos app. It's getting to match the iOS Photos app a lot more, and while there are a few things to get used to, it's not bad, and if you just give it a few minutes, you're likely to find what you need right there in the app.

For example, setting the 'default' picture in a Faces collection used to be a scrolling and selecting deal - could be hard with a lot of faces, but now you look at all the pictures for a given face, and then right-click to set the one you want. Very simple. Very easy.

It's really a great little app.

I wanted to make a head-shot for my brother, and Faces had picked out a nice one, but it didn't allow me to set it as the contact. But it did allow me to duplicate the one picture... crop it... save it... and then the Contacts app allowed me to use that from the Faces for this contact. I mean it's just super simple.

At the same time, I have a somewhat unique problem in that two of my kids are trans. This means their names have changed - and not just a little. Once I got all the pictures under one name, then I simply clicked on the name in the Faces view, and changed it. Simple. And for me, that's a lot of pictures. So much easier than having to do each picture.

Finally, the camera metadata. Wow! I looked a a picture we all took a while back. It had all the metadata - not just the geolocation tags, but also the camera, the conditions... it's crazy what a camera will stuff into the metadata, and Photos accurately stores all that.

Very slick tool. Very. Thanks, Apple.

I had a meeting this morning with my manager and I was reminded of the verse:

For what will it profit a man if he gains the whole world and forfeits his soul?
-- Matthew 16:26

My manager is a consultant - been here two weeks - had a plan on Day 1 to change how The Shop was going to handle all data storage and reporting tasks. Based on what? Nothing. His experience. Not that he couldn't make it work, but the agreement to bring him in and trust him to implement the solution without knowing what the long-term costs would be is simply making a bad deal.

I understand the problems businesses can get into. I've been there. But I'm glad to say that I got out before I had to make any of those kinds of deals.

This guy is going to achieve what he wants, and give to management what they want. And when they get it, they will look at it, and say "This wasn't worth it." And they will be right.

But it will be too late.

I read on twitter about a bug in ZooKeeper found by the folks at PagerDuty. The story is quite remarkable, and reminds me that some companies still invest the time to get to the bottom of things - as opposed to just putting it off once a work-around is found. The level of detail and investigation they did is simply... inspiring. I'm stunned.

I'd like to work at a place where that kind of stuff is done. Not all the time, of course, as I'm sure they were all glad that it was over when it was over, but to be able to devote the time to solving the problem as opposed to stopping it - that's nice.

I don't imagine anything I'd do would hit this series of bugs. Too many components that simply aren't something I'd ever want to use. But it's nice to know someone is there digging deep.