What's it all about, Alfie?

This morning I noticed that Clojure 1.10.0-RC1 was out, and that it was setting a minimum of JDK 1.8 as the required Java version. That's not a problem, as that's what I've been using, but I thought it might be a good thing to get the lates versions of the JDK installed on my box, and make sure that JDK 10 and JDK 11 work with the setjdk Bash function I wrote to make it easy to change from version to version in the environment.

So I went to the java.com web site, and first noticed that the current version they are suggesting is JDK 1.8 - and that really surprised me. I have known that JDK 9, 10, and 11 have been out for a bit, but to see that Oracle is suggesting that a new user install JDK 1.8.0_181 - that's just a little surprising.

Also, they made it a lot harder to find the other versions of the JDK - maybe that's because there was more confusion than necessary for folks just looking for the latest JDK - but therein is kinda my concerned - JDK 1.8. Still... I found it and was able to get the latest JDK 1.8.0_181, JDK 10.0.2, and JDK 11 - they have clearly decided to change the versioning, which is OK with me - but it means that I really need to make sure that I check the setjdk function when I install these guys.

When I got them installed, they all looked in place:

  peabody{drbob}516: ls -lsa /Library/Java/JavaVirtualMachines/
  total 0
  0 drwxr-xr-x  13 root  wheel  416 Oct 11 10:29 ./
  0 drwxr-xr-x   5 root  wheel  160 Sep 24 19:00 ../
  0 drwxr-xr-x   3 root  wheel   96 Jun 29  2011 1.6.0_26-b03-383.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Oct 11 10:29 jdk-10.0.2.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Oct 11 10:29 jdk-11.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Feb  5  2013 jdk1.7.0_13.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Oct 16  2013 jdk1.7.0_45.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Jan 17  2014 jdk1.7.0_51.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Mar 20  2015 jdk1.7.0_75.jdk/
  0 drwxr-xr-x   3 root  wheel   96 May  1  2017 jdk1.8.0_131.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Oct  2  2017 jdk1.8.0_144.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Oct 11 10:28 jdk1.8.0_181.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Mar 20  2015 jdk1.8.0_40.jdk/

and then a quick check of the setjdk script:

  peabody{drbob}504: setjdk 10
  peabody{drbob}505: echo $JAVA_HOME
  /Library/Java/JavaVirtualMachines/jdk-10.0.2.jdk/Contents/Home
  peabody{drbob}506: setjdk 11
  peabody{drbob}507: echo $JAVA_HOME
  /Library/Java/JavaVirtualMachines/jdk-11.jdk/Contents/Home
  peabody{drbob}508: setjdk 1.8
  peabody{drbob}509: echo $JAVA_HOME
  /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home

This is the current cut of my setjdk function in my ~/.bashrc file:

  #
  # Clever trick to leverage the /usr/bin/java commands to take advantage
  # of the JAVA_HOME environment variable and the /usr/libexec/java_home
  # executable to change the JDK on-the-fly. This is so easy I'm amazed.
  #
  function removeFromPath() {
    export PATH=$(echo $PATH | sed -E -e "s;:$1;;" -e "s;$1:?;;")
  }
 
  function setjdk() {
    if [ $# -ne 0 ]; then
      removeFromPath '/System/Library/Frameworks/JavaVM.framework/Home/bin'
      if [ -n "${JAVA_HOME+x}" ]; then
        removeFromPath $JAVA_HOME
      fi
      export JAVA_HOME=`/usr/libexec/java_home -v $@`
    fi
  }
  setjdk 1.8

So now I can work with any version of Java out there. What I did find interesting is that they have pulled JDK 9 - and that means it was really bad. Like Wow bad... at least they knew to pull it.

I was chatting with a Clojure developer friend this morning, and he mentioned getting the CryptoQuip solver running on the GraalVM and seeing how it performed. I didn't know anything about it, and I decided that if my friend was interested in it, then there's certainly something there, and I should read up on this. I'm very glad I did.

I've seen several attempts at making a Clojure compiler in the past - heck, I've thought about trying it - but each one falls a little short, and only handles a small subset of the complete Java byte codes, and so it's good for little things, but it doesn't really handle big projects that you'd run into in practice. So I started reading, and there is the GraalVM, and the Graal compiler. It's a project from Oracle, and is really about creating a universal VM that can run all kinds of languages - not just JVM-based, which is very interesting, and more than a little ambitious, but more power to them.

They have pages on compiling to native apps to avoid the startup costs of the JVM - which isn't horrible, but for Clojure, what's loaded is a lot more than just the JVM, and it really does make building command-line tools in Clojure hard. But I found another post about a real Clojure example, and it showed that it wasn't all that hard to add a post-processing step after the uberjar, and then you get a native executable that's pretty impressive.

When I did more reading, the big limitation is that code can't hit the class loader and be compiled - makes sense, but in my past one of the really nice libraries that used this to great effect was amazonica and it was really good at handling the AWS details, but I suppose if you're not using AWS, then you're not too bad off.

WHat's for sure is that I'll remember this as it makes Scala, Clojure, and all the JVM languages able to be native code, and that will make Docker images smaller, and everything spin up faster, etc. It even says it does Python, so there's a possibility there.

Very interesting.

I've just finished a book that I really enjoyed, but I'm glad I read it last of the three - in fact, I'm glad I read each of them in the order in which I did. It made for a much smoother reading experience - because I started with Pragmatic Scala and this was really based on the fact that we are using Scala in The Shop, and as such, I felt it was important to get up to speed on the language, and the Pragmatic Books have always been pretty good, in my experience.

Well, this was a fine book on Scala, but I have to say, I'm stunned that Scala is as popular as it is. I understand that one of the key design goals of the language was to be a pure Object Oriented language - and as such things like static methods and variables are not allowed in the instance variables, but they solve that with partner classes, and that's OK... but it is confusing for many folks - simply because most coders are not going to have a Theory of Languages in their past.

Still... Scala is a language we use at The Shop, and that means for better or worse, this is something that I needed to know. The next book was a little more fun because it was all about Xcode. In Xcode Treasures, the author walks through the tool not the SDK. This is nice, because in my recent work, I've re-written my crypto quip solver to Swift 4.2, in Xcode 9, and the times are really not horrible - compared to ObjC and Ruby, but Clojure still wins the speed race, but that's to be expected.

And while I spent plenty of time in Xcode quite a while ago, it's nice to see how much progress has been made in Xcode recently. The handling of assets like icons and images, and being able to load them without worrying about the resolution is really nice. Also, the entire Gatekeeper and Code Signing is now simple, and it used to be a pain. That's a great relief.

So this was a really good chance to catch up on a lot of the capabilities with Storyboards in Xcode and the scripting and such... very nice book. The final book was Mastering Clojure Macros and this is one I've been trying to get through for quite a while - many months, in fact. It's not a long book, but it's dense, and it takes time to make sure you understand the concepts because macros in Clojure are hard enough, and some of the examples are compact - as Clojure is want to do, and that just makes it all that much harder.

But the book was just amazing! What a great study of the subject. This was one of the few topics I really wanted to get better at with Clojure. Yes, I'd like to get a little more into spec, and core.async could be very useful if you're making small, independent apps, as opposed to big, multi-host apps where you typically share messaging queues, etc., but still... macros are in everything when you really get into things and I have been able to do quite a bit with them to date, but I wanted to know more.

Of the three - they are all worth reading - if you want to learn the material, but I really enjoyed the last two far more than the Scala book, but that's because of the subject - not the book.

In the work I'm doing, I've got a service that chooses to return a group of files in a standard Zip file format, and then I can easily store it in S3 using amazonica and a little bit of code:

  (defn save!
    "Given a byte array or string `data` and a bucket `b`, saves the
    bytes as an encrypted file with file name `fname`. Optional keys
    `content-type` and `overwrite` may also be passed, where the
    `:content-type` key indicates the content type (defaulted to S3's
    inference), and the `:overwrite` key indicates whether to replace\
    the file if it already exists (defauted to false)."
    [data b fname & {:keys [content-type overwrite input-stream-size]}]
    (if (and (or (string? data) (byte-array? data) (instance? InputStream data)) b)
      (let [nm (or fname (uuid))
            echo (if-not fname {:name nm})
            [inp cnt] (cond
                        (string? data)     [(ByteArrayInputStream. (.getBytes data))
                                            (count data)]
                        (byte-array? data) [(ByteArrayInputStream. data)
                                            (count data)]
                        (instance? InputStream data) [data input-stream-size]
                        :else [nil nil])]
        (if (or overwrite (false? (file-exists? b nm)))
          (try-amzn 3
            (merge
              echo
              (put-object (get-cred)
                          :bucket-name b
                          :key nm
                          :input-stream inp
                          :metadata (merge
                                      (if content-type {:content-type content-type})
                                      (if cnt {:content-length cnt})
                                      {:server-side-encryption "AES256"}))))
          (merge echo {:error "FileNotSaved"})))))

But what if it's a zip file? If we want to do this one-pass, we have to load the entire contents of the file into memory, and then piece it apart. That's certainly possible, but what if the files are very large? Why not unzip the stream, and write it back to S3 as a stream?

Then, we don't have to have a large memory footprint in order to process the large zip files. That would be nice.

  (defn unzip!
    "Function to unzip the provided file in the provided bucket into the same
    S3 bucket ehere the directory is the name of the file - without the extension,
    and all the files from the zip file are deposited into the directory under
    their names in the zip archive. The downside of this function is that it has
    to read the zip file from S3 'n' times - one for each of the files in the
    zip archive. That means that it's not at all fast. This returns a sequence
    of all the files that have been unzipped to S3:
      [\"13B7E73B053C497D82F8FCC28FC8127F/13b7e73b053c497d82f8fcc28fc8127f.XML\"
       \"13B7E73B053C497D82F8FCC28FC8127F/Form0001.PDF\"
       \"13B7E73B053C497D82F8FCC28FC8127F/Index.ctl\"]
    "
    [bkt fname]
    (if (file-exists? bkt fname)
      (if-let [base (some identity
                          (rest (re-matches #"(?i)(.*)\.zip$|(.*)\.xfr$" fname)))]
        (let [afn (atom [])
              push (fn [ze zis]
                     (if (not (.isDirectory ze))
                       (let [to-file (str base "/" (.getName ze))
                             res (save! zis bkt to-file
                                        :overwrite true
                                        :input-stream-size (.getSize ze))]
                         (if-not (:error res) (swap! afn conj to-file)))))
              s (get-stream bkt fname)]
          (with-open [z (ZipInputStream. s)]
            (doseq [e (entries z)
                    :let [en (.getName e)
                          ms (get-stream bkt fname)]]
              (with-open [mis (ZipInputStream. ms)]
                (let [me (entries mis)]
                  (push (first (drop-while #(not= en (.getName %)) me)) mis))))
            @afn)))))

The result is something that has to read the stream once for each file in the unzipped file, but then it can write each of these back to S3. It's not super network efficient, but the existing library closes the stream when it's done reading a file, and if they just hadn't done that, then I could have done it all in one pass.

Still... this is nice. It works great, and does just what I need.

This month has been a lot of work on the Auto App Package for The Shop. This is the package of legal documents that you get from your mortgage company after you give them all your information, and the details of the house you want to buy, and they work up a quote for the loan, and there are all these legal documents you need to sign before the loan goes into underwriting. It's a manual process, and they want it to be automatic.

So this is what I've been doing. It's not really amazing work, but it's pulling together a lot of the details from many different sources, an do all the annoying edge-case math to get all the numbers to line up, and then present it in a way so that when the other groups that need to use this aren't ready to finish this, we can go into the field with it and deliver value until such time as they are.

It's just another month at The Shop.

I'm a huge fan of clj-time - it's made so many of the time-related tasks so much simpler. But as is often the case, there are many applications for time that don't really fall into the highly-reusable category. So I don't blame them for not putting this into clj-time, but we needed to find the start of the year, start of the month, and start of the quarter.

The slight wrinkle is that the quarter could be on a few different starting cycles, so we had to be able to allow for that.

 (defn start-of-year
   "Gets the start of the year (midnight, 1 Jan) for the specified
   date-time."
   [dt]
   (date-time (year dt)))
 
 (defn start-of-month
   "Gets the start of the month (midnight) for the specified date-time."
   [dt]
   (date-time (year dt) (month dt)))
 
 (defn start-of-quarter
   "Gets the start of the quarter for the specified date time.
 
   This function assumes the standard quarterly cycle (Jan-Apr-Jul-Oct).
   A different cycle can be generated by providing the 1-based index
   (e.g., 1 => Jan) of the cycle start month."
   ([dt]
     (start-of-quarter dt 1))
   ([dt start-month]
     (let [offset (mod (- (month dt) start-month) 3)]
       (minus (start-of-month dt) (months offset)))))

None of this is Rocket Science, but it's nice to not have to mess with these conversions in the remainder of your code - which is kinda the point of the functional part of the language - no?

We wanted to be able to log, to the database, the name of the function that was called and there was no obvious way to do that. There were some half-baked ideas, but we really needed to have a little more robust scheme. So we made this:

  (defn fn-name
   "Function to get the name of the function passed in from the basic
   information of the function - or in the case of a hooked function, the
   metadata placed on that function by the hook."
   [f]
   (-> (or (:robert.hooke/original (meta f)) f)
       (class)
       (str)
       (cs/replace #"^class " "")
       (cs/replace #"\$" "/")
       (cs/replace #"_" "-")))

where:

  [clojure.string :as cs]

is in the namespace's require function.

This will take the normal function - or one that Robert Hooke has captured for additional magic, and return it to the caller.

Very handy thing to have for logging... very.

Bret and I finally got the CORS support for ring middleware working in our applications. It required that we get the starting point for the CORS support from another project, and then augment it to make sure it was working for all the cases we needed.

Basically, you need to start with:

  [rwilson/ring-cors "0.1.9"]

in your project.clj, and then in the server.clj for the library, we reference it:

  [ring.middleware.cors :as cors]

and then create the middleware function:

  (def wrap-cors
    "This is a simple convenience definition for enabling the CORS support in
    middleware for a service. The use of this will be far simpler than including
    all this in every service."
    #(cors/wrap-cors % :access-control-allow-origin #".+"
                       :access-control-allow-headers ["Origin" "Content-Type"
                                                      "Accept" "Authorization"
                                                      "Last-Modified" "Credentials"
                                                      "X-Request-Id" "X-Session-Id"]
                       :access-control-allow-methods [:get :put :post :delete]))

And with this, you can then simply put this into your middleware stack and it takes care of all the work:

  (def app
    "The actual ring handler that is run -- this is the routes above
     wrapped in various middlewares."
    (-> app-routes
        wrap-json-with-padding
        (wrap-ssl-redirect { :ssl-port (cfg/jetty :https-redirect) })
        handler/site
        wrap-cors
        wrap-params
        wrap-cookies
        wrap-logging
        wrap-proxy-loggly
        wrap-gzip))

We have a shared library that has a lot of really useful functions in it for all the apps we build, and one of the really useful things is to round to a fixed number of decimal places. Now, this isn't Rocket Science, but it is nice to have written well once, and then not have to mess with ever again.

The trick is that you need it to be fast for all Clojure datatypes as well. And several of the more common implementations aren't very fast due to boxing and the overhead associated with that. So we made some pretty simple little functions:

  (defn to-2dp
    "Function to simply resolve a number to a decimal (double), and then round
    it to 2 DP and return it. Pretty simple, but very useful."
    [x]
    (/ (math/round (* 100.0 (parse-double x))) 100.0))
 
  (defn to-3dp
    "Function to simply resolve a number to a decimal (double), and then round
    it to 3 DP and return it. Pretty simple, but very useful."
    [x]
    (/ (math/round (* 1000.0 (parse-double x))) 1000.0))

but we had a problem. Some of the inputs gave us a lot of grief because of some nasty rounding. And it wasn't all values - it was just those values that had a hard time being expressed in the floating point format.

The solution was to accept some of the inefficiency of rat eJVM BigDecimal and the advanced representation it had, and use that:

  (defn to-2dp
    "Function to simply resolve a number to a decimal (double), and then round
    it to 2 DP and return it. Pretty simple, but very useful."
    [x]
    (/ (math/round (* 100.0 (bigdec (parse-double x)))) 100.0))
 
  (defn to-3dp
    "Function to simply resolve a number to a decimal (double), and then round
    it to 3 DP and return it. Pretty simple, but very useful."
    [x]
    (/ (math/round (* 1000.0 (bigdec (parse-double x)))) 1000.0))

and then things settled down.

This morning I was dealing with yet another change to some clojure functions that are in a library that we have created at The Shop to make it easy to make the services that we make, and easy for new clojure developers to wrap their heads around. It's not perfect, or ideal, but it does strike a nice balance between purely functional code and re-use and state in the library.

Face it - connection pooling is state. It needs to exist, but it's not what you'd like to have if you could avoid it. So you bury it in a library, so that the new clojure developers can use this library with the connection pools clearly hidden off screen doing their thing.

It's a compromise. But one that favors the new clojure developer. And most days, that's the folks I have to work with. They typically know Java, or C#, but it's a rare individual that comes into the shop with a lot of clojure experience, and usually they "get it" about the decisions, and it's something they pretty easily adapt to.

But then there are others.

Today I've been dealing with Steve. Steve is a C# developer that hates his job. He's hated it for a long time. He hates the language he's using (C#). He hates the tools he is having to learn to do his job. He pretty much hates everything about his job.

But he wants to work in clojure.

Yet he's not really interested in learning the landscape. He wants to change it to be how he thinks is should be. I don't blame him - he questions a lot because he trusts no one. Having asked him this direct question, and receiving the affirmative response, I know beyond a shadow of a doubt, this is exactly who he is. But to not learn what is before suggesting what should be is a little short-sighted.

That he's annoying about it is just "icing on the cake" for me.

So this morning I decided that emotion has no place in this conversation. Neither does his opinions. I have listened, I have tried to explain, I have listened more, and tried to show why this is the path we have taken. All of this is immaterial to him, and so in the end, I have to make a decision, list the reasons for this decision, and then move on.

I don't relish being a jerk. Or a dictator. But at this point, more conversation is wasting time. It's time to accept that this is the paradigm we are working under, and that's it.

I do not expect him to like it.