What's it all about, Alfie?

It's Day 11 of Advent of Code, and I have to say that I'm having a great time this year. It helps to have a little spare time as they are working on the rearrangement of the groups with the acquisition, so I've got a little spare time most days.

I honestly expected to be using Swift, Clojure, ObjC - all mixed in there. This was the advantage of getting CodeRunner a little bit ago - that I could mix-n-match language to the problem based on the needs of each problem. But I have to say that I'm enjoying using Clojure 100% so far this year. It's just so good at what I need it to do... it's hard to find a need for another language.

I've also really enjoyed the times when the solution for the first part of the day is fine with my original implementation, but then the second part requires that I really re-think the solution and come up with a much more performant solution. Not always easy, but when I get something that really changes the scope of what can be done, it's a lot of fun.

One of the guys at The Shop pointed out that I'm doing the problems like Programming Golf - where the minimal solution is the best. And that's exactly what I enjoy about these - it's about a minimalist approach to the problems. What fun. 🙂

This morning, I wanted to get the first version of Cap'n Hook published on Clojars, but I needed to get a few things cleaned up. First, I needed to get the latest changes to durable-queue up to Clojars, as there were some dependencies I need to clear up - most notably in the nippy and byte-streams packages, so that was fixed up and a simple publish, and I'm up on Clojars.

At this point, I'm a little surprised that I have to type in my GPG passphrase each time I sign the jar for publishing. I can see the logic that I've read online - basically, signing is a fairly infrequent task, and needs to be done with care, so ask for the passphrase on those occasions. OK... I get it, but I'd still like to have it in my Keychain, and then just lock it up there. No need to keep typing it in.

I need to pay closer attention to the passphrase dialog on the next publish, and see if I can't get it to save the passphrase in the Keychain. But it's not horrible at this time - I don't publish all that often.

After I got durable-queue up, I could then clean up the capn-hook code a little and then publish it. Again, not bad - cleared out a few dependencies to streamline things just a touch, and then update the README.md for the folks coming to the GitHub page, and away it goes.

Not a bad morning - got a little more familiar with GPG, and published a few things. Good enough.

This morning I've finally put the docs and a few tests on my Clojure library called Cap'n Hook - after the cereal, and the fact that it's all about making web hooks easier to implement in services. When I was looking for my current position, I was thinking what could I write that would be fun and useful at the same time. One of the things that I knew were really useful were the web hooks that allowed one-way, asynchronous, message passing from a service to a set of registered clients.

While it's not really a good replacement for a real messaging system like any of the messaging offerings from AWS, or Tibco, or even Java MQ - it is very easy to implement, and if you're in the world of RESTful services, it's a really handy thing to have, as it's super simple to implement.

Also, if you put in retries, as I have, then it's pretty close to reliable - but only if your receiver is online and taking POST messages on that URL. Sure... it's not perfect, but for dealing with simple messaging between data centers, it's hard to beat. There are a ton of services (GitHub, etc.) that are using this to fire-off events on the client-side of things.

So it's pretty simple to use - the key is that for most cases, you are going to need to have some persistent set of registered URLs, These are the "endpoints" for the HTTP POST calls that this library will be sending in response to the main application saying "Here you go!".

I was a little blocked for several months on this - because I wanted to be able to implement a shared, persistent, reliable storage for the registrations - and then yesterday I realized that was just a bad idea. Each application that uses this will want to implement their own registration their way - and while I implemented a simplified registration that wasn't shared or persistent, the ideas are there, and it's easy to slot in a different registration scheme at any time.

I can see using redis to store the URLs - it has all the primitives to do what's needed. And there's Postgres - or any database, really, and I even cover that in one of the examples in the README.md on the GitHub repo. So there's no need to really solve the registration problem for everyone. Just give the users a simple one to get going, and then they can get as fancy as they need to.

I spent a little time on writing some unit tests, but it's going to be hard to really test the sending as it really needs to hit something, but I'll spend a little time today seeing if there isn't an "echo" test site out there for just this kind of testing, and then I can test sending the POST messages to see if they work.

But it's been a lot of fun to get this done and into testing... The documentation was great fun, and it just was a great little thing to finish.

This morning, I really wanted to see if I could get Clojure running in CodeRunner. It's a pretty nice lightweight development environment which is really targeted at scripting or prototyping ideas. The docs say it can do a full IDE for web development, and that's certainly possible - but I'm not sure that I'd use it for that, as there are likely far more dependencies - like the RESTful service, that need to be included, and that's just too much.

While CodeRunner has a lot of language support built-in, it doesn't handle Clojure, and that's the one language that I really wish they would have. So much so that I emailed the author to ask about adding it. When I didn't get a response, I decided to see if I couldn't do it on my own.

Turns out, it's not that hard.

Start by having Leiningen installed, and that you can get from Homebrew - which is a good idea to have installed on a Mac anyway. Assuming you have Homebrew installed, you simply need to say:

  $ brew install leiningen

and it'll be downloaded and ready to go.

Then you can look at the lein-exec GitHub page for the rest of the instructions as to how to get it going. You need to update your ~/.lein/profiles.clj file to include:

  {:user {:plugins [[lein-exec "0.3.7"]]}}

and then in your ~/bin directory, or really any place in your PATH you need to create the following two files - copied from the GitHub repo. First is lein-exec containing:

  #!/bin/bash
 
  if [[ $1 =~ ^[~\/] ]]
  then
    # its already an absolute path
    lein exec "$@"
  else
    # This is a relative path, so make it absolute,
    # using the current directory as the base.
    lein exec "`pwd`/$@"
  fi

and the second is lein-exec-p, which contains:

  #!/bin/bash
 
  if [[ $1 =~ ^[~\/] ]]
  then
    # its already an absolute path
    lein exec -p "$@"
  else
    # This is a relative path, so make it absolute,
    # using the current directory as the base.
    lein exec -p "`pwd`/$@"
  fi

At this point, you can make a script that looks like:

  #!/bin/bash lein-exec

  (prn (* 4 6))

and when you run it, you will get 24 on the console. Not bad.

In order to get this running in CodeRunner, we need to add a new language to the preferences. I chose to duplicate Perl as that's very close to what I needed. I changed the settings to look like:

and the Templates I deleted, and the Docs I added the Dash docs, as I have that installed on my mac, and it is very good:

Once I had that, I could create a script and run it:

While not perfect, it's exactly what I wanted to be able to do, and it does it perfectly.

This morning I noticed that Clojure 1.10.0-RC1 was out, and that it was setting a minimum of JDK 1.8 as the required Java version. That's not a problem, as that's what I've been using, but I thought it might be a good thing to get the lates versions of the JDK installed on my box, and make sure that JDK 10 and JDK 11 work with the setjdk Bash function I wrote to make it easy to change from version to version in the environment.

So I went to the java.com web site, and first noticed that the current version they are suggesting is JDK 1.8 - and that really surprised me. I have known that JDK 9, 10, and 11 have been out for a bit, but to see that Oracle is suggesting that a new user install JDK 1.8.0_181 - that's just a little surprising.

Also, they made it a lot harder to find the other versions of the JDK - maybe that's because there was more confusion than necessary for folks just looking for the latest JDK - but therein is kinda my concerned - JDK 1.8. Still... I found it and was able to get the latest JDK 1.8.0_181, JDK 10.0.2, and JDK 11 - they have clearly decided to change the versioning, which is OK with me - but it means that I really need to make sure that I check the setjdk function when I install these guys.

When I got them installed, they all looked in place:

  peabody{drbob}516: ls -lsa /Library/Java/JavaVirtualMachines/
  total 0
  0 drwxr-xr-x  13 root  wheel  416 Oct 11 10:29 ./
  0 drwxr-xr-x   5 root  wheel  160 Sep 24 19:00 ../
  0 drwxr-xr-x   3 root  wheel   96 Jun 29  2011 1.6.0_26-b03-383.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Oct 11 10:29 jdk-10.0.2.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Oct 11 10:29 jdk-11.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Feb  5  2013 jdk1.7.0_13.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Oct 16  2013 jdk1.7.0_45.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Jan 17  2014 jdk1.7.0_51.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Mar 20  2015 jdk1.7.0_75.jdk/
  0 drwxr-xr-x   3 root  wheel   96 May  1  2017 jdk1.8.0_131.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Oct  2  2017 jdk1.8.0_144.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Oct 11 10:28 jdk1.8.0_181.jdk/
  0 drwxr-xr-x   3 root  wheel   96 Mar 20  2015 jdk1.8.0_40.jdk/

and then a quick check of the setjdk script:

  peabody{drbob}504: setjdk 10
  peabody{drbob}505: echo $JAVA_HOME
  /Library/Java/JavaVirtualMachines/jdk-10.0.2.jdk/Contents/Home
  peabody{drbob}506: setjdk 11
  peabody{drbob}507: echo $JAVA_HOME
  /Library/Java/JavaVirtualMachines/jdk-11.jdk/Contents/Home
  peabody{drbob}508: setjdk 1.8
  peabody{drbob}509: echo $JAVA_HOME
  /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home

This is the current cut of my setjdk function in my ~/.bashrc file:

  #
  # Clever trick to leverage the /usr/bin/java commands to take advantage
  # of the JAVA_HOME environment variable and the /usr/libexec/java_home
  # executable to change the JDK on-the-fly. This is so easy I'm amazed.
  #
  function removeFromPath() {
    export PATH=$(echo $PATH | sed -E -e "s;:$1;;" -e "s;$1:?;;")
  }
 
  function setjdk() {
    if [ $# -ne 0 ]; then
      removeFromPath '/System/Library/Frameworks/JavaVM.framework/Home/bin'
      if [ -n "${JAVA_HOME+x}" ]; then
        removeFromPath $JAVA_HOME
      fi
      export JAVA_HOME=`/usr/libexec/java_home -v $@`
    fi
  }
  setjdk 1.8

So now I can work with any version of Java out there. What I did find interesting is that they have pulled JDK 9 - and that means it was really bad. Like Wow bad... at least they knew to pull it.

I was chatting with a Clojure developer friend this morning, and he mentioned getting the CryptoQuip solver running on the GraalVM and seeing how it performed. I didn't know anything about it, and I decided that if my friend was interested in it, then there's certainly something there, and I should read up on this. I'm very glad I did.

I've seen several attempts at making a Clojure compiler in the past - heck, I've thought about trying it - but each one falls a little short, and only handles a small subset of the complete Java byte codes, and so it's good for little things, but it doesn't really handle big projects that you'd run into in practice. So I started reading, and there is the GraalVM, and the Graal compiler. It's a project from Oracle, and is really about creating a universal VM that can run all kinds of languages - not just JVM-based, which is very interesting, and more than a little ambitious, but more power to them.

They have pages on compiling to native apps to avoid the startup costs of the JVM - which isn't horrible, but for Clojure, what's loaded is a lot more than just the JVM, and it really does make building command-line tools in Clojure hard. But I found another post about a real Clojure example, and it showed that it wasn't all that hard to add a post-processing step after the uberjar, and then you get a native executable that's pretty impressive.

When I did more reading, the big limitation is that code can't hit the class loader and be compiled - makes sense, but in my past one of the really nice libraries that used this to great effect was amazonica and it was really good at handling the AWS details, but I suppose if you're not using AWS, then you're not too bad off.

WHat's for sure is that I'll remember this as it makes Scala, Clojure, and all the JVM languages able to be native code, and that will make Docker images smaller, and everything spin up faster, etc. It even says it does Python, so there's a possibility there.

Very interesting.

I've just finished a book that I really enjoyed, but I'm glad I read it last of the three - in fact, I'm glad I read each of them in the order in which I did. It made for a much smoother reading experience - because I started with Pragmatic Scala and this was really based on the fact that we are using Scala in The Shop, and as such, I felt it was important to get up to speed on the language, and the Pragmatic Books have always been pretty good, in my experience.

Well, this was a fine book on Scala, but I have to say, I'm stunned that Scala is as popular as it is. I understand that one of the key design goals of the language was to be a pure Object Oriented language - and as such things like static methods and variables are not allowed in the instance variables, but they solve that with partner classes, and that's OK... but it is confusing for many folks - simply because most coders are not going to have a Theory of Languages in their past.

Still... Scala is a language we use at The Shop, and that means for better or worse, this is something that I needed to know. The next book was a little more fun because it was all about Xcode. In Xcode Treasures, the author walks through the tool not the SDK. This is nice, because in my recent work, I've re-written my crypto quip solver to Swift 4.2, in Xcode 9, and the times are really not horrible - compared to ObjC and Ruby, but Clojure still wins the speed race, but that's to be expected.

And while I spent plenty of time in Xcode quite a while ago, it's nice to see how much progress has been made in Xcode recently. The handling of assets like icons and images, and being able to load them without worrying about the resolution is really nice. Also, the entire Gatekeeper and Code Signing is now simple, and it used to be a pain. That's a great relief.

So this was a really good chance to catch up on a lot of the capabilities with Storyboards in Xcode and the scripting and such... very nice book. The final book was Mastering Clojure Macros and this is one I've been trying to get through for quite a while - many months, in fact. It's not a long book, but it's dense, and it takes time to make sure you understand the concepts because macros in Clojure are hard enough, and some of the examples are compact - as Clojure is want to do, and that just makes it all that much harder.

But the book was just amazing! What a great study of the subject. This was one of the few topics I really wanted to get better at with Clojure. Yes, I'd like to get a little more into spec, and core.async could be very useful if you're making small, independent apps, as opposed to big, multi-host apps where you typically share messaging queues, etc., but still... macros are in everything when you really get into things and I have been able to do quite a bit with them to date, but I wanted to know more.

Of the three - they are all worth reading - if you want to learn the material, but I really enjoyed the last two far more than the Scala book, but that's because of the subject - not the book.

In the work I'm doing, I've got a service that chooses to return a group of files in a standard Zip file format, and then I can easily store it in S3 using amazonica and a little bit of code:

  (defn save!
    "Given a byte array or string `data` and a bucket `b`, saves the
    bytes as an encrypted file with file name `fname`. Optional keys
    `content-type` and `overwrite` may also be passed, where the
    `:content-type` key indicates the content type (defaulted to S3's
    inference), and the `:overwrite` key indicates whether to replace\
    the file if it already exists (defauted to false)."
    [data b fname & {:keys [content-type overwrite input-stream-size]}]
    (if (and (or (string? data) (byte-array? data) (instance? InputStream data)) b)
      (let [nm (or fname (uuid))
            echo (if-not fname {:name nm})
            [inp cnt] (cond
                        (string? data)     [(ByteArrayInputStream. (.getBytes data))
                                            (count data)]
                        (byte-array? data) [(ByteArrayInputStream. data)
                                            (count data)]
                        (instance? InputStream data) [data input-stream-size]
                        :else [nil nil])]
        (if (or overwrite (false? (file-exists? b nm)))
          (try-amzn 3
            (merge
              echo
              (put-object (get-cred)
                          :bucket-name b
                          :key nm
                          :input-stream inp
                          :metadata (merge
                                      (if content-type {:content-type content-type})
                                      (if cnt {:content-length cnt})
                                      {:server-side-encryption "AES256"}))))
          (merge echo {:error "FileNotSaved"})))))

But what if it's a zip file? If we want to do this one-pass, we have to load the entire contents of the file into memory, and then piece it apart. That's certainly possible, but what if the files are very large? Why not unzip the stream, and write it back to S3 as a stream?

Then, we don't have to have a large memory footprint in order to process the large zip files. That would be nice.

  (defn unzip!
    "Function to unzip the provided file in the provided bucket into the same
    S3 bucket ehere the directory is the name of the file - without the extension,
    and all the files from the zip file are deposited into the directory under
    their names in the zip archive. The downside of this function is that it has
    to read the zip file from S3 'n' times - one for each of the files in the
    zip archive. That means that it's not at all fast. This returns a sequence
    of all the files that have been unzipped to S3:
      [\"13B7E73B053C497D82F8FCC28FC8127F/13b7e73b053c497d82f8fcc28fc8127f.XML\"
       \"13B7E73B053C497D82F8FCC28FC8127F/Form0001.PDF\"
       \"13B7E73B053C497D82F8FCC28FC8127F/Index.ctl\"]
    "
    [bkt fname]
    (if (file-exists? bkt fname)
      (if-let [base (some identity
                          (rest (re-matches #"(?i)(.*)\.zip$|(.*)\.xfr$" fname)))]
        (let [afn (atom [])
              push (fn [ze zis]
                     (if (not (.isDirectory ze))
                       (let [to-file (str base "/" (.getName ze))
                             res (save! zis bkt to-file
                                        :overwrite true
                                        :input-stream-size (.getSize ze))]
                         (if-not (:error res) (swap! afn conj to-file)))))
              s (get-stream bkt fname)]
          (with-open [z (ZipInputStream. s)]
            (doseq [e (entries z)
                    :let [en (.getName e)
                          ms (get-stream bkt fname)]]
              (with-open [mis (ZipInputStream. ms)]
                (let [me (entries mis)]
                  (push (first (drop-while #(not= en (.getName %)) me)) mis))))
            @afn)))))

The result is something that has to read the stream once for each file in the unzipped file, but then it can write each of these back to S3. It's not super network efficient, but the existing library closes the stream when it's done reading a file, and if they just hadn't done that, then I could have done it all in one pass.

Still... this is nice. It works great, and does just what I need.

This month has been a lot of work on the Auto App Package for The Shop. This is the package of legal documents that you get from your mortgage company after you give them all your information, and the details of the house you want to buy, and they work up a quote for the loan, and there are all these legal documents you need to sign before the loan goes into underwriting. It's a manual process, and they want it to be automatic.

So this is what I've been doing. It's not really amazing work, but it's pulling together a lot of the details from many different sources, an do all the annoying edge-case math to get all the numbers to line up, and then present it in a way so that when the other groups that need to use this aren't ready to finish this, we can go into the field with it and deliver value until such time as they are.

It's just another month at The Shop.

I'm a huge fan of clj-time - it's made so many of the time-related tasks so much simpler. But as is often the case, there are many applications for time that don't really fall into the highly-reusable category. So I don't blame them for not putting this into clj-time, but we needed to find the start of the year, start of the month, and start of the quarter.

The slight wrinkle is that the quarter could be on a few different starting cycles, so we had to be able to allow for that.

 (defn start-of-year
   "Gets the start of the year (midnight, 1 Jan) for the specified
   date-time."
   [dt]
   (date-time (year dt)))
 
 (defn start-of-month
   "Gets the start of the month (midnight) for the specified date-time."
   [dt]
   (date-time (year dt) (month dt)))
 
 (defn start-of-quarter
   "Gets the start of the quarter for the specified date time.
 
   This function assumes the standard quarterly cycle (Jan-Apr-Jul-Oct).
   A different cycle can be generated by providing the 1-based index
   (e.g., 1 => Jan) of the cycle start month."
   ([dt]
     (start-of-quarter dt 1))
   ([dt start-month]
     (let [offset (mod (- (month dt) start-month) 3)]
       (minus (start-of-month dt) (months offset)))))

None of this is Rocket Science, but it's nice to not have to mess with these conversions in the remainder of your code - which is kinda the point of the functional part of the language - no?