What's it all about, Alfie?

This morning I was updating iTerm2 and realized that ~/Library/Images/People was there but it wasn't something I could ls... and I thought Great! Corrupted disk... but then I started digging into the problem and found some interesting things with macOS 10.14 Mojave.

To start, this is what I was seeing:

  peabody{drbob}378: ls
  ls: .: Operation not permitted

and I thought permissions - but it wasn't. I checked the permissions on the directory, and then the ownership, and root wasn't doing any better. This was a bad situation. I had a back-up at home, and I was able to see that it was just this one directory, so maybe it was Mail.app - nope... that wasn't it.

Then I decided to see what the Finder said - and there were all the files! I could make a new directory - People2 and use the Finder to copy all the files to the new directory - and then remove the old People and rename the new one. But the files were invisible on the rename!

OK... this was getting closer - it's the directory name. So let's try Terminal.app - and when I tried to change to ~/Library/Images/People it asked if I wanted to allow Terminal.app to access my Contacts.

Ahh...

This was a simple issue of sandboxing for security. iTerm2 wasn't triggering the request for permissions to that directory, and so the app wasn't allowed to "see" it. Very clever. Sadly, you can't manually add an application to the 'Contacts' in the Security part of System Preferences. Too bad.

For now, I know the problem, and how to work around it - and when iTerm2 gets this fixed, it'll ask, and as I did with Terminal.app, I'll say "OK", and we'll be good to go. Wild.

UPDATE: There's a pretty easy fix for this in Mojave and it's in the Security System Preferences - you simply need to drag the iTerm app into the list, and then select the checkbox. Simple. It now has access to all the directories.

I was chatting with a Clojure developer friend this morning, and he mentioned getting the CryptoQuip solver running on the GraalVM and seeing how it performed. I didn't know anything about it, and I decided that if my friend was interested in it, then there's certainly something there, and I should read up on this. I'm very glad I did.

I've seen several attempts at making a Clojure compiler in the past - heck, I've thought about trying it - but each one falls a little short, and only handles a small subset of the complete Java byte codes, and so it's good for little things, but it doesn't really handle big projects that you'd run into in practice. So I started reading, and there is the GraalVM, and the Graal compiler. It's a project from Oracle, and is really about creating a universal VM that can run all kinds of languages - not just JVM-based, which is very interesting, and more than a little ambitious, but more power to them.

They have pages on compiling to native apps to avoid the startup costs of the JVM - which isn't horrible, but for Clojure, what's loaded is a lot more than just the JVM, and it really does make building command-line tools in Clojure hard. But I found another post about a real Clojure example, and it showed that it wasn't all that hard to add a post-processing step after the uberjar, and then you get a native executable that's pretty impressive.

When I did more reading, the big limitation is that code can't hit the class loader and be compiled - makes sense, but in my past one of the really nice libraries that used this to great effect was amazonica and it was really good at handling the AWS details, but I suppose if you're not using AWS, then you're not too bad off.

WHat's for sure is that I'll remember this as it makes Scala, Clojure, and all the JVM languages able to be native code, and that will make Docker images smaller, and everything spin up faster, etc. It even says it does Python, so there's a possibility there.

Very interesting.

I read a tweet this morning about Apache Pulsar achieving top-level project status in the Apache organization. I hadn't heard of it, so I spent a little time reading about it and getting familiar with the differences between it and Storm, Flink, etc. What I found was something that I might actually use in the future.

The important parts are that Pulsar is a message broker with reliable messaging, with storage and fail-over. This makes it way better than many other systems, but it goes a lot further than that. Because it's Apache, it's using ZooKeeper, but it's also using BookKeeper for the message storage - which is an interesting twist.

It's scalable up to a million topics... is meant to sync servers by geo-location, and all the goodies you'd expect from something like Cassandra or the like. Lost of resilience. Turns out, it was the message broker for Yahoo, and has been open sourced. It's low-latency, and meant to be server-to-server in the datacenter, even if they are separated by continents. So it's battle-tested. And all this makes it a useful project because there are lots of times I can imagine needing a better broker than SQS.

But what makes it even nicer is that it has all the Lambda capabilities that AWS has as well. So now you can add in the streaming capabilities of Storm in the messaging system - as opposed to having a streaming system talk to Kafka as a source and sink. This is a different take on the problem, but one of the problems I had with Storm was the scaling of the nodes - it was hard to get them to not back-up. If this is a message broker, then we have a lot more flexibility on the processing time, and every hand-off is to a queue/topic. That's good news.

I'm not sure if/when I'll need to have something like this, but I'm very happy to see that they have it, and it's gotten to a point that it looks pretty stable. Good news!

This morning I decided to take another look at the Xcode 10 performance tests on my CryptoQuip puzzle solver, but this time, looking at the Objective-C version and see if the Xcode 10 changes would make it any faster. Turns out, I'm very glad that I did.

I was getting a very odd error on the line:

int main(int argc, const char *argv[])
{
    return NSApplicationMain(argc, argv);
}

the value of argv was causing a SIGABRT - and I just had a feeling it was some setting in the old project file and I needed to clean those up - so after a little bit of updating, it compiled and ran just fine.

I then put in the timing of the call with a few lines:

- (BOOL) attemptWordBlockAttack
{
    // sort the puzzle pieces by the number of possible words they match
    [[self getPuzzlePieces] sortUsingSelector:@selector(comparePossibles:)];
    // ...now run through the standard block attack
    NSTimeInterval begin = [NSDate timeIntervalSinceReferenceDate];
    BOOL ans = [self doWordBlockAttackOnIndex:0 withLegend:[self getStartingLegend]];
    NSLog(@"%lu Solution(s) took %f msec",
        (unsigned long)[[self getSolutions] count],
        ([NSDate timeIntervalSinceReferenceDate] - begin) * 1000);
    return ans;
}

so that I'd get a similar result as the Swift version - basically, taking everything out of the time in the solution except the attack. That's as close as an apples-to-apples as I could get with the other versions.

What I found out blew me away:

CryptoQuip[87317:16056980] Loaded 25384 words from /Users/drbob/.../words
CryptoQuip[87317:16056980] Ready
CryptoQuip[87317:16056980] Solving puzzle: 'Fict O ncc bivteclnbklzn O lcpji ukl pt
                               vzglcddp' where b=t
CryptoQuip[87317:16056980] 1 Solution(s) took 0.464082 msec
CryptoQuip[87317:16056980] Solution found: 'When I see thunderstorms I reach for an
                               umbrella'

Now the times for Clojure were in the 6 msec... and Swift was about 7 msec... but I had to do a double-take on this... I had no idea Objective-C was going to be this fast! I had to compute, by hand, the number of milliseconds since the reference date - January 1, 2001 - just to make sure I was doing the math right. I was right... it was right... I checked the code several times, made sure I wasn't skipping the actual solution... and I wasn't.

Wow... Apple has really outdone themselves on this... sub-msec on the solution. Wow! I have to say this is excellent news in my book. I can live with Swift, and I know it's the future, and it's fast... but it's not this fast, and that's just amazingly cool!

Withe the recent improvements in performance from Xcode 10 for my Swift code, I thought I wanted to dust off DKit and make sure it still works on macOS 10.13.6 with Xcode 10 - it's been quite a few years since I've made any changes to it, and I just wanted to make sure it all builds and runs.

One of the things I remembered from a few months ago, when I tried to dust it off for a homework project for a job interview and I had some problems with the fact that the shared library was being built as a dylib and that just makes it a lot harder. At the time I originally wrote the library - some 15 yrs ago - there wasn't the support in Mac OS X for the standard unix SO shared library - so I had to use the dylib and figure out how to link it in properly.

Today, the story is entirely different - and honestly, for several years, I've known that macOS could generate the unix SO shared library. So I just switched over to that this morning, and gave it a spin. Turns out it worked perfectly.

I was able to compile all the components of the library, and then the test apps, where the templates are really being used, and all the tests ran. I checked in the simple changes to the Makefiles and everything is solid.

I'm sure there are a lot of things that this could add - I could make a test app that read a file and sent it out on a TCP or UDP transmitter - for the testing of the TCP and UDP receiver, but I'm not worried about that right now. It's enough that it's still working just fine, and builds on the latest macOS. Can't beat that!

Linus Torvalds is in the news again this morning, and I'm glad to see he's now aware of the effect he had on people, it got me thinking about git and I thought I remembered that he wrote the first version of git after something happened with the license on the SCM tool he was using on the Linux Kernel. So I hit Wikipedia, and sure enough, there was a license issue with BitKeeper, and so Linus wrote the first version of git.

What I hadn't remembered was the timeline - and that's what amazed me this morning:

• 3 April 2005 - Development Starts on git
• 6 April 2005 - the git project is announced
• 7 April 2005 - git is self-hosting
• 18 April 2005 - first merge of multiple branches
• 29 April 2005 - performance target is achieved: 6.7 patches per second
• 16 June 2005 - linux kernel 2.6.12 release is managed by git

So a couple of months to get a SCM that hosts one of the most influential codebases that currently exists... that's more than impressive... that's down right amazing.

Think about all that git has effected... GitHub, BitBucket, GitLabs - there's an industry built up around it - that's wild to think about. Sure, the guy may not be someone you want to have over for dinner, but what he accomplished in a few months was nothing short of amazing. And all of us in the tech community have benefited from it.

There is something to be said for talent... it's not necessarily sitting in the prettiest of packages, but we would all do well to look past the packaging, and try to see the real value under the covers. It can be revolutionary.

In the work I'm doing, I've got a service that chooses to return a group of files in a standard Zip file format, and then I can easily store it in S3 using amazonica and a little bit of code:

  (defn save!
    "Given a byte array or string `data` and a bucket `b`, saves the
    bytes as an encrypted file with file name `fname`. Optional keys
    `content-type` and `overwrite` may also be passed, where the
    `:content-type` key indicates the content type (defaulted to S3's
    inference), and the `:overwrite` key indicates whether to replace\
    the file if it already exists (defauted to false)."
    [data b fname & {:keys [content-type overwrite input-stream-size]}]
    (if (and (or (string? data) (byte-array? data) (instance? InputStream data)) b)
      (let [nm (or fname (uuid))
            echo (if-not fname {:name nm})
            [inp cnt] (cond
                        (string? data)     [(ByteArrayInputStream. (.getBytes data))
                                            (count data)]
                        (byte-array? data) [(ByteArrayInputStream. data)
                                            (count data)]
                        (instance? InputStream data) [data input-stream-size]
                        :else [nil nil])]
        (if (or overwrite (false? (file-exists? b nm)))
          (try-amzn 3
            (merge
              echo
              (put-object (get-cred)
                          :bucket-name b
                          :key nm
                          :input-stream inp
                          :metadata (merge
                                      (if content-type {:content-type content-type})
                                      (if cnt {:content-length cnt})
                                      {:server-side-encryption "AES256"}))))
          (merge echo {:error "FileNotSaved"})))))

But what if it's a zip file? If we want to do this one-pass, we have to load the entire contents of the file into memory, and then piece it apart. That's certainly possible, but what if the files are very large? Why not unzip the stream, and write it back to S3 as a stream?

Then, we don't have to have a large memory footprint in order to process the large zip files. That would be nice.

  (defn unzip!
    "Function to unzip the provided file in the provided bucket into the same
    S3 bucket ehere the directory is the name of the file - without the extension,
    and all the files from the zip file are deposited into the directory under
    their names in the zip archive. The downside of this function is that it has
    to read the zip file from S3 'n' times - one for each of the files in the
    zip archive. That means that it's not at all fast. This returns a sequence
    of all the files that have been unzipped to S3:
      [\"13B7E73B053C497D82F8FCC28FC8127F/13b7e73b053c497d82f8fcc28fc8127f.XML\"
       \"13B7E73B053C497D82F8FCC28FC8127F/Form0001.PDF\"
       \"13B7E73B053C497D82F8FCC28FC8127F/Index.ctl\"]
    "
    [bkt fname]
    (if (file-exists? bkt fname)
      (if-let [base (some identity
                          (rest (re-matches #"(?i)(.*)\.zip$|(.*)\.xfr$" fname)))]
        (let [afn (atom [])
              push (fn [ze zis]
                     (if (not (.isDirectory ze))
                       (let [to-file (str base "/" (.getName ze))
                             res (save! zis bkt to-file
                                        :overwrite true
                                        :input-stream-size (.getSize ze))]
                         (if-not (:error res) (swap! afn conj to-file)))))
              s (get-stream bkt fname)]
          (with-open [z (ZipInputStream. s)]
            (doseq [e (entries z)
                    :let [en (.getName e)
                          ms (get-stream bkt fname)]]
              (with-open [mis (ZipInputStream. ms)]
                (let [me (entries mis)]
                  (push (first (drop-while #(not= en (.getName %)) me)) mis))))
            @afn)))))

The result is something that has to read the stream once for each file in the unzipped file, but then it can write each of these back to S3. It's not super network efficient, but the existing library closes the stream when it's done reading a file, and if they just hadn't done that, then I could have done it all in one pass.

Still... this is nice. It works great, and does just what I need.

I'm a huge fan of clj-time - it's made so many of the time-related tasks so much simpler. But as is often the case, there are many applications for time that don't really fall into the highly-reusable category. So I don't blame them for not putting this into clj-time, but we needed to find the start of the year, start of the month, and start of the quarter.

The slight wrinkle is that the quarter could be on a few different starting cycles, so we had to be able to allow for that.

 (defn start-of-year
   "Gets the start of the year (midnight, 1 Jan) for the specified
   date-time."
   [dt]
   (date-time (year dt)))
 
 (defn start-of-month
   "Gets the start of the month (midnight) for the specified date-time."
   [dt]
   (date-time (year dt) (month dt)))
 
 (defn start-of-quarter
   "Gets the start of the quarter for the specified date time.
 
   This function assumes the standard quarterly cycle (Jan-Apr-Jul-Oct).
   A different cycle can be generated by providing the 1-based index
   (e.g., 1 => Jan) of the cycle start month."
   ([dt]
     (start-of-quarter dt 1))
   ([dt start-month]
     (let [offset (mod (- (month dt) start-month) 3)]
       (minus (start-of-month dt) (months offset)))))

None of this is Rocket Science, but it's nice to not have to mess with these conversions in the remainder of your code - which is kinda the point of the functional part of the language - no?

We wanted to be able to log, to the database, the name of the function that was called and there was no obvious way to do that. There were some half-baked ideas, but we really needed to have a little more robust scheme. So we made this:

  (defn fn-name
   "Function to get the name of the function passed in from the basic
   information of the function - or in the case of a hooked function, the
   metadata placed on that function by the hook."
   [f]
   (-> (or (:robert.hooke/original (meta f)) f)
       (class)
       (str)
       (cs/replace #"^class " "")
       (cs/replace #"\$" "/")
       (cs/replace #"_" "-")))

where:

  [clojure.string :as cs]

is in the namespace's require function.

This will take the normal function - or one that Robert Hooke has captured for additional magic, and return it to the caller.

Very handy thing to have for logging... very.

Bret and I finally got the CORS support for ring middleware working in our applications. It required that we get the starting point for the CORS support from another project, and then augment it to make sure it was working for all the cases we needed.

Basically, you need to start with:

  [rwilson/ring-cors "0.1.9"]

in your project.clj, and then in the server.clj for the library, we reference it:

  [ring.middleware.cors :as cors]

and then create the middleware function:

  (def wrap-cors
    "This is a simple convenience definition for enabling the CORS support in
    middleware for a service. The use of this will be far simpler than including
    all this in every service."
    #(cors/wrap-cors % :access-control-allow-origin #".+"
                       :access-control-allow-headers ["Origin" "Content-Type"
                                                      "Accept" "Authorization"
                                                      "Last-Modified" "Credentials"
                                                      "X-Request-Id" "X-Session-Id"]
                       :access-control-allow-methods [:get :put :post :delete]))

And with this, you can then simply put this into your middleware stack and it takes care of all the work:

  (def app
    "The actual ring handler that is run -- this is the routes above
     wrapped in various middlewares."
    (-> app-routes
        wrap-json-with-padding
        (wrap-ssl-redirect { :ssl-port (cfg/jetty :https-redirect) })
        handler/site
        wrap-cors
        wrap-params
        wrap-cookies
        wrap-logging
        wrap-proxy-loggly
        wrap-gzip))