What's it all about, Alfie?

Today has been a day for colorization, and while I was at it, I decided to see what I could find that did this for Leiningen's REPL. What I found was simply too cool to believe.

There'a a GitHub project called Ultra - and it's a plugin for Leiningen that colorized the return values from the REPL so that they are a lot easier to read. Keywords, strings, numbers, maps... all are a lot easier to read with this plugin. And installing it was easy.

First, I needed to update to the latest Leiningen - and since I'm using the one in Homebrew, that's just:

  $ brew upgrade leiningen

and let it finish. This brought me to version 2.5.1.

Now just add to my ~/.lein/profiles.clj the user section:

  {
   :user {:plugins [[venantius/ultra "0.3.3"]]
          :ultra {:color-scheme :solarized_dark}}
  }

and then restart the REPL:

  $ lein repl

All the dependencies will be downloaded, and then you'll see something really cool like:

And this is going to make it all so much easier to work in the REPL. I remember going from a black-and-white editor to a colorized editor and it was as if the world of coding had changed. No longer did I have to stare intently at the code to discern it's meaning, the color told me what was important, and what wasn't. It was just amazing to my productivity.

I'm hoping I get even a fraction of that boost using these new colorization tools for log4j files and the REPL. How exciting!

UPDATE: I played around with the colors to make it look a little more like the clojure syntax file for Vim:

where the new ~/.lein/profiles.clj file looks like:

  {
   :user {:plugins [[venantius/ultra "0.3.3"]]
          :ultra {:color-scheme {:delimiter [:bold :yellow]
                                 :tag [:bold :yellow]
                                 :nil [:cyan]
                                 :boolean [:bold :cyan]
                                 :number [:bold :blue]
                                 :string [:bold :red]
                                 :character [:cyan]
                                 :keyword [:yellow]
                                 :symbol nil
                                 :function-symbol [:bold :blue]
                                 :class-delimiter [:blue]
                                 :class-name nil
                                 :exception nil}}}
  }

I'm not completely happy with these colors, but they are about as good as I'm likely to get with the ANSI color selections at my disposal. And hey... let's not forget... it could be a lot worse! 🙂

[11/20/2018] UPDATE: I've gone back and updated to the latest version of the plugin, and updated the colors for something that's much more interesting to me. The plugin is now specified as:

  {
   :user {:plugins [[venantius/ultra "0.5.2"]]
          :ultra {:color-scheme {:delimiter [:bold :yellow]
                                 :tag [:bold :yellow]
                                 :nil [:cyan]
                                 :boolean [:bold :cyan]
                                 :number [:bold :green]
                                 :string [:bold :red]
                                 :character [:cyan]
                                 :keyword [:yellow]
                                 :symbol [:bold :magenta]
                                 :function-symbol [:bold :blue]
                                 :class-delimiter [:blue]
                                 :class-name [:green]
                                 :exception [:bold :red]}}}
  }

which includes the new colors for the symbol, number, class name, and exception. I was never a fan of the bold blue for numbers, because on a dark background, it was way too hard to read. Now it's just great. Much improved.

I was chatting with a friend today, and they are having a rough time, and feeling very much unlike themselves. Up and down... slight depression... things that are so very unlike this person it is amazing that they recognize them when they are happening to them. They were really struggling, and all I could think of was "I've been here! I know this!"

Which reminded me of this scene in The West Wing between Josh and Leo, who said:

This guy's walking down a street, when he falls in a hole. The walls are so steep, he can't get out. A doctor passes by, and the guy shouts up "Hey you! Can you help me out?" The doctor writes him a prescription, throws it down the hole and moves on. Then a priest comes along and the guy shouts up "Father, I'm down in this hole, can you help me out?" The priest writes out a prayer, throws it down in the hole and moves on. Then a friend walks by. "Hey Joe, it's me, can you help me out?" And the friend jumps in the hole! Our guy says "Are you stupid? Now we're both down here!" and the friend says, "Yeah, but I've been down here before, and I know the way out." As long as I got a job, you got a job, you understand me?

I told my friend, "Hey, I've been here for years, and I know my way around this hole. Trust me - I've got your back. It'll be OK."

And for the first time since I was kicked out, I felt that all that I had been through was worth it. Every horrible day - worth it. Because I could help my friend.

Amen

I have been doing a bit of updating in the last 24 hours, which includes getting the latest Apple compilers and support tools - git 2.3.2, even (nice!)... but I've also been checking with Homebrew for the grc command, and updating it based on the new compilers, etc. from Apple. And it just hit me (again), that a tool like Homebrew makes installation of hundreds of Open Source tools so incredibly easy. It's just almost impossible to think that it's necessary to compile many programs from source on a Mac these days.

Lots of great software is in Homebrew - with sections for Science and all kinds of things. Just an amazing community resource. Amazingly all built on Ruby and git. What a way to build something great.

I've been looking for something to do a decent job of colorizing the output of tail -f for the log files I watch. Nothing really amazing - just something that will highlight the different levels of the log4j file, and maybe something to pick out IP addresses, etc. Nothing fancy. But what it needs to be is simple.

I don't want to have a tool that takes over the tail command - I still want to be able to scroll up on the terminal and see what was there. I don't want something that's got all the search features, or sorting, or filtering... that's all nice, but I have all the other unix tools for those, and that's how I think. No, what I need is a simple, streaming, colorizer of the log4j stream, and then that's it.

After doing a considerable (for me, for this) digging, it looks like a little app called grc is just the thing I need. The source site is here, but there's a fork of the code on GitHub, and given that the savba.sk domain isn't responding this morning, it's nice to have the GitHub version available.

Home-brew has it as well, and so installing it is pretty simple:

  $ brew install grc

and then the info on grace in Homebrew says that it's simple to start colorizing your Bash scripts by simply adding a config file to your ~/.bashrc. But for me, the configuration is a little more complex.

After reading the README at the GitHub fork, it was clear that I needed to make two files, and put them in a new ~/.grc directory: grc.conf for the general configuration of the tool, and conf.log4j for the colorization rules for the log4j files. The contents of the grc.conf file is pretty simple, and just tells grc what config file to use for what source files:

  # log4j files
  \b\w+\b.*log\b
  conf.log4j

the first line is the regex for the name of the file, and the second line is config file to use. Speaking of which, the conf.log4j was created to look like:

  # From http://fahdshariff.blogspot.ca/2011/04/
  #           highlighting-command-output-with.html
  # this configuration file is suitable for displaying log files
  #errors
  regexp=^.*(ERROR|Error|Exception).*$
  colours=bold red
  ======
  #stack trace
  regexp=^\s+at [^:]*:\d*\)$
  colours=red
  ======
  regexp=^.*(FATAL|Fatal).*$
  colours=on_red
  ======
  regexp=^.*(WARNING|Warning).*$
  colours=bold yellow
  ======
  regexp=^.*(DEBUG|Debug).*$
  colours=bold black
  ======
  # this is a time
  regexp=[^\d]*\d\d:\d\d:\d\d[\.,]{0,1}\d{0,1}\d{0,1}\d{0,1}
  colours=bold blue
  count=once
  ======
  # ip number
  regexp=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}
  colours=bold magenta
  count=more

At this point, I should be able to say:

  $ grc tail -f log/cw.log

and grace would see the log in the filename and pick the right config file and then colorize the output of the tail command. To make this even easier, I added a simple alias in my ~/.bashrc:

  alias lt="grc tail -f"

where, to me, the lt stands for Log Tail, and that's so easy for me to remember.

Now if I can just get that site to show up, and download the code. I should be ready to go. Fingers crossed...

UPDATE: I was able to download grc, and sure enough it's working like a champ!

This morning I was reminded that I read on Twitter last night that Xcode 6.3 was out, and along with OS X 10.10.3 and iOS 8.3, it was time to update the build tools to get the latest in Swift, as well as the libraries for the new platforms. It was a simple update right out of the App Store.

No mess... No fuss... Just better tools. Gotta love this company.

After updating my iPhone, I realized that I hadn't checked for updates to my laptop this afternoon, so I checked there, and sure enough - OS X 10.10.3 dropped as well. This is a 2.02 GB download and updates the Rescue Disk as well as several parts of Yosemite - including the new Photos app.

Now, I've been struggling with the Guest WiFi for a while, and to a point, I understand that there's a limit to the bandwidth they want to provide. But it's kinda silly to limit the bandwidth to what they have now given that you can get Comcast to deliver 50 Mbps to your home. It's kinda silly to cheap out on the network bandwidth.

So it's going to take me over two hours to download 10.10.3 to my personal laptop. It's OK... I've got time... it's just this kind of thing that makes it hard to convince people that this is a Software Company and not the traditional Service Company.

Until then, I wait and watch the download progress...

[3:28pm] UPDATE: finally finished, and then the 10.10 Command Line Tools needed to be updated as well. In the end, it was all done before I had to leave, and that's a win in my book.

I saw a tweet from Wil Shipley about iOS upgrading, so I decided to check and see is indeed there had been an update drop, or if it was a developer update for something he's working on. Turns out iOS 8.3 dropped, and so I spent the few minutes to download it and make sure it runs well.

They re-did the emoji keyboard, and it looks a little more dense to me, which is fine because before it was pages and pages of emoji. Not bad. They also said that it was a few bug fixes that impacted performance, and who doesn't like a little performance boost now and again?

After I was sure it was all OK, I sent out a text to me family saying that it was ready to use, and that I'd checked it out already. I know they don't need me to say this, but it's nice to know what someone is thinking of you - even if it's as silly as an automatic update of iOS.

They'll get it when they can, and that's fine.

I've been a Dropbox user for quite a while, and I've held off on the storing of Photos for now because of iCloud, and the fact that my iPhone and Mac already have those things handled quite nicely. But for all the other things I might need a cloud storage system for, I have to concede that Dropbox really is the best - by far.

When I need to move a file from one laptop to another - Dropbox is there. When I want to have it available on my phone - Dropbox is there. The seamlessness of Dropbox is really about as frictionless as possible. It really just something that fits in with the Mac user-experience very well. It Just Works.

What more could someone ask for?

I'm in the beginning of helping my niece learn to code, and it's so easy to share a folder with the code files to her. Simple. Easy. Painless. I just can't think of another product that's not made by Apple - that's this frictionless. I'm sure they exist, but none come to mind right now.

What a treat to be able to use great tools like these.

This morning I had to really smile... and even do my Happy Dance in my chair. I used something I found in the clojure library adworj to make reporting on Facebook ad server data as nice and clean as the code I lifted from adworj. I really liked adworj, but he wanted to handle credentials one way, and I had to have them in a separate store, so there were enough differences that the only really good things were the metadata on the reports from AdWords.

So what I decided to start with was a slight variant on the report metadata that he used. I started with a similar metadata structure, but instead of the macros and defrecord, I chose a simpler map of the data:

  (def reportstats-fields
    "The complete list of all allowed fields in the Facebook ReportStats
    reporting system keyed on the nice clojure name, and including the
    Facebook name and an optional parser for the returned data."
    {:time-start               {:name "time_start" :parse parse-long}
     :time-stop                {:name "time_stop" :parse parse-long}
     :date-start               {:name "date_start" :parse to-date-time}
     :date-stop                {:name "date_stop" :parse to-date-time}
     :account-currency         "account_currency"
     :account-id               {:name "account_id" :parse parse-long}
     :account-name             "account_name"
     :adgroup-id               {:name "adgroup_id" :parse parse-long}
     :adgroup-name             "adgroup_name"})

where I've clearly truncated the list of fields from Facebook, but you get the idea. There is either a string that's the Facebook field name, or there's a map with two keys: :name for the Facebook name of the field, and :parse for the function to mars the value into it's clojure data element.

My real divergence starts with how the reverse-mapping is done. I start by having a defined set of all the field names, as opposed to computing it on the fly over and over again:

  (def all-fields
    "Set of all valid keyword/field names for the Facebook ReportStats
    reports."
    (set (keys reportstats-fields)))

and then another that defines a map for how to get a Facebook field into closure-land:

  (def coercions
    "Map of the Facebook field names to the `:parse` functions for those
    fields - if they exist in the report definition. If they don't, then
    don't map anything and they won't then be handled in the
    reading/parsing."
    (into {} (for [[n md] reportstats-fields
                   :let [nf (if (string? md) md (:name md))
                         cf (if (string? md) identity (:parse md))]]
               [(keyword nf) (fn [v] [n (cf v)])])))

What I like about this approach is that we don't have to deal with the overhead of doing this for each report each time it's run. While that's probably not a big overhead, why? Why spend any time on this once it's done? The report structures are fixed in both cases, and there's just no reason for that level of flexibility.

What the coercions map gives me is a collection of functions to map the values coming back from Facebook - keyed by their Facebook field name, and suitable for inclusion into a map - with the correct clojure keyword for the field name. For instance:

  (defn coerce-record
    "Function to take a record from Facebook, and apply the known mappings
    to get it **back** into decent clojure names and datatypes."
    [m]
    (into {} (for [[k v] m] ((get coercions k) v))))

This function simply creates the right map (record) from the Facebook map (record) in a very simple, easy way. If there's a change in the formatting, just change the structure and it'll automatically be fixed in subsequent calls. That's nice.

Need to have a new field? Add it. Or change the name... it's all so easy to report like this.

Best of all, the real reporting code is just a vector of keywords to define what to get, and then a simple all. Very nice indeed!

This past weekend, I visited my sister in Iowa City, and my niece was there and asked about learning to program in Python. There were no classes in her graduate school (St. Louis University), and so I told her to just pick up a book and go at it. My sister pointed out that a lot of people learn better with the structure of a class, so I told my niece that I'd pick a book, and walk her through it - like a correspondence course in Python.

So I'm back in Python.

I sent her a few book recommendations - an O'Reilly book, and this book from Pragmatic Programmers. I wanted her to be able to read the excerpts of both and see which voice of the author she liked more. It was no surprise to me that she picked the Pragmatic Programmer's book - I have several myself, and they are excellent.

So this morning I'm getting the Pragmatic Programmer book, and scanning the first two chapters. I think it'll be better to use Python 3 from Homebrew as opposed to another source, just because it's self-contained, updatable, and simple. I need to write up a few notes for the first two chapters and then shoot them off to her with a link to a shared Dropbox folder for us to exchange files in - that was her idea, and a good one.

If I can just get these things downloaded on The Shop's WiFi, I'd be in business... but it's a pain in the neck, so it's taking a lot longer than it should. But it'll get there... eventually...