Case Insensitive Regex for Clojure

March 24th, 2016

Clojure.jpg

One of the things that I find I need to do every now and then is to have a case-insensitive regex pattern match in clojure, and I always have to spend 15 mins looking for it on the net. Not that it's hard to find, but it's 15 mins, and if I know I wrote about it here, then I can do a much quicker search, and get the same answer.

The key is that the Java regex parser recognizes two interesting options to the source for the regex: (?i) and (?iu). The first, (?i), is the flag to indicate that the entire regex is to be considered case-insensitive. The second, (?iu) is for unicode support.

So how's it work?

  (re-find #"Cook" "This is how to cook.")
  => nil
 
  (re-find #"(?i)Cook" "This is how to cook.")
  => "cook"

It's simple and easy, and typically saves you from calling lcase on the string before the regex test.

Upgraded to Adium 1.5.11b3

March 16th, 2016

Adium.jpg

This morning I noticed an upgrade notice in Adium, and while I wasn't going to go back to 1.5.11b2 - due to the Sparkle issues and security, I read it and the maintainers had created 1.5.11b3! While I know it's not making a lot of progress, it's something to see these updates shipping - it's not dead. And that's great news in my book.

Sure, Microsoft has killed Hotmail's IM support - and that's a shame, I don't understand why they wouldn't want to have users on their service, but that's their choice and not mine. I still hope that one day libpurple will support it, but that's a wish, not something I'm really holding my breath for.

But today it's nice to have Adium updated with the latest Sparkle and a few other fixes. Nice!

Back to Adium 1.5.10.1

February 10th, 2016

Adium.jpg

This morning I saw that Adium 1.5.10.1 had been released, and while I've been running 1.5.11b2 for quite a while, and while I was reluctant to go back to 1.5.10.1, because there had to be something in 1.5.11b2 I liked, I realized that there really wasn't. Not a thing. And as long as I had the ability to get back to 1.5.11b2, I could download it and go back to it again.

No downside.

Then again, it's important to note that Adium hasn't been getting a lot of love from the maintainers lately. I have been toying with the idea of getting the code and building it, and then seeing if I could fix the Microsoft Live connection so that I could once again get my Hotmail IM again.

Then word spread of the Sparkle security issue. I didn't know the depth of the issue on the software I have - but certainly a lot of the OS X software uses Sparkle, and the HTTP vs HTTPS endpoints is a big issue, and I'm glad they have fixed it, but then Adium reported a new version with the new Sparkle, and I decided that maybe 1.5.10.1 isn't so bad.

Maybe it's stable, and not abandoned. Maybe I just need to realize it's working great for me - save the Hotmail issue, and that's good enough.

So back I am. Happy to be here.

Being Happy is Not the Absence of Sadness

January 31st, 2016

Path

This morning I realized that may path really has shown me something I never expected to see: that being happy does not require the absence of sadness. Not at all. Instead, it now seems to me that being happy - having a giggle, or a good laugh at something - is really about being able to focus on that which is right in front of you, and not what has been happening, or maybe is continuing to happen.

This isn't to say that the sadness isn't there - I felt it very much this morning, but that didn't stop the giggle and smile I got from what I was doing and looking at.

It's been so easy to focus on the darkness - as if it needed fixing, and because it was present, nothing could be positive until that negative was erased. But that's not the case at all. It seems that sometimes you have to accept that erasing the negative will take a little more time than you have right now, and yet that shouldn't stop you from enjoying what you are doing.

Like getting out of debt. It may take you a couple of years, but that's no reason to not enjoy today... you're doing everything you can today to deal with that, but it's just going to take more time to finish off that goal.

Sadness, too.

Smile more. It's not ignorance, or denial... it's just enjoying the thing you are doing right now.

Sharing Login Auth Tokens

January 30th, 2016

Javascript

Over the past several months we have built several Clojure/Javascript apps at The Shop, and each one has needed to have some level of authentication, and thankfully, that's all been provided by the Security Service that was already built and maintained by another guy at The Shop. It works, what else would we need? You get authentication tokens (UUIDs) and you can get information about the user based on the token, and you can't really guess an auth token for a user, so they are pretty decently secure.

But they were app-specific because we tied them to the cookies on the pages, and those pages were all from a site that was different for each app we were building. This meant that you'd have to login to three, four apps a day with the same credentials, and still not have a clean, guaranteed way of invoking a page in one app from a page in another.

This is what I wanted to solve.

This morning, it came to me - there had to be a way to store the auth tokens in a cookie that was across the apps, and then just use that as a standard for all apps. Simple. Since it was still secure (enough), we didn't have to worry about folks yanking the token, or even knowing who the token belonged to.

The question was How?

Thankfully, jQuery has this all solved with the domain: tag in the $.cookie() function. All we needed to do was to have a consistent upper-level domain for the apps that also worked for local development.

What I came up with was a simple function:

  function loginDomain() {
    var here = document.domain;
    if (here === 'localhost') {
      return here;
    }
    var parts = here.split('.');
    parts.shift();
    return parts.join('.');
  }

so that we would have a single login for the production hosts on theshop.com and another for the development hosts on da-shop.com. This would work perfectly if I could update all the places that the cookie was accessed.

What I learned was that the accessing of the cookie did not need to have the domain: tag, but setting it - or deleting it did. So for example, to read the token, I'd say:

  var tok = $.cookie('shop-auth-token');
  if (tok) {
    // ...
  }

and no matter where the cookie was saved, if it's in the "domain path", then it's OK. But when I needed to set the cookie I'd need to:

  var exp = new Date();
  exp.setTime(exp.getTime() + (8 * 60  * 60 * 1000));
  $.cookie('shop-auth-token', tok, { domain: loginDomain(),
                                     expires: exp );

and when I needed to invalidate it:

  $.removeCookie('shop-auth-token', { domain: loginDomain() });

Once I got these all set up, and in each application, one login on the specific login domain would then be successfully used on all apps in that domain with this code. Sweet! It worked like a charm, and it took me about 90 mins to figure this out, and put it in all the apps we have.

Big win!

Another Year in the Books

December 31st, 2015

Cake.jpg

Well... I can't say as I'm all that surprised that I've had a hard time keeping up with writing this year. It's been a little better this year, but it's not really been a great year, and I know myself well enough to know that I need to be in a reasonably positive frame of mind to be able to write on a regular basis. But hey... I can keep trying.

Advent of Code

This year, a friend of mine started doing the Advent of Code problems, and so I joined in. They started off fun and interesting - almost in a little competition with my friend who was also using clojure for the solutions. It was nice, and it made me smile, and fir that I am very grateful.

But then the problems got to be massive searches, as opposed to interesting little problems, and the fun faded. Still, it's something that I think is a lot of fun to try - and do all the problems you want to do. For fun.

Change of Jobs

This year I changed jobs, and this time it was to a Shop that wanted to do clojure. There were a lot of good signs in the interview, and so far, it's been pretty decent. Nothing is perfect, so it's learning what is important to me, and what isn't, and learning to live with the latter, and relish the former.

I would like to be here next year at this time, but who knows... I know I don't, that's for certain. But I will try, and maybe that's all I need to do.

My Friends and My Recovery

Probably the most significant thing this year was that I actually saw old friends that I haven't seen in nearly 20 years. These are guys that defined my childhood, but since my marriage, and moving to Chicago, I really haven't seen them. A year or so ago, one of my old friends called me, and we started talking. Every month or so, he'd call, and check-in on me to make sure I was still alive.

Finally, this year, I was invited to his Dad's 85th Birthday Party in Indy, and I decided to go. It was a nice time, and it was very nice to see my friends again. We have all changed a lot - mid-30s to mid-50s is a lot of change. And in the end I think we all cut each other a lot of slack, and can move forward from here.

My recovery is going about like I figure it'll go - slowly, but inevitably. There is nothing that replaces the passage of time. It's not a cure-all, because people can hold grudges for a very long time, but in my case, I think it's a vital part because "distance" is the thing I need most. Distance provides perspective, and that's what I've lost in this Season. Hopefully, things will look better in a few months, but I'm guessing it's probably a few years.

But I hope it comes.

Indexing Dates in Postgres JSONB Data

November 25th, 2015

PostgreSQL.jpg

Postgres is an amazing database. With 9.4's JSONB datatype, and a little clojure code, I can save data structures to and from Postgres, and have them query-able by any JDBC client. It's just amazing. But recently we had a problem at The Shop where we needed to index a field in a JSONB element, but it was a date, and so we ran into the problem of Immutable Functions.

Simply trying to create an index on the field:

  CREATE INDEX idx_d ON my_table ((fields->>'DateStarted'))::DATE;

didn't work with Postgres saying you can't have a mutable function in the creation of the index. So it was time to dig out the browser and look up what's going on. Sure enough, this was something well-known, and all involving the conversion of dates by locale, and why that was not allowed for an index.

Makes sense, but I still needed an index because the queries using this field in the WHERE clause were taking 6+ min, and we needed to really bring them down - like way down. Then I found that you could make functions that were tagged as IMMUTABLE and so made these:

  CREATE OR REPLACE FUNCTION mk_date(src VARCHAR)
  RETURNS DATE
  immutable AS $body$
    SELECT src::DATE;
  $body$ LANGUAGE SQL;
 
  CREATE OR REPLACE FUNCTION mk_datetime(src VARCHAR)
  RETURNS TIMESTAMP
  immutable AS $body$
    SELECT src::TIMESTAMP;
  $body$ LANGUAGE SQL;

the reason I chose to use ::date and ::timestamp is that they handle NULL values very well, and the corresponding functions don't do that so well.

With these functions, I was then able to create the index:

  CREATE INDEX idx_d ON my_table (mk_date((fields->>'DateStarted')));

and then as long as I used the same functions in my WHERE clause, everything was great.

The result was a 1000x increase in speed - 319,000 msec to 317 msec. Amazing.

I do love Postgres.

Just Slogging through Stuff

October 30th, 2015

Path

There are days that I don't know how I'll make it through the day back to my little house, and the one place that I really feel safe. And I'm continually surprised that every day around 6:00-ish, I find myself looking at that facade, and unlocking the door, and walking in and feeling an amazing sense of relief. I'm not better - I'm just a little bit stronger. I can take the heartache and pain that seems to follow my day-to-day existence.

A very nice friend of mine once said "It either gets better, or you get used to it. Either way, it seems like it's better." Not bad advice. It's not something that's easy to heed, but it is true nonetheless. Very true, indeed.

So this past month has been a lot more more of getting used to it than it getting really better. Just gotta keep slogging through the muck...

Upgraded to iOS 9.0 and AdBlocking

September 17th, 2015

iPhone 4

This morning I saw that iOS 9 dropped from Apple, and I upgraded my iPhone right away - why not, right? I've been hearing of all the nice things in iOS 9, and one of the things I was really interested in was the Ad Blocking in WebKit/Safari. I've worked in the Ad business now, and I have to say that while it has the potential to be a very useful thing, it's really degenerated into a cheap way for some folks to get a little revenue for their web sites without having to do anything other than a little code added to their pages.

I get it... you pay $20/yr for a site, and you'd like to see $30/yr in revenue, but the "easy money" is not in advertising - at least not now. It's the wrong game to let Double-click or AdWords into your site. They just they show is just not good stuff - and they'll say it's not their fault - it's the publishers - and it is, but it's all a race to the bottom.

So I've just gotten sick of it, and got Peace, the ad blocker for iOS 9 from Marco A. as he's one of the good guys, and he teamed up with Ghostery, the desktop Chrome/Safari ad blocker, and it's pretty slick. The point was just to get a better experience on my iPhone as well as getting rid of all the tracking. If a site wants data about me, and they offer a decent reason, I'll do it. But as it is now, Google is the company that's collecting all this data, and I'm not so sure I like them knowing so much.

Just too much power.

So I turned on the ad blocking with Peace on iOS 9, and with Ghostery on my laptops. I have whitelisted a few sites: Bank of America, Loggly, Daring Fireball... they are OK. But the rest - no thanks. Too much.

Datadog Gauges in Clojure

August 18th, 2015

Datadog

The Shop is big into Datadog, and it's not a bad metrics collection tool, which I've used very successfully from clojure. Based on the use of the local Datadog Agent (freely available from Datadog) and how easily it's placed on linux hosts - AWS or otherwise, it's a clear win for collecting metrics from your code and shipping them to a nice graphing/alerting platform like Datadog.

The code I've set up for this is pretty simple, and based on the com.codahale.metrics java libraries. With a simple inclusion into your project.clj file:

  [io.dropwizard.metrics/metrics-core "3.1.0"]
  [org.coursera/dropwizard-metrics-datadog "1.0.2"]

you can then write a very nice metrics namespace:

  (ns ns-toolkit.metrics
    "This is the code that handles the metrics and events through the Dropwizard
    Metrics core library, which, in turn, will ship it over UDP to the DataDog
    Agent running on localhost."
    (:require [clojure.tools.logging :refer [infof debugf warnf errorf]])
    (:import [com.codahale.metrics MetricRegistry]
             [org.coursera.metrics.datadog DatadogReporter]
             [org.coursera.metrics.datadog.transport UdpTransportFactory
                                                     UdpTransport]
             [java.util.concurrent TimeUnit]))
 
  ;; Create a simple MetricRegistry - but make it only when it's needed
  (defonce def-registry
    (delay
      (let [reg (MetricRegistry.)
            udp (.build (UdpTransportFactory.))
            rpt (-> (DatadogReporter/forRegistry reg)
                  (.withTransport udp)
                  (.withHost "localhost")
                  (.convertDurationsTo TimeUnit/MILLISECONDS)
                  (.convertRatesTo TimeUnit/SECONDS)
                  (.build))]
        (.start rpt 5 TimeUnit/SECONDS)
        reg)))
 
  ;; Somewhat faking java.jdbc's original *connection* behavior so that
  ;; we don't have to pass one around.
  (def ^:dynamic *registry* nil)
 
  (defn registry
    "Function to return either the externally provided MetricRegistry, or the
    default one that's constructed when it's needed, above. This allows the user
    the flexibility to live with the default - or make one just for their needs."
    []
    (or *registry* @def-registry))

And then we can define the simple instrumentation types from this:

  ;;
  ;; Functions to create/locate the different Metrics instruments available
  ;;
 
  (defn meter
    "Function to return a Meter for the registry with the provided tag
    (a String)."
    [tag]
    (if (string? tag)
      (.meter (registry) tag)))
 
  (defn counter
    "Function to return a Counter for the registry with the provided tag
    (a String)."
    [tag]
    (if (string? tag)
      (.counter (registry) tag)))
 
  (defn histogram
    "Function to return a Histogram for the registry with the provided tag
    (a String)."
    [tag]
    (if (string? tag)
      (.histogram (registry) tag)))
 
  (defn timer
    "Function to return a Timer for the registry with the provided tag
    (a String)."
    [tag]
    (if (string? tag)
      (.timer (registry) tag)))

These can then be held in maps or used for any reason at all. They automatically send their data to the local Datadog Agent over UDP so there's no delay to the logger, and since it's on the same box, the likelihood that something will be dropped is very small. It's a wonderful scheme.

But one of the things that's not covered in these metrics is the Gauge. And there's a really good reason for that - the Gauge for Datadog is something that is read from the Datadog Agent, and so has to be held onto by the code so that subsequent calls can be made against it for it's value.

In it's simplest form, the Gauge is just a value that's read by the agent on some interval and sent to the Datadog service. This callback functionality is done with a simple anonymous inner class in Java, but that's hard to do in clojure - or is it?

With Clojure 1.6, we have something that makes this quite easy - reify. If we simply add an import:

  (:import [com.codahale.metrics Gauge])

and then we can write the code to create an instance of Gauge with a custom getValue() method where we can put any clojure code in there we want. Like:

  ;;
  ;; Java functions for the Metrics library (DataDog) so that we can
  ;; constantly monitor the breakdown of the active docs in the system
  ;; by these functions.
  ;;
  (defn cnt-status
    "Function that takes a status value and finds the count of loans
    in the `laggy-counts` response that has that status. This is used
    in all the metrics findings - as it's the exact same code - just
    different status values."
    [s]
    (reify
      Gauge
      (getValue [this]
        (let [sm (first (filter #(= s (:status %)) (laggy-counts)))]
          (parse-int (:count sm))))))
 
  (defn register-breakdown
    "Function to register all the breakdowns of the loan status counts
    with the local Datadog agent to be sent to Datadog for plotting. This
    is a little interesting because Datadog will call *these* functions
    as needed to get the data to send, and we will control the load by
    using memoized functions."
    []
    (.register (met/registry)
      "trident.loan_breakdown.unset"
      (cnt-status nil))
    (.register (met/registry)
      "trident.loan_breakdown.submit_to_agent"
      (cnt-status "Submit to Agent"))
    (.register (met/registry)
      "trident.loan_breakdown.submit_to_lender"
      (cnt-status "Submit to Lender"))
    (.register (met/registry)
      "trident.loan_breakdown.submit_to_lender_approved"
      (cnt-status "Submit to Lender - Agent Approved"))
    (.register (met/registry)
      "trident.loan_breakdown.lender_approved"
      (cnt-status "Lender Approved")))

What I like about this is that I can allow the Datadog Agent to hit this code as often as it wants, and don't have to worry about the freshness of the data - or an excessive loan on the server resources for being hit too much. I can simply memoize the functions I'm using and then control the load on my end. It's very clean, and very nice.