What's it all about, Alfie?

I just had a conversation with a co-worker, a good guy with all the best of intentions, and a solid core of wanting to help people. In the past, I've been a little taken aback by some of the things he's advised me to do - most notably, to have patience with this organization, and to accept that it'll be a year or so before people really start to listen to me.

He is not alone. I've heard that many times from several very senior people, and it still makes me giggle... as if the hiring process were somehow just a game, and the real vetting of people comes after they are hired. I still giggle...

But today it's about people being adults and the problem of putting a bunch of folks into a group and expecting them to play nice with one another. Most frustrating to me today is that you can't expect them to act like adults much of the time. Forget acting professional - I'm just asking that when someone is nice, polite, patient - that it would be really nice to have the person... I don't know... answer you?

I know this isn't unique to any one organization. Dilbert is filled with examples of this every week and we all laugh at the antics there... but it really happens, and it really is a chore to be doing a job you don't necessarily enjoy, and being polite, kind, patient - just doesn't seem to work.

I'm old enough to know that this has nothing to do with me, or this line of work, or the specific people involved. I've seen it at almost every place I've ever worked. Some are more up-front about it than others, but they all suffer from it. It's people. It's just people. Sadly, it's a frustration that I know has no outlet. It's just something that has to be accepted with a quiet shake of the head, and then let it go.

People don't change because they see a better way. They don't change because someone asks them. No... they change because they want to - simple as that. There may be a million reasons why - but they have to choose it, and nothing I can possibly do - other than what I've already done, is going to change that. It's their path, and they have to take the steps and learn what they have to learn.

And I do too.

I've worked with folks that identify requirements for a system, or a group of systems, that are going to be significant issues for the system - and might be used in maybe 10-20% of the cases that we'll run into. Yes... there is no doubt that if it is needed, then having it integrated into the very core of the system will make it very easy to add. But for those times when it's not needed, it's an unnecessary complexity that will cost every project in many little ways:

• Increased Dependencies - there is no need to include things that aren't used, but if you make it part of the scaffolding in the web app - it's there whether you want it tor not.
• Training and Discipline - since this is not natural for Javascript, it's going to mean that the developers that do this coding will have to be trained not to break the rules of the new scaffolding, and they will have to have more discipline than they'd otherwise have to in order not to violate a rule of the system and endanger the system.

and while this doesn't seem like a lot - it's really quite a bit when you're trying to bring in large groups of decoupled web developers. They don't mean to be careless, but UIs seem to get re-written about every nine months to a year, as the new Javascript framework is released, and if not compatible with the old. So it's almost like the UI code is throw-away code.

Not that I'm a fan fan of throw-away code, but I do recognize what's happening in this industry, and that's just whee things are headed. Evidence is hard to ignore.

So... when is a requirement not really a requirement? If it's for a small percentage - really it's hedging a bet that this will be needed. Because if it's never needed, or has limited need, the cost will far exceed the benefit, and this will be seen as a massively complex system. No one wants that.

For now, I'm being told it's a requirement and that means in it goes. If they are right - then it'll be one of the best projections I've ever seen, and they will be heralded as a true visionary. But if not... well... it could easily go the other way.

Gruber was poking fun at the MoviePass folks not being able to buy tickets for it's users, and brought up this fantastic faux commercial that aired on Saturday Night Live years ago, and it just made me laugh. It's the "Volume" at the end that really got me - still does.

One problem with changing an organization is that the Old Guard will do just that - Guard the old ways, and make it increasingly more difficult to make the changes that are part of the overall transformation of the organization. Change isn't easy, change can be annoying, horrible, scary, and painful. But an organization is similar to a living thing, in that it has to adapt (change) with the times, or it's going to face extinction.

This commercial is all about failure on an epic scale. It's something we've all seen in comics, stories, movies, and most of us - real life. It can be hard to see what you are doing when you are doing it - because the motivations are often set long before it comes around to execution. But it's there. If you can step back for just a second to take a look.

At work we are in the process of generating the technical architecture docs for all aspects of the platform - from data and cloud deployment to applications and integrations. It is meant to be a complete picture of what we need to really be able to take the current products globally, and not crush us with costs. As a part of that, things are understandably moving a little slowly, as this impacts how people view themselves, and their contributions to the business.

If and emergency room doctor sees what they do as the number of patients patched up, then telling them to take a part-time position in Administration, then they might likely feel like that which makes them uniquely them is being questioned. They like the testing of new cases - you never know what could walk through the doors any second, and to many - that's a rush.

I was talking to my Boss about this as I was getting a little antsy about this as I wasn't really doing a lot - other than writing a few short docs about different parts of the architecture. It's needed, yes, but I'm used to being far more productive, and so just writing docs is not really on the same scale as delivering systems and having users react to them.

His statement to me was that this disconnect was normal. Perfectly normal. Yes, it can be frustrating, but it's a sign that things are starting to change. Patience is important, but not just sitting around. That there was a playbook that was being run, and it was just as expected.

Just like the 76ers - Trust the Process.

I felt a little silly. More than a little silly. I didn't really trust that someone saw all this, and was working on other things that had to be in place for the next set of changes.

Patience can also mean to trust that just because you don't see it - doesn't mean it's not happening. And that's hard... because it is indistinguishable from nothing happening. In both cases you can't see anything being done. It's tough.

But I'm going to trust the process.

As a long-time fan of ObjC, and what NeXT and Apple have done with it, I was very interested when they released Swift 1.0 back in 2014, I was excited to see them move a little in the direction of Ruby. Everything in one file, more along the lines of a semi-scripting language, and while it wasn't complete to be sure, I was impressed that Apple was moving in this direction.

This week, I've been working on some architecture docs for work, and one of them is about the mobile platform space, and what we should be doing there to really increase the stability and speed of feature releases. Something that's been historically tough for NeXTSTEP and Apple with regards to ObjC is memory management and threading.

The days of retain/release were somewhat tough for new developers to ObjC to get right. It was really just about understanding the patterns for the use of retain and release and when to use autorelease. Once you had those down, you were pretty much golden. The run loop would pick up the instances with a retain count of zero, and didn't have to go looking for them. Memory didn't grow unless you retain-ed it and that was then pretty easy to spot with something like OmniAlloc.

Then Apple added Automatic Reference Counting and it got a little better, but we lost a few things in the mix. For example, compare-and-swap operations could not be known to have succeeded or failed by the compiler, so it could not know what to do with the retain count. I understood this, but ARC, as it was known, did a lot about removing the need for retain/release and so it got folks up to speed faster.

Losing a little functionality is something I can understand - if the overall gains are there, and there is at least some possibility of implementing similar functionality another way. This was disappointing, and Apple was clearly forcing ARC on the developers, as they had warnings about the use of retain and release.

Then I didn't pay a lot of attention to Swift 2 and 3, but I had reason to look at Swift 4 because of these architecture docs for the mobile platform. Wow. Apple has gone crazy and complex with Swift 4.

Several things have been added in 4.0 that wasn't in 1.0 - and some of them are probably quite nice. But at this point, the costs don't seem to outweigh the complexity of the language. But it certainly explains the Playgrounds as a concept - to get people writing code in a more friendly, less stressful environment.

Swift has lost the ability to use classic threading concepts like pthreads, and mutexes, and put in their place Grand Central Dispatch (GCD), which is a great idea for the kinds of "work unit" things that end up really complicating the code base when you have to do lots of different things - like most apps these days. But it shouldn't be used for little things... it's just too much overhead.

At least that's how tools should be used. You don't use a hand grenade to kill a mosquito, and you don't need Grand Central Dispatch to atomically increment an integer... or do you?

Yet that's what I was to come to learn from the docs I was reading. Apple is really pushing the concept of the main thread of the app, and that makes sense - it is the hold-over from the run-loop, and the single-threaded nature of classic Mac apps. This is a lot like the run loop of X11 apps, and a lot of the UI apps for a long time. But then why not abstract that thread from the programming model?

But there's more...

Because of the dynamic relocation of memory in iOS, there really can't be any fixed memory locations in the code. It's a cool feature, but now you have to add all the things in the language to indicate mutability and access control. They could have used this syntax to indicate which variables are to be fixed... or at least tracked carefully so that they always appear to have a stable location in the memory map of the process.

The end result is that Swift has become quite limiting in how to do things. You really have to go all in with their constructs. That would be fine if they had taken the extra steps and built in that level of abstraction. If they wanted Grand Central to play such an important role, then hide that role with some simple syntax. Make the 'jobs' that go onto the GCD queue simple and allow their specification to indicate their need for thread safety.

When I re-wrote some code from ObjC to Ruby, the project went from 2,700 lines to about 790. That's a significant savings, and makes code easier to write, maintain, and enhance. That's good stuff. But with the things that I'm seeing now in Swift 4 - I would not be surprised if the reduction to Swift wasn't at all like that for Ruby. I hadn't done the conversion at the time because Swift 1.0 didn't have the kinds of class libraries that I needed to make the conversion. Now, it probably does.

Some day I'll actually have to knuckle down and write the project in Swift and see what it turns out to be. I hope this is just a step along the way for Swift... just a snapshot of where they want to go.

Well... it's the first day at the new job, and I've got a lot of good feelings about this place. First, it's about giving back - at least a little to the folks that grow all the food we eat. After spending a lot of time making "business" - it's nice to be a part of something that is really making a thing. Nice.

Then there's the idea of possibly staying with Clojure - there's a good bit of it here, and it's supported at the highest levels, and that's both a good sign, and a nice chance to keep doing some of the really good work I've been doing for several years now.

I have no idea what will happen… it’s only the first day, but I’d really like to think that this could be a place to stay for a while. I’m getting tired of the stress of the last year.

Today was my last day at the Shop. I was excited about leaving, but Management in IT did some Spring Cleaning and several of us found that we were dust bunnies in the process. I've spent the last two weeks transitioning all the work I've done to others that are staying, but in truth, that meant a couple of meetings because the transition was rather poorly planned. Their choice.

I'll get a nice severance out of this, and that's nice, and in truth, it was probably the best thing for me at this time... over the last several months I've been feeling that there's been an axe hanging over my head, and I even emailed the CEO on Christmas to ask if that's the case, then maybe we needed to make a transition plan to move me out.

His response was "No, you're just what we need!" and from that, I assumed that things were stable. That would be a poor assumption.

So, today is it. Nearly 3 years at the Shop, and now it's time to find a new place. The search continues.

In the work I'm doing, I've got a service that chooses to return a group of files in a standard Zip file format, and then I can easily store it in S3 using amazonica and a little bit of code:

  (defn save!
    "Given a byte array or string `data` and a bucket `b`, saves the
    bytes as an encrypted file with file name `fname`. Optional keys
    `content-type` and `overwrite` may also be passed, where the
    `:content-type` key indicates the content type (defaulted to S3's
    inference), and the `:overwrite` key indicates whether to replace\
    the file if it already exists (defauted to false)."
    [data b fname & {:keys [content-type overwrite input-stream-size]}]
    (if (and (or (string? data) (byte-array? data) (instance? InputStream data)) b)
      (let [nm (or fname (uuid))
            echo (if-not fname {:name nm})
            [inp cnt] (cond
                        (string? data)     [(ByteArrayInputStream. (.getBytes data))
                                            (count data)]
                        (byte-array? data) [(ByteArrayInputStream. data)
                                            (count data)]
                        (instance? InputStream data) [data input-stream-size]
                        :else [nil nil])]
        (if (or overwrite (false? (file-exists? b nm)))
          (try-amzn 3
            (merge
              echo
              (put-object (get-cred)
                          :bucket-name b
                          :key nm
                          :input-stream inp
                          :metadata (merge
                                      (if content-type {:content-type content-type})
                                      (if cnt {:content-length cnt})
                                      {:server-side-encryption "AES256"}))))
          (merge echo {:error "FileNotSaved"})))))

But what if it's a zip file? If we want to do this one-pass, we have to load the entire contents of the file into memory, and then piece it apart. That's certainly possible, but what if the files are very large? Why not unzip the stream, and write it back to S3 as a stream?

Then, we don't have to have a large memory footprint in order to process the large zip files. That would be nice.

  (defn unzip!
    "Function to unzip the provided file in the provided bucket into the same
    S3 bucket ehere the directory is the name of the file - without the extension,
    and all the files from the zip file are deposited into the directory under
    their names in the zip archive. The downside of this function is that it has
    to read the zip file from S3 'n' times - one for each of the files in the
    zip archive. That means that it's not at all fast. This returns a sequence
    of all the files that have been unzipped to S3:
      [\"13B7E73B053C497D82F8FCC28FC8127F/13b7e73b053c497d82f8fcc28fc8127f.XML\"
       \"13B7E73B053C497D82F8FCC28FC8127F/Form0001.PDF\"
       \"13B7E73B053C497D82F8FCC28FC8127F/Index.ctl\"]
    "
    [bkt fname]
    (if (file-exists? bkt fname)
      (if-let [base (some identity
                          (rest (re-matches #"(?i)(.*)\.zip$|(.*)\.xfr$" fname)))]
        (let [afn (atom [])
              push (fn [ze zis]
                     (if (not (.isDirectory ze))
                       (let [to-file (str base "/" (.getName ze))
                             res (save! zis bkt to-file
                                        :overwrite true
                                        :input-stream-size (.getSize ze))]
                         (if-not (:error res) (swap! afn conj to-file)))))
              s (get-stream bkt fname)]
          (with-open [z (ZipInputStream. s)]
            (doseq [e (entries z)
                    :let [en (.getName e)
                          ms (get-stream bkt fname)]]
              (with-open [mis (ZipInputStream. ms)]
                (let [me (entries mis)]
                  (push (first (drop-while #(not= en (.getName %)) me)) mis))))
            @afn)))))

The result is something that has to read the stream once for each file in the unzipped file, but then it can write each of these back to S3. It's not super network efficient, but the existing library closes the stream when it's done reading a file, and if they just hadn't done that, then I could have done it all in one pass.

Still... this is nice. It works great, and does just what I need.

This month has been a lot of work on the Auto App Package for The Shop. This is the package of legal documents that you get from your mortgage company after you give them all your information, and the details of the house you want to buy, and they work up a quote for the loan, and there are all these legal documents you need to sign before the loan goes into underwriting. It's a manual process, and they want it to be automatic.

So this is what I've been doing. It's not really amazing work, but it's pulling together a lot of the details from many different sources, an do all the annoying edge-case math to get all the numbers to line up, and then present it in a way so that when the other groups that need to use this aren't ready to finish this, we can go into the field with it and deliver value until such time as they are.

It's just another month at The Shop.

I'm a huge fan of clj-time - it's made so many of the time-related tasks so much simpler. But as is often the case, there are many applications for time that don't really fall into the highly-reusable category. So I don't blame them for not putting this into clj-time, but we needed to find the start of the year, start of the month, and start of the quarter.

The slight wrinkle is that the quarter could be on a few different starting cycles, so we had to be able to allow for that.

 (defn start-of-year
   "Gets the start of the year (midnight, 1 Jan) for the specified
   date-time."
   [dt]
   (date-time (year dt)))
 
 (defn start-of-month
   "Gets the start of the month (midnight) for the specified date-time."
   [dt]
   (date-time (year dt) (month dt)))
 
 (defn start-of-quarter
   "Gets the start of the quarter for the specified date time.
 
   This function assumes the standard quarterly cycle (Jan-Apr-Jul-Oct).
   A different cycle can be generated by providing the 1-based index
   (e.g., 1 => Jan) of the cycle start month."
   ([dt]
     (start-of-quarter dt 1))
   ([dt start-month]
     (let [offset (mod (- (month dt) start-month) 3)]
       (minus (start-of-month dt) (months offset)))))

None of this is Rocket Science, but it's nice to not have to mess with these conversions in the remainder of your code - which is kinda the point of the functional part of the language - no?