Archive for the ‘Cube Life’ Category

Some Deals are Too Bad to Take

Friday, May 8th, 2015

PHB.gif

I had a meeting this morning with my manager and I was reminded of the verse:

For what will it profit a man if he gains the whole world and forfeits his soul?
-- Matthew 16:26

My manager is a consultant - been here two weeks - had a plan on Day 1 to change how The Shop was going to handle all data storage and reporting tasks. Based on what? Nothing. His experience. Not that he couldn't make it work, but the agreement to bring him in and trust him to implement the solution without knowing what the long-term costs would be is simply making a bad deal.

I understand the problems businesses can get into. I've been there. But I'm glad to say that I got out before I had to make any of those kinds of deals.

This guy is going to achieve what he wants, and give to management what they want. And when they get it, they will look at it, and say "This wasn't worth it." And they will be right.

But it will be too late.

Taking a Walk for The Good Stuff

Friday, May 8th, 2015

Diet Coke

There are very few things I'll walk two floors for - but a nice Diet Coke is certainly one of them. There is just nothing like a good Diet Coke for me. Coffee doesn't do it, Sprite, Lemonade - forget about it. So when I found this morning that the fountain on this floor was serving up soda water where once Diet Coke flowed freely... well... I had to grab some glasses and head downstairs. To the other fountains in the building.

Now I have a reasonable supply and I'm sure the kind gentlemen (or ladies) whose job it is to detect such outages will be right on the case and correcting this injustice as soon as possible.

But until then... Ah...

Loaded up on Diet Coke

LinkedIn API for a Recruiting Tool

Thursday, May 7th, 2015

LinkedIn

I was asked today to look at the LinkedIn API to see if I could access the data at LinkedIn to make an advanced recruiting tool for the Recruiters here at The Shop. The idea was to take a resume we received, match it to a LinkedIn profile (I'd venture that 90%+ of them are there), and then use advanced analytics to rate the prospective resumes for potential success at this job.

It's an interesting idea. The real advantage for LinkedIn is that companies like ours pays several thousand dollars a month to have access. With this kind of tool, that same data could be used to classify candidates by the data they have entered, and then use a nice predictive model to say who is most likely to succeed. It's simple feedback.

We take everyone that's been successful at this company. Reference their LinkedIn profiles for the training data, and then use any and all reviews to say which of these people are likely to be the successful ones - based on all the classified data that's on LinkedIn.

It's kinda neat. We don't have to wonder what factor(s) matter most - we can get all that LinkedIn has in their API, and then use the success factor - say a 1 - 5 rating as the outcome, and then train away. After that, every submitted resume can run through the trained net, and come up with a score and a confidence number. Pretty simple.

It's not meant to be fool-proof, but when you have a ton of openings, it's nice to be able to have something that whittles down the list of thousands to hundreds, or less - so that you can really focus on these people.

We'll see where it goes - if it goes anywhere.

Building with jQuery and Bootstrap

Wednesday, May 6th, 2015

JQuery Framework

This morning I decided that I wanted to be able to start doing decent web site design. This means getting a lot better at jQuery - even though it's old, it's still useful and it's out there, and learning Bootstrap. Now the latter was really the influence of the folks at Groupon, because if they used it, then it had to be a good tool. Sure, they improved a lot on it, and they were really fond of their whitespace, but I want to get to the point that I can make a decent web site with decent tools so that it's not like I'm just HTML/CSS and nothing else.

So this morning I did this:

better colors

it's not amazing, but for about 3 hrs, it's not bad, and it's backed by the data, and I've got a good start on the way to make it all work in a Clojure app. So that's what I'm going to write up, so I don't forget all the issues.

Get The Tools

First off, go get the jQuery and Bootstrap downloads. I'm not going to be fiddling with the code, so I don't need to build it, just get it and use it. One important thing I noted was that Bootstrap 3.3.4 was very particular about the version of jQuery that needed to be used. Specifically, 1.11.2.

Start in the resources/ directory of the Clojure project and make a public directory where all the static assets will be served from. Anything in resources/ will be packaged up into the 'uberjar', so once there, it's save to develop and deploy on these files. When you are done, make sure the filesystem looks like this:

  resources
      +- public
          +- index.html
          +- css
          |   +- bootstrap.min.css
          |   +- bootstrap-theme.min.css
          |   +- theme.css
          +- js
              +- bootstrap.min.js
              +- jquery-1.11.2.min.js

where the contents of the theme.css is very simple and yet needs to be there to make Bootstrap look decent:

  body: {
    padding-top: 70px;
    padding-bottom: 30px;
  }
 
  .theme-dropdown .dropdown-menu {
    position: static;
    display: block;
    margin-bottom: 20px;
  }
 
  .theme-showcase > p > .btn {
    margin: 5px 0;
  }
 
  .theme-showcase .navbar.container {
    width: auto;
  }

In the index.html there will be a few places where you need to match the names of these -showcase - and possibly make the names unique to your project.

Setting Up the Home Page

Now that you have the tools copied into the right places, we need to throw together the boilerplate for the home page. This is pretty easy, and Bootstrap examples all have it as clear as day for you:

  <!DOCTYPE html>
  <html lang="en">
    <head>
      <meta charset="utf-8">
      <meta http-equiv="X-UA-Compatible" content="IE=edge">
      <meta name="viewport" content="width=device-width, initial-scale=1">
      <!-- The above 3 meta tags *must* come first in the head;
           any other head content must come *after* these tags -->
      <title>Bartender</title>
      <!-- Bootstrap core CSS -->
      <link href="/css/bootstrap.min.css" rel="stylesheet">
      <!-- Bootstrap theme -->
      <link href="/css/bootstrap-theme.min.css" rel="stylesheet">
      <link href="/css/theme.css" rel="stylesheet">
    </head>
 
    <body role="document">
 
 
 
      <!-- Bootstrap core JavaScript
      ================================================== -->
      <!-- Placed at the end of the document so the pages load faster -->
      <script src="/js/jquery-1.11.2.min.js"></script>
      <script src="/js/bootstrap.min.js"></script>
      <script src="/js/bartender.js"></script>
      <script type="text/javascript">
        // when we are all done loading the page, hit the data...
        $( document ).ready(reloadAllData());
        // $( window ).load(reloadAllData());
      </script>
    </body>
  </html>

In the middle there, you can put in all the things you want. For me it was a simple menu and a table in a container. All this is on the Bootstrap examples and it's easy to pick-n-choose.

Wiring it in with Ring

The other big thing was to have it all work from the Clojure web project. There are a few things that needed to be done here, but in the end, they really aren't all that hard, and the results are quite nice.

First, make sure you have the necessary ring components in the project.clj:

  [ring/ring-core "1.3.2"]
  [ring/ring-jetty-adapter "1.3.2"]
  [ring.middleware.jsonp "0.1.6"]

and then make sure to require:

  [compojure.route :as route]
  [ring.util.response :as resp]

so that you can add the route:

  (GET "/" []
    (resp/redirect "/index.html"))

and then at the end of all the routes, add:

  (route/resources "/")
  (route/not-found "<h1>Page not Found!</h1>")

where the resources function specifies where to find the static resources, like the index.html.

At this point, restart the web server, and things should just work. Not very interesting, but the log will show the static resources being requested, and that verifies that they are found, and loaded, and you can see this all from the javascript console as well.

Adding the Database Access

At this point, we need to add the useful stuff. To keep things separate, I created a separate bartender.js to hold all the javascript I was going to write, and for now, that is simply:

  /*
   * Function to add a single row to the 'last-import-times' table on the
   * main page. This will be called for each key/value pair returned from
   * the RESTful API on the server, and it's just a simple way to make it
   * easy to see what's loaded up in the service.
   */
  function addLastImportRow(src, vals) {
    var dates = vals.report_date
    var tr = '<tr>';
    tr += '<td>' + src + '</td>';
    tr += '<td>' + vals.generated_at + '</td>';
    tr += '<td>' + dates[dates.length - 1].substring(0,10) + '</td>';
    tr += '<td>' + dates[0].substring(0,10) + '</td>';
    tr += '</tr>';
    $("#last-import-times").append(tr);
  }
 
  /*
   * Function to reload the data in the 'last-import-times' table on the
   * main page. This will clear out what's there, and hit the endpoint that
   * will bring back all the data it needs to properly make the page.
   */
  function refreshLastImportTable() {
    console.log("refreshing the last-import data");
    // remove all the non-header rows in the table
    $("#last-import-times tr").remove();
    // make the call to get the data from Bartender
    $.getJSON("/v1/latest-imports", function(data) {
      $.each(data, function(k, v) {
        addLastImportRow(k, v);
      })
    })
  }
 
  /*
   * Simple function to be called when the page is reloaded and we need to
   * update ALL the data on the page.
   */
  function reloadAllData() {
    refreshLastImportTable();
  }

There are some nice things here that I learned. First, jQuery may be old, but it's awfully nice for simple things like I'm building. Sure, if you want to make a single-page app on the web, it might be weak, but for me, it's very nice. And stable.

It's easy to ask for JSON from the server, and then have a function that simply takes each key/value pair and puts it in the emptied table. In the HTML I put an id on the table to make it easy to find, and that's all that I needed. I cleared it out, then one by one, I added the new rows back in. Works like a charm.

With this, I should be able to start making decent looking web sites. Much more to learn.

It’s Always the People

Monday, May 4th, 2015

Management.jpg

I took a walk with a friend today to get lunch. I had already eaten by they hadn't, and needed to get out of the office and vent a little. The problem they were having was a classic Little Red Hen story: someone does a lot of work - in this case months of work, and then when it starts to look interesting and executives are hyped about it, there's all kinds of folks wanting to take credit for the one person's work.

I've run into this quite a bit in the last decade, and I've actually come to bargain this with a manager. I let them take credit for the work, and they allow me to work in a way that I find most productive. In the end, they can take all the credit they want because there's no way anyone will really think they did it - they were the Manager - not the Doer, but they love the attention, and the sign on for more. Eventually, they are in a difficult position because they can't hope to have the same productivity from other people, and so the relationship becomes more symbiotic. Which is good.

But this is my friend's first experience with this, and I can remember being just as upset as they are, when it first happened to me. So on the walk to pick up a sandwich we talked about these problems, and what the real solution was. They were convinced that it was the structure of an organization that breeds this problem, but my assertion was that it's the people in it.

It's always the people.

Their point was that there was no need for the traditional manager in a successful organization. While I'm not saying that managers are required, I accept that most people who run organizations feel more comfortable when there's a Chain of Responsibility - if not a Chain of Command, so that blame can be assigned when things go wrong.

The only time I saw this not being used was in small start-up companies. There, it's assumed that everyone shares in the responsibility, and so there's no need or even interest in assigning blame. But when an organization gets big enough, and that size depends on the group of people, then it's either time to stop growing, or start compartmentalizing the organization, and that leads to the silos and the need to be able to attribute responsibility and it's evil alter ego - blame.

It's hard to forego growth when times are good. It's something I've learned first-hand at the company I started. You want to do well, and you are happy when things are going well. People are happy, work is good, and you think "Why not?" But that's what leads to this problem because the question is the wrong one to ask. Better would be "Why?"

It's not easy to do, I know. But if we had asked "Why?" more than "Why not?" - I might still be at that company I started. But I'm not, and it was a direct result of the decisions we made. In the beginning we had to achieve consensus... after several years, it wasn't required, and so we let it slide. That was a mistake. I wish he'd have asked "Why?"

Anyway, I was glad to let a co-worker vent today, and I got a nice walk to The Mart out of it. That's a win-win to me. But it's always the people... no matter how bad the structure is, good people will make it work. But even the best structure will be dysfunctional when populated with bad people.

Command-and-Control Costs

Wednesday, April 29th, 2015

Management.jpg

Today I'm trying to appease management after being told that my promise to the CEO wasn't being covered by the CEO, and whatever management wants, management gets. And so I've finished the work I know needs to be done... given copies of the database to others for their use, and then emailed a few things about the production set-up. In all, it's been a good day. But now I'm stuck. Nothing to do.

So it's struck me that this compartmentalization of people and concerns is nice when you're dealing with insurgent cells, but it's kinda silly when you're in a business environment, and there are people that would help - if they only knew what they could do. This compartmentalization is a real killer of productivity - and morale, but let's stick with the real measurables - productivity.

Having one person control everything is a Bus Problem at the top - a bus hits this one guy and no one knows what was supposed to be done, or how things worked together, or anything. Cut of the head and the snake dies. I'm pretty sure that's something to be avoided.

Having one person control all the communication is inefficient as it places all communication in a serial queue that has but one actor: the processor. If he is busy, in a meeting, out for the day, on vacation, then nothing can be done because no one has the ability to process his messages. That's another form of the ...head of the snake problem, but it will occur no matter where in the organization this person is. Doesn't have to be the head.

All scheduling has to go through this one person, and that means that it's incredibly hard for them to effectively react to quick changes in individual's productivity. They dish things out - as they see fit - not asking anything - not sharing any plans - and so it's hard to adjust the plans when you aren't putting several minds towards the task.

The funny thing is that I don't believe they want to be treated like this - they just feel that it's their right to treat us this way. So sad... such a waste...

The Cost of Keeping a Promise

Friday, April 24th, 2015

Path

Yesterday was a very bad day for me. Yeah, to be fair, it's just one more in a long line of very bad days, but they tend to have a cumulative effect... after more than 5 or 6, it gets really hard to have the reserves that you had before the first. I'm not going to get into what it was, or anything like that - it's not the important point right now. What is important is that this morning I realized that it's just the cost of keeping a promise.

And yeah... it's just that simple.

A few weeks ago, I promised the CEO, in response to his worry about hitting a delivery date, that I would guarantee completion - but I had to be able to do it my way. It would work, and it would be stable, and flexible, but I couldn't be forced into another language, or another platform, etc. He agreed, and from that point on, even sub-consciously, I was working on that promise. When a new manager appeared, I assumed he was here to assist me. He believed he was here to implement his vision.

He didn't like that I wasn't willing to do just exactly as he said. I can even see his point of view - if it weren't for this promise I made. I should have told him right then and there of the promise, but I didn't want to make it more complicated than it already was. He wasn't interested in me liking his decision, he just wanted me to do it. That attitude just seemed wrong for the fact this was his second day.

But back to the point. This promise I made put me in the cross-hairs for his anger. It made me look to some like I wasn't being responsive - or even responsible. I was even painted as insubordinate. And what I realized this morning was that whether or not this was true, it was the cost of keeping the promise.

When I talk to the CEO and this new guy next, I'm going to ask the CEO if he wants to release me from this promise. If he does, then I'll do just what the new guy wants. Because then it's not my promise that's on the line. And it doesn't matter that the new guys says it's now his problem - the promise was from me to the CEO -- the new guy doesn't enter into the picture at all.

A great scene from Rob Roy:

Son: Father, will the MacGregors ever be kings again?
Robert Roy MacGregor: All men with honor are kings. But not all kings have honor.
Son: What is honor?
Robert Roy MacGregor: Honor is...
[Mary looking on]
Robert Roy MacGregor: what no man can give ya. And none can take away. Honor is a man's gift to himself.
Son: Do women have it?
Robert Roy MacGregor: Women have the heart of honor. And we cherish and protect it in them. You must never mistreat a woman, or a lame man. Or stand by a see another do so.
Son: How do you know if you have it?
Robert Roy MacGregor: Never worry on the getting of it. It grows in you, and speaks to you. All you need do is listen.

That's it. A promise is a promise - no matter what. Even when it's costly.

Quite the Unusual Exchange

Tuesday, April 21st, 2015

PHB.gif

Today I had another in a series of very unusual exchanges with a new guy at The Shop that was brought in to assist me in getting a few things finished up in time for an upcoming release. He was brought in - as I understood it - to be the heat shield for me to enable me to get a lot more done with someone cutting through the red-tape and such. It's not like this is Rocket Science, but it'd really help to have someone running interference for me, and then I'd be able to stay focused and writing code. But that's not how this first week has been turning out.

The guy is nice. I don't believe he's a mean or nasty person at all. I think he's got skills, talents, and drive - all good things. I think for a lot of people, this guy would be great. But for me... for this project, it's not really working out. Yet.

This project is supposed to be about getting something done. There are plans for longer-term solutions in the works, but they are going to take time to build and get up to production standards. If they were running now, there'd be no point in the work I'm doing - we'd get data from external sources, and simply throw it onto Kafka, and the Big Data Team would take care of it. It's a good solution.

If it were built.

But it's not.

So in order to meet timelines, I believe we agreed to have something more short-term that would hold off the need to rush the implementation and do a poor job. Everything I've heard reinforces that - save all the conversations with the new guy. To him, this is an opportunity to build a replacement for the Big Data - and do it at scale.

To me it's a classic Don't use a hand grenade to kill a mosquito - but to him, that's just a matter of opinion. And in truth, I suppose it is. Depends on the size of that mosquito.

He wants the problem broken down into different phases - services, and then different database tables, and all that would certainly work, but it's far too much and too complicated for what we need, and we have the time factor against us. The more complex we try to make this, the greater the risk we won't finish on time.

But that's not the odd part about this exchange today... No, today was about the order of building things. He wanted me to build all the acquirers and parsers and then worry about what to do with the data, where I have done enough of them to know that I don't need to do more, but the architecture I've picked might be wrong if I run into something I don't expect in the subsequent processing of the data. So I wanted to finish one or two of the sources all the way to completion, and then go back and finish the rest.

He disagreed. He said that he understood, but it's what He wanted that counted, and he wanted all the 'first stage' work done.

I tried to talk to him about my concerns of the risks to having to go back and re-work the code - as I'd already had to do for persisting the data to the filesystem, but he wasn't swayed. In the least.

He kept at it - asking me "Are you going to do it?" - seemingly waiting for me to say "No." I told him I didn't want to lie to him - to tell him I'm doing one thing when I'm not. And that I didn't feel at all comfortable with his plan - for this part of the project. He wasn't swayed.

In the end, I had to say "I can't."

I have no idea what the upshot of this is going to be. I worry it won't be good. But maybe it was a test? Maybe he wanted to see how I'd respond to a stressful situation? Maybe a lot of things. I just don't know.

I told my HR rep, as she has been up to date on this from the beginning. I don't expect anything from HR, but it's important to understand that when I was being insubordinate, there was a reason, and it was for the good of the company. I don't know if it'll make any difference. We'll have to wait and see.

Keep Going… Just Keep Going… and Try to Do Good

Monday, April 20th, 2015

Path

These last few weeks have been up and down, and I feel certain that it's going to get a lot worse before it gets any better. My back set me back - a herniated disc - nothing you can help, I do all the stretching and exercises, it's just life. But I was down for a few days last week. I'm on the mend, thanks to this not being the first time it's happened, and in a few weeks I am sure I'll be better. But it's a blow...

I also had a run-in with family this past weekend and while I love to see them, I know that they stiller unhappy with me - kids of divorce are something I'm very familiar with. It is what happens... sides are chosen... blame is assigned... it is what it is, but it also hurts. Again, in a few weeks, this will all be a memory, and things will seem to be back to "normal".

It's really just giving things time to become The Past. I've heard that over and over again in the last few years, and every time I hear it I'm hurt that it's the only thing anyone seems to say, but then 6 months later, I realize that it's the only thing they could say. It's an honest assessment of what happens when people get hurt.

Bones need time to heal. Discs need time to stop swelling. And the heart needs time to let things go. There's no pill to heal a bone... and while the prescription to reduce the swelling on my disc is really useful, I know that given time, it would go down on it's own. It'd just take a lot longer. And when people hurt each other, no words are going to shorten the time it takes to get past it.

I just need to keep going... get up in the morning. Try to do some good in the day. Have a good, healthful, lunch. Walk like you mean it. Get involved in your work, in your life. These don't really make anything better - they just make it a lot easier to pass the time. That's what really needs to happen. But these make the time seem to pass faster.

I just have to keep at it.

Clojure Makes a Great Reporting Framework

Wednesday, April 8th, 2015

Clojure.jpg

This morning I had to really smile... and even do my Happy Dance in my chair. I used something I found in the clojure library adworj to make reporting on Facebook ad server data as nice and clean as the code I lifted from adworj. I really liked adworj, but he wanted to handle credentials one way, and I had to have them in a separate store, so there were enough differences that the only really good things were the metadata on the reports from AdWords.

So what I decided to start with was a slight variant on the report metadata that he used. I started with a similar metadata structure, but instead of the macros and defrecord, I chose a simpler map of the data:

  (def reportstats-fields
    "The complete list of all allowed fields in the Facebook ReportStats
    reporting system keyed on the nice clojure name, and including the
    Facebook name and an optional parser for the returned data."
    {:time-start               {:name "time_start" :parse parse-long}
     :time-stop                {:name "time_stop" :parse parse-long}
     :date-start               {:name "date_start" :parse to-date-time}
     :date-stop                {:name "date_stop" :parse to-date-time}
     :account-currency         "account_currency"
     :account-id               {:name "account_id" :parse parse-long}
     :account-name             "account_name"
     :adgroup-id               {:name "adgroup_id" :parse parse-long}
     :adgroup-name             "adgroup_name"})

where I've clearly truncated the list of fields from Facebook, but you get the idea. There is either a string that's the Facebook field name, or there's a map with two keys: :name for the Facebook name of the field, and :parse for the function to mars the value into it's clojure data element.

My real divergence starts with how the reverse-mapping is done. I start by having a defined set of all the field names, as opposed to computing it on the fly over and over again:

  (def all-fields
    "Set of all valid keyword/field names for the Facebook ReportStats
    reports."
    (set (keys reportstats-fields)))

and then another that defines a map for how to get a Facebook field into closure-land:

  (def coercions
    "Map of the Facebook field names to the `:parse` functions for those
    fields - if they exist in the report definition. If they don't, then
    don't map anything and they won't then be handled in the
    reading/parsing."
    (into {} (for [[n md] reportstats-fields
                   :let [nf (if (string? md) md (:name md))
                         cf (if (string? md) identity (:parse md))]]
               [(keyword nf) (fn [v] [n (cf v)])])))

What I like about this approach is that we don't have to deal with the overhead of doing this for each report each time it's run. While that's probably not a big overhead, why? Why spend any time on this once it's done? The report structures are fixed in both cases, and there's just no reason for that level of flexibility.

What the coercions map gives me is a collection of functions to map the values coming back from Facebook - keyed by their Facebook field name, and suitable for inclusion into a map - with the correct clojure keyword for the field name. For instance:

  (defn coerce-record
    "Function to take a record from Facebook, and apply the known mappings
    to get it **back** into decent clojure names and datatypes."
    [m]
    (into {} (for [[k v] m] ((get coercions k) v))))

This function simply creates the right map (record) from the Facebook map (record) in a very simple, easy way. If there's a change in the formatting, just change the structure and it'll automatically be fixed in subsequent calls. That's nice.

Need to have a new field? Add it. Or change the name... it's all so easy to report like this.

Best of all, the real reporting code is just a vector of keywords to define what to get, and then a simple all. Very nice indeed!