LinkedIn API for a Recruiting Tool

May 7th, 2015

LinkedIn

I was asked today to look at the LinkedIn API to see if I could access the data at LinkedIn to make an advanced recruiting tool for the Recruiters here at The Shop. The idea was to take a resume we received, match it to a LinkedIn profile (I'd venture that 90%+ of them are there), and then use advanced analytics to rate the prospective resumes for potential success at this job.

It's an interesting idea. The real advantage for LinkedIn is that companies like ours pays several thousand dollars a month to have access. With this kind of tool, that same data could be used to classify candidates by the data they have entered, and then use a nice predictive model to say who is most likely to succeed. It's simple feedback.

We take everyone that's been successful at this company. Reference their LinkedIn profiles for the training data, and then use any and all reviews to say which of these people are likely to be the successful ones - based on all the classified data that's on LinkedIn.

It's kinda neat. We don't have to wonder what factor(s) matter most - we can get all that LinkedIn has in their API, and then use the success factor - say a 1 - 5 rating as the outcome, and then train away. After that, every submitted resume can run through the trained net, and come up with a score and a confidence number. Pretty simple.

It's not meant to be fool-proof, but when you have a ton of openings, it's nice to be able to have something that whittles down the list of thousands to hundreds, or less - so that you can really focus on these people.

We'll see where it goes - if it goes anywhere.

Dug a Little on Interactive Brokers

May 7th, 2015

WallSt.jpg

A friend of mine has asked me to look into the Interactive Brokers offerings as they have an API for trading and he's interested in moving off the Windows platform he's on now (.NET) and have me help him scale up his trading strategy quite a bit. So I started looking at what they have.

First off, they are cross-platform, as well as web-based. That's nice. Their API access has virtually everything you need to get market data as well as execute and monitor trades, as well as do all the portfolio management and reporting. Very nicely done. It's also all on GitHub, and they are willing to look at pull-requests on the API. This is really nice for a lot of reasons, but the most significant to me today is that they don't see this as a complete control situation. This tells me a lot.

Honestly, I was getting kinda jazzed about it because I remember all this stuff from Finance. I can see making a process that runs alongside the Trader Workbench - their flagship product and register for ticks and trades and then calculate everything it needs and submit trades as needed. These could all be viewed in the TWS screen as the account would have to be the same. It's like having an automated trader working for you and all you have to do is watch it work.

I read a book on their Java API, and dug into the API Docs as well as reading on the relationship between the IB API and TWS. It's not a bad system. Given that this is not a high-frequency strategy, there's no need for co-location, and the associated costs. It's something that hopefully can run on a MacBook Pro - or if not, then a Mac Pro, and a nice, fast cable modem.

I looked it up, and Comcast can go to 105 Mbps in my area, and I'm at 50 Mbps already. The package for 105 Mbps is pretty decent and about a wash with what I'm paying now for 50 Mbps. I'd have to call and verify and make sure the cost delta is what the web site says, but if so, then it's very doable.

It might be really nice to be back in finance. Start-up again. Holy Cow! What an idea. 🙂

On a Roll with jQuery and Bootstrap

May 6th, 2015

JQuery Framework

OK, since I was on a roll with the work I was doing at The Shop, it made sense to keep going and apply it to putting a web front-end on the CryptoQuip solver a friend and I have been working on. The solver is all Clojure, and so it took just a few tweaks to get all the static files in-place and ready, and then getting the POST working was really the challenge.

Turns out, the way to get it to work was:

  function solveThePuzzle() {
    console.log("attempting to solve the puzzle");
    // get what we need for the call...
    var inp = {};
    inp.cyphertext = $("#plaintext").val();
    inp.clue = {};
    inp.clue[$("#key1").val()] = $("#key2").val();
    // make the call to solve the puzzle
    $.ajax({type: "POST",
            url: "/solve",
            processData: false,
            contentType: 'application/json',
            data: JSON.stringify(inp),
            success: function(resp) {
              var cont = '<div class="alert alert-success" role="alert">';
              cont += '<strong>Solved:</strong> ' + resp.plaintext;
              cont += '</div>';
              $("#status").replaceWith(cont);
            }
    })
  }

and the results are very nice:

better colors

Building with jQuery and Bootstrap

May 6th, 2015

JQuery Framework

This morning I decided that I wanted to be able to start doing decent web site design. This means getting a lot better at jQuery - even though it's old, it's still useful and it's out there, and learning Bootstrap. Now the latter was really the influence of the folks at Groupon, because if they used it, then it had to be a good tool. Sure, they improved a lot on it, and they were really fond of their whitespace, but I want to get to the point that I can make a decent web site with decent tools so that it's not like I'm just HTML/CSS and nothing else.

So this morning I did this:

better colors

it's not amazing, but for about 3 hrs, it's not bad, and it's backed by the data, and I've got a good start on the way to make it all work in a Clojure app. So that's what I'm going to write up, so I don't forget all the issues.

Get The Tools

First off, go get the jQuery and Bootstrap downloads. I'm not going to be fiddling with the code, so I don't need to build it, just get it and use it. One important thing I noted was that Bootstrap 3.3.4 was very particular about the version of jQuery that needed to be used. Specifically, 1.11.2.

Start in the resources/ directory of the Clojure project and make a public directory where all the static assets will be served from. Anything in resources/ will be packaged up into the 'uberjar', so once there, it's save to develop and deploy on these files. When you are done, make sure the filesystem looks like this:

  resources
      +- public
          +- index.html
          +- css
          |   +- bootstrap.min.css
          |   +- bootstrap-theme.min.css
          |   +- theme.css
          +- js
              +- bootstrap.min.js
              +- jquery-1.11.2.min.js

where the contents of the theme.css is very simple and yet needs to be there to make Bootstrap look decent:

  body: {
    padding-top: 70px;
    padding-bottom: 30px;
  }
 
  .theme-dropdown .dropdown-menu {
    position: static;
    display: block;
    margin-bottom: 20px;
  }
 
  .theme-showcase > p > .btn {
    margin: 5px 0;
  }
 
  .theme-showcase .navbar.container {
    width: auto;
  }

In the index.html there will be a few places where you need to match the names of these -showcase - and possibly make the names unique to your project.

Setting Up the Home Page

Now that you have the tools copied into the right places, we need to throw together the boilerplate for the home page. This is pretty easy, and Bootstrap examples all have it as clear as day for you:

  <!DOCTYPE html>
  <html lang="en">
    <head>
      <meta charset="utf-8">
      <meta http-equiv="X-UA-Compatible" content="IE=edge">
      <meta name="viewport" content="width=device-width, initial-scale=1">
      <!-- The above 3 meta tags *must* come first in the head;
           any other head content must come *after* these tags -->
      <title>Bartender</title>
      <!-- Bootstrap core CSS -->
      <link href="/css/bootstrap.min.css" rel="stylesheet">
      <!-- Bootstrap theme -->
      <link href="/css/bootstrap-theme.min.css" rel="stylesheet">
      <link href="/css/theme.css" rel="stylesheet">
    </head>
 
    <body role="document">
 
 
 
      <!-- Bootstrap core JavaScript
      ================================================== -->
      <!-- Placed at the end of the document so the pages load faster -->
      <script src="/js/jquery-1.11.2.min.js"></script>
      <script src="/js/bootstrap.min.js"></script>
      <script src="/js/bartender.js"></script>
      <script type="text/javascript">
        // when we are all done loading the page, hit the data...
        $( document ).ready(reloadAllData());
        // $( window ).load(reloadAllData());
      </script>
    </body>
  </html>

In the middle there, you can put in all the things you want. For me it was a simple menu and a table in a container. All this is on the Bootstrap examples and it's easy to pick-n-choose.

Wiring it in with Ring

The other big thing was to have it all work from the Clojure web project. There are a few things that needed to be done here, but in the end, they really aren't all that hard, and the results are quite nice.

First, make sure you have the necessary ring components in the project.clj:

  [ring/ring-core "1.3.2"]
  [ring/ring-jetty-adapter "1.3.2"]
  [ring.middleware.jsonp "0.1.6"]

and then make sure to require:

  [compojure.route :as route]
  [ring.util.response :as resp]

so that you can add the route:

  (GET "/" []
    (resp/redirect "/index.html"))

and then at the end of all the routes, add:

  (route/resources "/")
  (route/not-found "<h1>Page not Found!</h1>")

where the resources function specifies where to find the static resources, like the index.html.

At this point, restart the web server, and things should just work. Not very interesting, but the log will show the static resources being requested, and that verifies that they are found, and loaded, and you can see this all from the javascript console as well.

Adding the Database Access

At this point, we need to add the useful stuff. To keep things separate, I created a separate bartender.js to hold all the javascript I was going to write, and for now, that is simply:

  /*
   * Function to add a single row to the 'last-import-times' table on the
   * main page. This will be called for each key/value pair returned from
   * the RESTful API on the server, and it's just a simple way to make it
   * easy to see what's loaded up in the service.
   */
  function addLastImportRow(src, vals) {
    var dates = vals.report_date
    var tr = '<tr>';
    tr += '<td>' + src + '</td>';
    tr += '<td>' + vals.generated_at + '</td>';
    tr += '<td>' + dates[dates.length - 1].substring(0,10) + '</td>';
    tr += '<td>' + dates[0].substring(0,10) + '</td>';
    tr += '</tr>';
    $("#last-import-times").append(tr);
  }
 
  /*
   * Function to reload the data in the 'last-import-times' table on the
   * main page. This will clear out what's there, and hit the endpoint that
   * will bring back all the data it needs to properly make the page.
   */
  function refreshLastImportTable() {
    console.log("refreshing the last-import data");
    // remove all the non-header rows in the table
    $("#last-import-times tr").remove();
    // make the call to get the data from Bartender
    $.getJSON("/v1/latest-imports", function(data) {
      $.each(data, function(k, v) {
        addLastImportRow(k, v);
      })
    })
  }
 
  /*
   * Simple function to be called when the page is reloaded and we need to
   * update ALL the data on the page.
   */
  function reloadAllData() {
    refreshLastImportTable();
  }

There are some nice things here that I learned. First, jQuery may be old, but it's awfully nice for simple things like I'm building. Sure, if you want to make a single-page app on the web, it might be weak, but for me, it's very nice. And stable.

It's easy to ask for JSON from the server, and then have a function that simply takes each key/value pair and puts it in the emptied table. In the HTML I put an id on the table to make it easy to find, and that's all that I needed. I cleared it out, then one by one, I added the new rows back in. Works like a charm.

With this, I should be able to start making decent looking web sites. Much more to learn.

Moved to Adium 1.5.11b2

May 6th, 2015

Adium.jpg

The nightly builds of Adium 1.5.11 are nice, but they are getting a little unstable with the new OAuth2 GTalk credentials - like having to re-do them when awaking from sleep. So I found that the latest Adium 1.5.11 beta is pretty recent (2015-04-16), and it works with all the services I need, and I've kinda given up on MSN coming back to life... so why go through the pain of the bleeding-edge?

None that I can think of.

So it's back to the straight beta for now. Nothing against the nightlies, and I may go back if they get MSN and GTalk stable, but it's just not worth it today.

The State of Javascript Charting Frameworks

May 5th, 2015

CorePlot.jpg

Since I don't have a lot to do right now, I wanted to see what the current state of the Javascript charting libraries was. I know that D3 is out there, and it's one of the tried-and-true open source tools out there, but it's not necessarily meant for real-time applications, and given that I think in real-time applications, I tend to think of it as the static graphs toolset that allows you to fiddle with every little bit. In fact, you have to build everything yourself. It's not bad... it's just not high-level, and it's not necessarily meant for real-time.

I found Highcharts on a search, and it looks very nice. Now it's commercial, but you get what you pay for, and the very best charting package I've ever used is the Vantage Point Java library. It was, and is, just amazing. But it's Java, and while that's OK, it's still very costly, and you can't embed it in a web page these days because of the Java applet situation.

Highcharts has several packages - simple charting, stock (financial) charting, mapping, etc. It's not a bad system, and the demos they have include the code to generate them, and I have to say, it looks pretty nice. Again, you get what you pay for.

Smoothie Charts looks very nice for real-time strip-chart plotting. The code is very simple, and it's got all the auto-scaling that you can imagine, but it's a strip-chart and not a full-blown charting system. But it is fast.

Fusioncharts is one of the older ones, and it's got a lot of nice functionality, but it's commercial, and a bit on the steep side unless you're really wanting to spend a lot on this.

There were others, but they didn't stack up well against these. Open Source is nice because it's a chance to give back as well as fix problems. Paid is nice in that you have support and someone else fixing the issues - maybe. Lots of trade-offs, and I have no doubt that more are on the way. This is just something that's seen really amazing growth, and I can't believe it's going to stop any time soon.

Microsoft Azure is Getting Very Close

May 5th, 2015

GoogleAppEngine.jpg

This morning I was reading about some recent additions to Microsoft Azure and so I decided to give it a look-see about what it would be like deploying some things I've been noodling on to Azure. What I found was really pretty nice. Almost.

Azure is getting built out as a series of services that you can add on - not unlike Google's App Engine, and Heroku - but in the case of the first two, they are all owned and operated by the parent companies, where as Heroku allows other companies to offer services to customers. What's different with Azure and Google is that they are about hosting more than a web service or app - Azure now has Batch Processing, Hadoop, Hive, Storm, and a host of other services that can all be brought to bear at the problem you are trying to solve.

This is very nice because Azure also has all the nice monitoring tools like NewRelic, as they own it all, so it's easy to see what's happening, and what you might need to do next. Very slick. They even have Machine Learning as an add-on with code you can write in R and Python. Not my favorite languages, but I get that these are the most popular for the platform, so I'm willing to go with it for now.

In all, I was very impressed with Azure - until I got to the technical details. Then I kinda felt a little cheated. Azure is (understandably) written to use Windows Servers as the back-end first. I can believe they might have linux in the plans, but it's kinda tough to work with Windows Servers underneath if you're writing batch jobs. After all, you need to know what you're writing on - and in the case of clojure, that's the JVM, and while the JVM runs on Windows Server, it's not anything I can reproduce easily.

That's not to say Azure is out of the running. Google App Engine can do many of the same things, but doesn't have the breath and depth of the monitoring - or the additional services like Hadoop and Storm. Sure, you can get machines from Google and put Hadoop and Storm on them, but that's forcing you to be in the role of administrator, and Azure is trying to take you out of that role - on purpose.

Google is giving you a little more choice on the lower levels of the stack, but isn't supporting you as well from a platform point of view. If I were going to make a relatively complex system, and didn't want to mess with the administration - because the performance wasn't that big a deal, and the cost of managed services wasn't too much, then I might really pick Azure.

But that's a lot of "ifs". Most of the times I'm going to know exactly what I need, and for those cases, Heroku is far more likely to be the choice.

Rainy Walking into Work

May 5th, 2015

Umbrella

This morning it was one of those days where I knew it was going to be a rainy walk from the train station to work. It was raining before I left the house this morning, and so I dressed accordingly, and carried my umbrella.

Generally, I don't mind rain. If rain were my biggest problem in life - I'd be leading a charmed existence. So I knew it wasn't a matter of if I got a little wet, it was really the wetness factor I was contending with. It wasn't a particularly warm rain, so I was glad I had my raincoat that's a pretty warm one - considering, and it kept the wicking to a minimum.

I got to work and realized that my umbrella had been a casualty of the rain - one of the rivets connecting an arm to the spreader had popped, and so I needed to see about fixing that. I gave it a good once-over, and thought paper clip. I walked over to the office supplies and picked up two - one a little bit bigger than the other.

I then sized it up, bent it back and forth several times to cause a fatigue crack, and then went about the fixing. I was able to put the "pin" through the spreader, and then the arm, and then used the handle of my scissors to bend it back on itself to keep from falling out. I was pretty impressed with the fix! It looks like it's going to hold for quite a while.

Still... I'm going to have to get a new umbrella soon. These large golf umbrellas are just too handy to not have one (or two) right there by the front door.

It’s Always the People

May 4th, 2015

Management.jpg

I took a walk with a friend today to get lunch. I had already eaten by they hadn't, and needed to get out of the office and vent a little. The problem they were having was a classic Little Red Hen story: someone does a lot of work - in this case months of work, and then when it starts to look interesting and executives are hyped about it, there's all kinds of folks wanting to take credit for the one person's work.

I've run into this quite a bit in the last decade, and I've actually come to bargain this with a manager. I let them take credit for the work, and they allow me to work in a way that I find most productive. In the end, they can take all the credit they want because there's no way anyone will really think they did it - they were the Manager - not the Doer, but they love the attention, and the sign on for more. Eventually, they are in a difficult position because they can't hope to have the same productivity from other people, and so the relationship becomes more symbiotic. Which is good.

But this is my friend's first experience with this, and I can remember being just as upset as they are, when it first happened to me. So on the walk to pick up a sandwich we talked about these problems, and what the real solution was. They were convinced that it was the structure of an organization that breeds this problem, but my assertion was that it's the people in it.

It's always the people.

Their point was that there was no need for the traditional manager in a successful organization. While I'm not saying that managers are required, I accept that most people who run organizations feel more comfortable when there's a Chain of Responsibility - if not a Chain of Command, so that blame can be assigned when things go wrong.

The only time I saw this not being used was in small start-up companies. There, it's assumed that everyone shares in the responsibility, and so there's no need or even interest in assigning blame. But when an organization gets big enough, and that size depends on the group of people, then it's either time to stop growing, or start compartmentalizing the organization, and that leads to the silos and the need to be able to attribute responsibility and it's evil alter ego - blame.

It's hard to forego growth when times are good. It's something I've learned first-hand at the company I started. You want to do well, and you are happy when things are going well. People are happy, work is good, and you think "Why not?" But that's what leads to this problem because the question is the wrong one to ask. Better would be "Why?"

It's not easy to do, I know. But if we had asked "Why?" more than "Why not?" - I might still be at that company I started. But I'm not, and it was a direct result of the decisions we made. In the beginning we had to achieve consensus... after several years, it wasn't required, and so we let it slide. That was a mistake. I wish he'd have asked "Why?"

Anyway, I was glad to let a co-worker vent today, and I got a nice walk to The Mart out of it. That's a win-win to me. But it's always the people... no matter how bad the structure is, good people will make it work. But even the best structure will be dysfunctional when populated with bad people.

Being Thankful for the Little Things

May 4th, 2015

Path

This weekend, Pastor Ross spoke about Philippians and how sometimes we need to be thankful and grateful for what we know will come - before it comes. And I know I've had many times in this chapter of my life where I've been unable to find the happiness in my life - but I can honestly say that I can't remember even once when I felt God had forsaken me.

Teaching me a lesson? Sure. The Bible is filled with stories of people that messed up and had to deal with the consequences even when they were profoundly sorry. So yeah... there have been a lot of times like that for me. I'd search my memory for things I did wrong - face it, it's pretty easy for all of us to remember when we did something we shouldn't have - and feel that was the source of the pain.

But I never felt that I was alone. Sad, in pain, isolated, punished - sure... all that and more, but not forsaken. Never once.

So I'm trying to be thankful for what might seem like the little things in my life. In reality, they aren't the little things they are the Timeless Things. Those things that are all around each of us that have the ability to make us smile... to make us stop and take pause for what they are.

This weekend it was the light coming through my bedroom window in the morning. It was just amazing. Simple, clean, just exactly how I wanted my bedroom to look, and with this amazing sunbeam coming through the white curtain I had hung my first weekend in the house, and spilling onto the bed. It made me smile.

Then later, I was talking to myself as I walked around the house, and when I walked into my bedroom, the echo changed, and it because what I can only really describe as "safe" and "protecting". At least that's how I interpreted the sound. I knew this house wanted to protect me... wanted to give me a place to feel safe and secure. Buying this house had been difficult. I had to keep fighting for it with the Bank. But as I look back now, it was one of the best decisions I've made in my life.

So there are a few things I'm very thankful for. The sunlight, and the house. I don't think of this as my house - it's more than 100 years old. I'm just the current caretaker. When I'm gone, this house will still be here, and it'll be someone else's. It's just letting me stay here for a while. I appreciate it very much.

Life is what we make of it. It's hard to remember when you're getting beaten up, but it's true. Some times it's a lot easier to have that view than others.