Archive for May, 2015

Life Is What We Make of It

Friday, May 8th, 2015

Path

This morning I don't have anything to do at The Shop - I'm waiting on people to decide what to do, and ordinarily that's a stressful time for me because some of the people blocking me aren't the most helpful of people I've run into. In fact, they just plain make me nervous. But this morning, I'm listening to Hootie and the Blowfish and I just know that this life of mine is in my hands - no one else's.

Not Liza's. Not the kids. No one - but me.

I can choose to feel beaten up, or not. I can choose to feel the victim, or not.

This isn't the first time I've felt this way, but it's been an exceptionally long time since I've felt this way. Just a silly smile on my face realizing that: No weapon formed against me shall prosper. It really is that simple - and it's really just that hard.

It's been so easy to feel beaten up - face it, there are people emotionally, and financially, beating up on me, so it's pretty easy to think that I am who they see me to be. They want to beat up on someone - they want a victim. So it's easy to see myself as they see me, and I then become the victim.

But the alternative is to realize that I have a code of ethics, a morality, that says I will honor my obligations, and then there's the divorce law, but that doesn't cover emotions. That doesn't cover honor. That isn't the sum total of who I am.

No, that's much more. So I can take the slings and arrows, because I choose how to handle them. They can be deadly - or they can be nothing more than the emotions ejected from others, and while I can see them, I don't have to internalize them.

Good start on the day!

Taking a Walk for The Good Stuff

Friday, May 8th, 2015

Diet Coke

There are very few things I'll walk two floors for - but a nice Diet Coke is certainly one of them. There is just nothing like a good Diet Coke for me. Coffee doesn't do it, Sprite, Lemonade - forget about it. So when I found this morning that the fountain on this floor was serving up soda water where once Diet Coke flowed freely... well... I had to grab some glasses and head downstairs. To the other fountains in the building.

Now I have a reasonable supply and I'm sure the kind gentlemen (or ladies) whose job it is to detect such outages will be right on the case and correcting this injustice as soon as possible.

But until then... Ah...

Loaded up on Diet Coke

Some days things work out…

Friday, May 8th, 2015

Great News

This morning I saw on HipChat messages from two folks at The Shop:

Chris M. said that you helped him to setup our new hardware. Guess what? I just ran a test ETL in new hardware, it is 5 times faster. The full MMS ETL cycle takes about 2-2.5 hours. In new server, it takes 0.5 hour. THANK YOU for whatever you helped Chris M. 🙂
-- Okji

and from Chris:

So, you were 100% right on the hardware specs for the Pentaho stuff.

Okji is running initial stuff now and it's insanely fast.

thank you for dealing with a stubborn asshat me through the ordeal and lighting a fire under my rear.

tbh I'd probably be flogging the dead virtualization horse at this point w/o that back and forth we had.

So yeah, thanks 🙂
-- Chris

I don't often have people sending me these kinds of notes for work I did for them. The problem was simple and obvious - to me, but if you have never seen the other way, you often think your way is the only way. I've seen it a million times. The point is to get them started, let them see, and then be very gracious when they thank you.

That last part is key.

You want to build up everyone - not just yourself. Help others feel good about what they did, and they will want to work with - or for - you again. It's simple. Who wants to be around someone that makes them feel bad about themselves? No one I know.

Interestingly enough, this is going to make things work a lot better for the short-term goals. There's a consultant at the shop, and this is going to make his Uber Plan much less attractive, and necessary.

Postgres has Added UPSERT

Friday, May 8th, 2015

PostgreSQL.jpg

One of the things I've always wanted in Postgres is the UPSERT - an INSERT that would update certain fields if the row (defined by the primary key) already existed. In the past, I've had to implement this as a custom function (stored procedure) in pl/sql, by checking for the existence of the row and then doing an UPDATE, or failing that, do an INSERT. It's workable, and it's not horrible, but it's also something that's in several other databases, and I wanted it in my database. 🙂

This morning I read a tweet that said:

better colors

and read the commit log message that described it. I love it! This is exactly what I've been hoping for.

The only question now is - When is it released?

LinkedIn API for a Recruiting Tool

Thursday, May 7th, 2015

LinkedIn

I was asked today to look at the LinkedIn API to see if I could access the data at LinkedIn to make an advanced recruiting tool for the Recruiters here at The Shop. The idea was to take a resume we received, match it to a LinkedIn profile (I'd venture that 90%+ of them are there), and then use advanced analytics to rate the prospective resumes for potential success at this job.

It's an interesting idea. The real advantage for LinkedIn is that companies like ours pays several thousand dollars a month to have access. With this kind of tool, that same data could be used to classify candidates by the data they have entered, and then use a nice predictive model to say who is most likely to succeed. It's simple feedback.

We take everyone that's been successful at this company. Reference their LinkedIn profiles for the training data, and then use any and all reviews to say which of these people are likely to be the successful ones - based on all the classified data that's on LinkedIn.

It's kinda neat. We don't have to wonder what factor(s) matter most - we can get all that LinkedIn has in their API, and then use the success factor - say a 1 - 5 rating as the outcome, and then train away. After that, every submitted resume can run through the trained net, and come up with a score and a confidence number. Pretty simple.

It's not meant to be fool-proof, but when you have a ton of openings, it's nice to be able to have something that whittles down the list of thousands to hundreds, or less - so that you can really focus on these people.

We'll see where it goes - if it goes anywhere.

Dug a Little on Interactive Brokers

Thursday, May 7th, 2015

WallSt.jpg

A friend of mine has asked me to look into the Interactive Brokers offerings as they have an API for trading and he's interested in moving off the Windows platform he's on now (.NET) and have me help him scale up his trading strategy quite a bit. So I started looking at what they have.

First off, they are cross-platform, as well as web-based. That's nice. Their API access has virtually everything you need to get market data as well as execute and monitor trades, as well as do all the portfolio management and reporting. Very nicely done. It's also all on GitHub, and they are willing to look at pull-requests on the API. This is really nice for a lot of reasons, but the most significant to me today is that they don't see this as a complete control situation. This tells me a lot.

Honestly, I was getting kinda jazzed about it because I remember all this stuff from Finance. I can see making a process that runs alongside the Trader Workbench - their flagship product and register for ticks and trades and then calculate everything it needs and submit trades as needed. These could all be viewed in the TWS screen as the account would have to be the same. It's like having an automated trader working for you and all you have to do is watch it work.

I read a book on their Java API, and dug into the API Docs as well as reading on the relationship between the IB API and TWS. It's not a bad system. Given that this is not a high-frequency strategy, there's no need for co-location, and the associated costs. It's something that hopefully can run on a MacBook Pro - or if not, then a Mac Pro, and a nice, fast cable modem.

I looked it up, and Comcast can go to 105 Mbps in my area, and I'm at 50 Mbps already. The package for 105 Mbps is pretty decent and about a wash with what I'm paying now for 50 Mbps. I'd have to call and verify and make sure the cost delta is what the web site says, but if so, then it's very doable.

It might be really nice to be back in finance. Start-up again. Holy Cow! What an idea. 🙂

On a Roll with jQuery and Bootstrap

Wednesday, May 6th, 2015

JQuery Framework

OK, since I was on a roll with the work I was doing at The Shop, it made sense to keep going and apply it to putting a web front-end on the CryptoQuip solver a friend and I have been working on. The solver is all Clojure, and so it took just a few tweaks to get all the static files in-place and ready, and then getting the POST working was really the challenge.

Turns out, the way to get it to work was:

  function solveThePuzzle() {
    console.log("attempting to solve the puzzle");
    // get what we need for the call...
    var inp = {};
    inp.cyphertext = $("#plaintext").val();
    inp.clue = {};
    inp.clue[$("#key1").val()] = $("#key2").val();
    // make the call to solve the puzzle
    $.ajax({type: "POST",
            url: "/solve",
            processData: false,
            contentType: 'application/json',
            data: JSON.stringify(inp),
            success: function(resp) {
              var cont = '<div class="alert alert-success" role="alert">';
              cont += '<strong>Solved:</strong> ' + resp.plaintext;
              cont += '</div>';
              $("#status").replaceWith(cont);
            }
    })
  }

and the results are very nice:

better colors

Building with jQuery and Bootstrap

Wednesday, May 6th, 2015

JQuery Framework

This morning I decided that I wanted to be able to start doing decent web site design. This means getting a lot better at jQuery - even though it's old, it's still useful and it's out there, and learning Bootstrap. Now the latter was really the influence of the folks at Groupon, because if they used it, then it had to be a good tool. Sure, they improved a lot on it, and they were really fond of their whitespace, but I want to get to the point that I can make a decent web site with decent tools so that it's not like I'm just HTML/CSS and nothing else.

So this morning I did this:

better colors

it's not amazing, but for about 3 hrs, it's not bad, and it's backed by the data, and I've got a good start on the way to make it all work in a Clojure app. So that's what I'm going to write up, so I don't forget all the issues.

Get The Tools

First off, go get the jQuery and Bootstrap downloads. I'm not going to be fiddling with the code, so I don't need to build it, just get it and use it. One important thing I noted was that Bootstrap 3.3.4 was very particular about the version of jQuery that needed to be used. Specifically, 1.11.2.

Start in the resources/ directory of the Clojure project and make a public directory where all the static assets will be served from. Anything in resources/ will be packaged up into the 'uberjar', so once there, it's save to develop and deploy on these files. When you are done, make sure the filesystem looks like this:

  resources
      +- public
          +- index.html
          +- css
          |   +- bootstrap.min.css
          |   +- bootstrap-theme.min.css
          |   +- theme.css
          +- js
              +- bootstrap.min.js
              +- jquery-1.11.2.min.js

where the contents of the theme.css is very simple and yet needs to be there to make Bootstrap look decent:

  body: {
    padding-top: 70px;
    padding-bottom: 30px;
  }
 
  .theme-dropdown .dropdown-menu {
    position: static;
    display: block;
    margin-bottom: 20px;
  }
 
  .theme-showcase > p > .btn {
    margin: 5px 0;
  }
 
  .theme-showcase .navbar.container {
    width: auto;
  }

In the index.html there will be a few places where you need to match the names of these -showcase - and possibly make the names unique to your project.

Setting Up the Home Page

Now that you have the tools copied into the right places, we need to throw together the boilerplate for the home page. This is pretty easy, and Bootstrap examples all have it as clear as day for you:

  <!DOCTYPE html>
  <html lang="en">
    <head>
      <meta charset="utf-8">
      <meta http-equiv="X-UA-Compatible" content="IE=edge">
      <meta name="viewport" content="width=device-width, initial-scale=1">
      <!-- The above 3 meta tags *must* come first in the head;
           any other head content must come *after* these tags -->
      <title>Bartender</title>
      <!-- Bootstrap core CSS -->
      <link href="/css/bootstrap.min.css" rel="stylesheet">
      <!-- Bootstrap theme -->
      <link href="/css/bootstrap-theme.min.css" rel="stylesheet">
      <link href="/css/theme.css" rel="stylesheet">
    </head>
 
    <body role="document">
 
 
 
      <!-- Bootstrap core JavaScript
      ================================================== -->
      <!-- Placed at the end of the document so the pages load faster -->
      <script src="/js/jquery-1.11.2.min.js"></script>
      <script src="/js/bootstrap.min.js"></script>
      <script src="/js/bartender.js"></script>
      <script type="text/javascript">
        // when we are all done loading the page, hit the data...
        $( document ).ready(reloadAllData());
        // $( window ).load(reloadAllData());
      </script>
    </body>
  </html>

In the middle there, you can put in all the things you want. For me it was a simple menu and a table in a container. All this is on the Bootstrap examples and it's easy to pick-n-choose.

Wiring it in with Ring

The other big thing was to have it all work from the Clojure web project. There are a few things that needed to be done here, but in the end, they really aren't all that hard, and the results are quite nice.

First, make sure you have the necessary ring components in the project.clj:

  [ring/ring-core "1.3.2"]
  [ring/ring-jetty-adapter "1.3.2"]
  [ring.middleware.jsonp "0.1.6"]

and then make sure to require:

  [compojure.route :as route]
  [ring.util.response :as resp]

so that you can add the route:

  (GET "/" []
    (resp/redirect "/index.html"))

and then at the end of all the routes, add:

  (route/resources "/")
  (route/not-found "<h1>Page not Found!</h1>")

where the resources function specifies where to find the static resources, like the index.html.

At this point, restart the web server, and things should just work. Not very interesting, but the log will show the static resources being requested, and that verifies that they are found, and loaded, and you can see this all from the javascript console as well.

Adding the Database Access

At this point, we need to add the useful stuff. To keep things separate, I created a separate bartender.js to hold all the javascript I was going to write, and for now, that is simply:

  /*
   * Function to add a single row to the 'last-import-times' table on the
   * main page. This will be called for each key/value pair returned from
   * the RESTful API on the server, and it's just a simple way to make it
   * easy to see what's loaded up in the service.
   */
  function addLastImportRow(src, vals) {
    var dates = vals.report_date
    var tr = '<tr>';
    tr += '<td>' + src + '</td>';
    tr += '<td>' + vals.generated_at + '</td>';
    tr += '<td>' + dates[dates.length - 1].substring(0,10) + '</td>';
    tr += '<td>' + dates[0].substring(0,10) + '</td>';
    tr += '</tr>';
    $("#last-import-times").append(tr);
  }
 
  /*
   * Function to reload the data in the 'last-import-times' table on the
   * main page. This will clear out what's there, and hit the endpoint that
   * will bring back all the data it needs to properly make the page.
   */
  function refreshLastImportTable() {
    console.log("refreshing the last-import data");
    // remove all the non-header rows in the table
    $("#last-import-times tr").remove();
    // make the call to get the data from Bartender
    $.getJSON("/v1/latest-imports", function(data) {
      $.each(data, function(k, v) {
        addLastImportRow(k, v);
      })
    })
  }
 
  /*
   * Simple function to be called when the page is reloaded and we need to
   * update ALL the data on the page.
   */
  function reloadAllData() {
    refreshLastImportTable();
  }

There are some nice things here that I learned. First, jQuery may be old, but it's awfully nice for simple things like I'm building. Sure, if you want to make a single-page app on the web, it might be weak, but for me, it's very nice. And stable.

It's easy to ask for JSON from the server, and then have a function that simply takes each key/value pair and puts it in the emptied table. In the HTML I put an id on the table to make it easy to find, and that's all that I needed. I cleared it out, then one by one, I added the new rows back in. Works like a charm.

With this, I should be able to start making decent looking web sites. Much more to learn.

Moved to Adium 1.5.11b2

Wednesday, May 6th, 2015

Adium.jpg

The nightly builds of Adium 1.5.11 are nice, but they are getting a little unstable with the new OAuth2 GTalk credentials - like having to re-do them when awaking from sleep. So I found that the latest Adium 1.5.11 beta is pretty recent (2015-04-16), and it works with all the services I need, and I've kinda given up on MSN coming back to life... so why go through the pain of the bleeding-edge?

None that I can think of.

So it's back to the straight beta for now. Nothing against the nightlies, and I may go back if they get MSN and GTalk stable, but it's just not worth it today.

The State of Javascript Charting Frameworks

Tuesday, May 5th, 2015

CorePlot.jpg

Since I don't have a lot to do right now, I wanted to see what the current state of the Javascript charting libraries was. I know that D3 is out there, and it's one of the tried-and-true open source tools out there, but it's not necessarily meant for real-time applications, and given that I think in real-time applications, I tend to think of it as the static graphs toolset that allows you to fiddle with every little bit. In fact, you have to build everything yourself. It's not bad... it's just not high-level, and it's not necessarily meant for real-time.

I found Highcharts on a search, and it looks very nice. Now it's commercial, but you get what you pay for, and the very best charting package I've ever used is the Vantage Point Java library. It was, and is, just amazing. But it's Java, and while that's OK, it's still very costly, and you can't embed it in a web page these days because of the Java applet situation.

Highcharts has several packages - simple charting, stock (financial) charting, mapping, etc. It's not a bad system, and the demos they have include the code to generate them, and I have to say, it looks pretty nice. Again, you get what you pay for.

Smoothie Charts looks very nice for real-time strip-chart plotting. The code is very simple, and it's got all the auto-scaling that you can imagine, but it's a strip-chart and not a full-blown charting system. But it is fast.

Fusioncharts is one of the older ones, and it's got a lot of nice functionality, but it's commercial, and a bit on the steep side unless you're really wanting to spend a lot on this.

There were others, but they didn't stack up well against these. Open Source is nice because it's a chance to give back as well as fix problems. Paid is nice in that you have support and someone else fixing the issues - maybe. Lots of trade-offs, and I have no doubt that more are on the way. This is just something that's seen really amazing growth, and I can't believe it's going to stop any time soon.