Archive for the ‘Coding’ Category

Wild Bug in ZooKeeper

Friday, May 8th, 2015

ZooKeeper

I read on twitter about a bug in ZooKeeper found by the folks at PagerDuty. The story is quite remarkable, and reminds me that some companies still invest the time to get to the bottom of things - as opposed to just putting it off once a work-around is found. The level of detail and investigation they did is simply... inspiring. I'm stunned.

I'd like to work at a place where that kind of stuff is done. Not all the time, of course, as I'm sure they were all glad that it was over when it was over, but to be able to devote the time to solving the problem as opposed to stopping it - that's nice.

I don't imagine anything I'd do would hit this series of bugs. Too many components that simply aren't something I'd ever want to use. But it's nice to know someone is there digging deep.

Some days things work out…

Friday, May 8th, 2015

Great News

This morning I saw on HipChat messages from two folks at The Shop:

Chris M. said that you helped him to setup our new hardware. Guess what? I just ran a test ETL in new hardware, it is 5 times faster. The full MMS ETL cycle takes about 2-2.5 hours. In new server, it takes 0.5 hour. THANK YOU for whatever you helped Chris M. πŸ™‚
-- Okji

and from Chris:

So, you were 100% right on the hardware specs for the Pentaho stuff.

Okji is running initial stuff now and it's insanely fast.

thank you for dealing with a stubborn asshat me through the ordeal and lighting a fire under my rear.

tbh I'd probably be flogging the dead virtualization horse at this point w/o that back and forth we had.

So yeah, thanks πŸ™‚
-- Chris

I don't often have people sending me these kinds of notes for work I did for them. The problem was simple and obvious - to me, but if you have never seen the other way, you often think your way is the only way. I've seen it a million times. The point is to get them started, let them see, and then be very gracious when they thank you.

That last part is key.

You want to build up everyone - not just yourself. Help others feel good about what they did, and they will want to work with - or for - you again. It's simple. Who wants to be around someone that makes them feel bad about themselves? No one I know.

Interestingly enough, this is going to make things work a lot better for the short-term goals. There's a consultant at the shop, and this is going to make his Uber Plan much less attractive, and necessary.

Postgres has Added UPSERT

Friday, May 8th, 2015

PostgreSQL.jpg

One of the things I've always wanted in Postgres is the UPSERT - an INSERT that would update certain fields if the row (defined by the primary key) already existed. In the past, I've had to implement this as a custom function (stored procedure) in pl/sql, by checking for the existence of the row and then doing an UPDATE, or failing that, do an INSERT. It's workable, and it's not horrible, but it's also something that's in several other databases, and I wanted it in my database. πŸ™‚

This morning I read a tweet that said:

better colors

and read the commit log message that described it. I love it! This is exactly what I've been hoping for.

The only question now is - When is it released?

LinkedIn API for a Recruiting Tool

Thursday, May 7th, 2015

LinkedIn

I was asked today to look at the LinkedIn API to see if I could access the data at LinkedIn to make an advanced recruiting tool for the Recruiters here at The Shop. The idea was to take a resume we received, match it to a LinkedIn profile (I'd venture that 90%+ of them are there), and then use advanced analytics to rate the prospective resumes for potential success at this job.

It's an interesting idea. The real advantage for LinkedIn is that companies like ours pays several thousand dollars a month to have access. With this kind of tool, that same data could be used to classify candidates by the data they have entered, and then use a nice predictive model to say who is most likely to succeed. It's simple feedback.

We take everyone that's been successful at this company. Reference their LinkedIn profiles for the training data, and then use any and all reviews to say which of these people are likely to be the successful ones - based on all the classified data that's on LinkedIn.

It's kinda neat. We don't have to wonder what factor(s) matter most - we can get all that LinkedIn has in their API, and then use the success factor - say a 1 - 5 rating as the outcome, and then train away. After that, every submitted resume can run through the trained net, and come up with a score and a confidence number. Pretty simple.

It's not meant to be fool-proof, but when you have a ton of openings, it's nice to be able to have something that whittles down the list of thousands to hundreds, or less - so that you can really focus on these people.

We'll see where it goes - if it goes anywhere.

Dug a Little on Interactive Brokers

Thursday, May 7th, 2015

WallSt.jpg

A friend of mine has asked me to look into the Interactive Brokers offerings as they have an API for trading and he's interested in moving off the Windows platform he's on now (.NET) and have me help him scale up his trading strategy quite a bit. So I started looking at what they have.

First off, they are cross-platform, as well as web-based. That's nice. Their API access has virtually everything you need to get market data as well as execute and monitor trades, as well as do all the portfolio management and reporting. Very nicely done. It's also all on GitHub, and they are willing to look at pull-requests on the API. This is really nice for a lot of reasons, but the most significant to me today is that they don't see this as a complete control situation. This tells me a lot.

Honestly, I was getting kinda jazzed about it because I remember all this stuff from Finance. I can see making a process that runs alongside the Trader Workbench - their flagship product and register for ticks and trades and then calculate everything it needs and submit trades as needed. These could all be viewed in the TWS screen as the account would have to be the same. It's like having an automated trader working for you and all you have to do is watch it work.

I read a book on their Java API, and dug into the API Docs as well as reading on the relationship between the IB API and TWS. It's not a bad system. Given that this is not a high-frequency strategy, there's no need for co-location, and the associated costs. It's something that hopefully can run on a MacBook Pro - or if not, then a Mac Pro, and a nice, fast cable modem.

I looked it up, and Comcast can go to 105 Mbps in my area, and I'm at 50 Mbps already. The package for 105 Mbps is pretty decent and about a wash with what I'm paying now for 50 Mbps. I'd have to call and verify and make sure the cost delta is what the web site says, but if so, then it's very doable.

It might be really nice to be back in finance. Start-up again. Holy Cow! What an idea. πŸ™‚

Building with jQuery and Bootstrap

Wednesday, May 6th, 2015

JQuery Framework

This morning I decided that I wanted to be able to start doing decent web site design. This means getting a lot better at jQuery - even though it's old, it's still useful and it's out there, and learning Bootstrap. Now the latter was really the influence of the folks at Groupon, because if they used it, then it had to be a good tool. Sure, they improved a lot on it, and they were really fond of their whitespace, but I want to get to the point that I can make a decent web site with decent tools so that it's not like I'm just HTML/CSS and nothing else.

So this morning I did this:

better colors

it's not amazing, but for about 3 hrs, it's not bad, and it's backed by the data, and I've got a good start on the way to make it all work in a Clojure app. So that's what I'm going to write up, so I don't forget all the issues.

Get The Tools

First off, go get the jQuery and Bootstrap downloads. I'm not going to be fiddling with the code, so I don't need to build it, just get it and use it. One important thing I noted was that Bootstrap 3.3.4 was very particular about the version of jQuery that needed to be used. Specifically, 1.11.2.

Start in the resources/ directory of the Clojure project and make a public directory where all the static assets will be served from. Anything in resources/ will be packaged up into the 'uberjar', so once there, it's save to develop and deploy on these files. When you are done, make sure the filesystem looks like this:

  resources
      +- public
          +- index.html
          +- css
          |   +- bootstrap.min.css
          |   +- bootstrap-theme.min.css
          |   +- theme.css
          +- js
              +- bootstrap.min.js
              +- jquery-1.11.2.min.js

where the contents of the theme.css is very simple and yet needs to be there to make Bootstrap look decent:

  body: {
    padding-top: 70px;
    padding-bottom: 30px;
  }
 
  .theme-dropdown .dropdown-menu {
    position: static;
    display: block;
    margin-bottom: 20px;
  }
 
  .theme-showcase > p > .btn {
    margin: 5px 0;
  }
 
  .theme-showcase .navbar.container {
    width: auto;
  }

In the index.html there will be a few places where you need to match the names of these -showcase - and possibly make the names unique to your project.

Setting Up the Home Page

Now that you have the tools copied into the right places, we need to throw together the boilerplate for the home page. This is pretty easy, and Bootstrap examples all have it as clear as day for you:

  <!DOCTYPE html>
  <html lang="en">
    <head>
      <meta charset="utf-8">
      <meta http-equiv="X-UA-Compatible" content="IE=edge">
      <meta name="viewport" content="width=device-width, initial-scale=1">
      <!-- The above 3 meta tags *must* come first in the head;
           any other head content must come *after* these tags -->
      <title>Bartender</title>
      <!-- Bootstrap core CSS -->
      <link href="/css/bootstrap.min.css" rel="stylesheet">
      <!-- Bootstrap theme -->
      <link href="/css/bootstrap-theme.min.css" rel="stylesheet">
      <link href="/css/theme.css" rel="stylesheet">
    </head>
 
    <body role="document">
 
 
 
      <!-- Bootstrap core JavaScript
      ================================================== -->
      <!-- Placed at the end of the document so the pages load faster -->
      <script src="/js/jquery-1.11.2.min.js"></script>
      <script src="/js/bootstrap.min.js"></script>
      <script src="/js/bartender.js"></script>
      <script type="text/javascript">
        // when we are all done loading the page, hit the data...
        $( document ).ready(reloadAllData());
        // $( window ).load(reloadAllData());
      </script>
    </body>
  </html>

In the middle there, you can put in all the things you want. For me it was a simple menu and a table in a container. All this is on the Bootstrap examples and it's easy to pick-n-choose.

Wiring it in with Ring

The other big thing was to have it all work from the Clojure web project. There are a few things that needed to be done here, but in the end, they really aren't all that hard, and the results are quite nice.

First, make sure you have the necessary ring components in the project.clj:

  [ring/ring-core "1.3.2"]
  [ring/ring-jetty-adapter "1.3.2"]
  [ring.middleware.jsonp "0.1.6"]

and then make sure to require:

  [compojure.route :as route]
  [ring.util.response :as resp]

so that you can add the route:

  (GET "/" []
    (resp/redirect "/index.html"))

and then at the end of all the routes, add:

  (route/resources "/")
  (route/not-found "<h1>Page not Found!</h1>")

where the resources function specifies where to find the static resources, like the index.html.

At this point, restart the web server, and things should just work. Not very interesting, but the log will show the static resources being requested, and that verifies that they are found, and loaded, and you can see this all from the javascript console as well.

Adding the Database Access

At this point, we need to add the useful stuff. To keep things separate, I created a separate bartender.js to hold all the javascript I was going to write, and for now, that is simply:

  /*
   * Function to add a single row to the 'last-import-times' table on the
   * main page. This will be called for each key/value pair returned from
   * the RESTful API on the server, and it's just a simple way to make it
   * easy to see what's loaded up in the service.
   */
  function addLastImportRow(src, vals) {
    var dates = vals.report_date
    var tr = '<tr>';
    tr += '<td>' + src + '</td>';
    tr += '<td>' + vals.generated_at + '</td>';
    tr += '<td>' + dates[dates.length - 1].substring(0,10) + '</td>';
    tr += '<td>' + dates[0].substring(0,10) + '</td>';
    tr += '</tr>';
    $("#last-import-times").append(tr);
  }
 
  /*
   * Function to reload the data in the 'last-import-times' table on the
   * main page. This will clear out what's there, and hit the endpoint that
   * will bring back all the data it needs to properly make the page.
   */
  function refreshLastImportTable() {
    console.log("refreshing the last-import data");
    // remove all the non-header rows in the table
    $("#last-import-times tr").remove();
    // make the call to get the data from Bartender
    $.getJSON("/v1/latest-imports", function(data) {
      $.each(data, function(k, v) {
        addLastImportRow(k, v);
      })
    })
  }
 
  /*
   * Simple function to be called when the page is reloaded and we need to
   * update ALL the data on the page.
   */
  function reloadAllData() {
    refreshLastImportTable();
  }

There are some nice things here that I learned. First, jQuery may be old, but it's awfully nice for simple things like I'm building. Sure, if you want to make a single-page app on the web, it might be weak, but for me, it's very nice. And stable.

It's easy to ask for JSON from the server, and then have a function that simply takes each key/value pair and puts it in the emptied table. In the HTML I put an id on the table to make it easy to find, and that's all that I needed. I cleared it out, then one by one, I added the new rows back in. Works like a charm.

With this, I should be able to start making decent looking web sites. Much more to learn.

The State of Javascript Charting Frameworks

Tuesday, May 5th, 2015

CorePlot.jpg

Since I don't have a lot to do right now, I wanted to see what the current state of the Javascript charting libraries was. I know that D3 is out there, and it's one of the tried-and-true open source tools out there, but it's not necessarily meant for real-time applications, and given that I think in real-time applications, I tend to think of it as the static graphs toolset that allows you to fiddle with every little bit. In fact, you have to build everything yourself. It's not bad... it's just not high-level, and it's not necessarily meant for real-time.

I found Highcharts on a search, and it looks very nice. Now it's commercial, but you get what you pay for, and the very best charting package I've ever used is the Vantage Point Java library. It was, and is, just amazing. But it's Java, and while that's OK, it's still very costly, and you can't embed it in a web page these days because of the Java applet situation.

Highcharts has several packages - simple charting, stock (financial) charting, mapping, etc. It's not a bad system, and the demos they have include the code to generate them, and I have to say, it looks pretty nice. Again, you get what you pay for.

Smoothie Charts looks very nice for real-time strip-chart plotting. The code is very simple, and it's got all the auto-scaling that you can imagine, but it's a strip-chart and not a full-blown charting system. But it is fast.

Fusioncharts is one of the older ones, and it's got a lot of nice functionality, but it's commercial, and a bit on the steep side unless you're really wanting to spend a lot on this.

There were others, but they didn't stack up well against these. Open Source is nice because it's a chance to give back as well as fix problems. Paid is nice in that you have support and someone else fixing the issues - maybe. Lots of trade-offs, and I have no doubt that more are on the way. This is just something that's seen really amazing growth, and I can't believe it's going to stop any time soon.

Microsoft Azure is Getting Very Close

Tuesday, May 5th, 2015

GoogleAppEngine.jpg

This morning I was reading about some recent additions to Microsoft Azure and so I decided to give it a look-see about what it would be like deploying some things I've been noodling on to Azure. What I found was really pretty nice. Almost.

Azure is getting built out as a series of services that you can add on - not unlike Google's App Engine, and Heroku - but in the case of the first two, they are all owned and operated by the parent companies, where as Heroku allows other companies to offer services to customers. What's different with Azure and Google is that they are about hosting more than a web service or app - Azure now has Batch Processing, Hadoop, Hive, Storm, and a host of other services that can all be brought to bear at the problem you are trying to solve.

This is very nice because Azure also has all the nice monitoring tools like NewRelic, as they own it all, so it's easy to see what's happening, and what you might need to do next. Very slick. They even have Machine Learning as an add-on with code you can write in R and Python. Not my favorite languages, but I get that these are the most popular for the platform, so I'm willing to go with it for now.

In all, I was very impressed with Azure - until I got to the technical details. Then I kinda felt a little cheated. Azure is (understandably) written to use Windows Servers as the back-end first. I can believe they might have linux in the plans, but it's kinda tough to work with Windows Servers underneath if you're writing batch jobs. After all, you need to know what you're writing on - and in the case of clojure, that's the JVM, and while the JVM runs on Windows Server, it's not anything I can reproduce easily.

That's not to say Azure is out of the running. Google App Engine can do many of the same things, but doesn't have the breath and depth of the monitoring - or the additional services like Hadoop and Storm. Sure, you can get machines from Google and put Hadoop and Storm on them, but that's forcing you to be in the role of administrator, and Azure is trying to take you out of that role - on purpose.

Google is giving you a little more choice on the lower levels of the stack, but isn't supporting you as well from a platform point of view. If I were going to make a relatively complex system, and didn't want to mess with the administration - because the performance wasn't that big a deal, and the cost of managed services wasn't too much, then I might really pick Azure.

But that's a lot of "ifs". Most of the times I'm going to know exactly what I need, and for those cases, Heroku is far more likely to be the choice.

Atom 0.194.0 is Getting a Lot Better

Thursday, April 30th, 2015

Atom Editor

With all the brew-ha-ha about GitHub's Electron and Atom editors, I wanted to give Atom another look. What I found was that it's amazingly close to an editor that I could use on a daily basis - moving Sublime Text out of the way.

They have solved the "re-opening with old files" issue - that was huge. They have gotten to the point that it even matches the command keys of Sublime Text. It's close.

There are a few things that are annoying... the command palette on the default Atom Light UI is too wide for my 80-column windows, and I can't for the life of me figure out what the CSS is to make it more narrow - like 90% of the display width. Fix that, and I might think about it.

OK... this is cool... get into the developer mode:

  $ atom --dev

and then you can get the standard WebKit Inspector for all the components. Then it's just a matter of making the element visible, inspecting it, seeing what's the dimension settings, and then fixing it.

In Atom -> Open Your Stylesheet, you can then add this little snippet:

  atom-panel.modal, .overlay {
    width: 90%;
    max-width: 500px;
    margin-left: 0;
    transform: translate(-50%, 0);
  }

and then the pop-up modal windows will be 90% of the width, and centered. Now it'd be nice to have this be limited to 500px, but I'll work on that next. For now, it's pretty close to being usable. I'll have to give it a go.

UPDATE: by adding the max-width and the transform tags, as well as setting the default margin-left, I can get it to be 90% and centered, or up to 500px and centered. Very nice!

connect bar

Interesting Tweet about Ruby Development

Thursday, April 16th, 2015

This morning an old co-worker of mine retweeted this very interesting tweet about Ruby development:

It's true, and I think it's one of the reasons I liked moving from C++ to Ruby - the OO was familiar, and the tools and Convention over Configuration made it nice and easy to get things done - and fast.

But then the reality set in. It was always going to be like this. That's when I realized I don't like big cities. Clojure is the way I think with writing code now, and it's the cleanest way I know to write with simplicity.

Acknowledge truth when you hear it. Amen.