Archive for the ‘Coding’ Category

Dug a Little on Interactive Brokers

Thursday, May 7th, 2015

WallSt.jpg

A friend of mine has asked me to look into the Interactive Brokers offerings as they have an API for trading and he's interested in moving off the Windows platform he's on now (.NET) and have me help him scale up his trading strategy quite a bit. So I started looking at what they have.

First off, they are cross-platform, as well as web-based. That's nice. Their API access has virtually everything you need to get market data as well as execute and monitor trades, as well as do all the portfolio management and reporting. Very nicely done. It's also all on GitHub, and they are willing to look at pull-requests on the API. This is really nice for a lot of reasons, but the most significant to me today is that they don't see this as a complete control situation. This tells me a lot.

Honestly, I was getting kinda jazzed about it because I remember all this stuff from Finance. I can see making a process that runs alongside the Trader Workbench - their flagship product and register for ticks and trades and then calculate everything it needs and submit trades as needed. These could all be viewed in the TWS screen as the account would have to be the same. It's like having an automated trader working for you and all you have to do is watch it work.

I read a book on their Java API, and dug into the API Docs as well as reading on the relationship between the IB API and TWS. It's not a bad system. Given that this is not a high-frequency strategy, there's no need for co-location, and the associated costs. It's something that hopefully can run on a MacBook Pro - or if not, then a Mac Pro, and a nice, fast cable modem.

I looked it up, and Comcast can go to 105 Mbps in my area, and I'm at 50 Mbps already. The package for 105 Mbps is pretty decent and about a wash with what I'm paying now for 50 Mbps. I'd have to call and verify and make sure the cost delta is what the web site says, but if so, then it's very doable.

It might be really nice to be back in finance. Start-up again. Holy Cow! What an idea. 🙂

Building with jQuery and Bootstrap

Wednesday, May 6th, 2015

JQuery Framework

This morning I decided that I wanted to be able to start doing decent web site design. This means getting a lot better at jQuery - even though it's old, it's still useful and it's out there, and learning Bootstrap. Now the latter was really the influence of the folks at Groupon, because if they used it, then it had to be a good tool. Sure, they improved a lot on it, and they were really fond of their whitespace, but I want to get to the point that I can make a decent web site with decent tools so that it's not like I'm just HTML/CSS and nothing else.

So this morning I did this:

better colors

it's not amazing, but for about 3 hrs, it's not bad, and it's backed by the data, and I've got a good start on the way to make it all work in a Clojure app. So that's what I'm going to write up, so I don't forget all the issues.

Get The Tools

First off, go get the jQuery and Bootstrap downloads. I'm not going to be fiddling with the code, so I don't need to build it, just get it and use it. One important thing I noted was that Bootstrap 3.3.4 was very particular about the version of jQuery that needed to be used. Specifically, 1.11.2.

Start in the resources/ directory of the Clojure project and make a public directory where all the static assets will be served from. Anything in resources/ will be packaged up into the 'uberjar', so once there, it's save to develop and deploy on these files. When you are done, make sure the filesystem looks like this:

  resources
      +- public
          +- index.html
          +- css
          |   +- bootstrap.min.css
          |   +- bootstrap-theme.min.css
          |   +- theme.css
          +- js
              +- bootstrap.min.js
              +- jquery-1.11.2.min.js

where the contents of the theme.css is very simple and yet needs to be there to make Bootstrap look decent:

  body: {
    padding-top: 70px;
    padding-bottom: 30px;
  }
 
  .theme-dropdown .dropdown-menu {
    position: static;
    display: block;
    margin-bottom: 20px;
  }
 
  .theme-showcase > p > .btn {
    margin: 5px 0;
  }
 
  .theme-showcase .navbar.container {
    width: auto;
  }

In the index.html there will be a few places where you need to match the names of these -showcase - and possibly make the names unique to your project.

Setting Up the Home Page

Now that you have the tools copied into the right places, we need to throw together the boilerplate for the home page. This is pretty easy, and Bootstrap examples all have it as clear as day for you:

  <!DOCTYPE html>
  <html lang="en">
    <head>
      <meta charset="utf-8">
      <meta http-equiv="X-UA-Compatible" content="IE=edge">
      <meta name="viewport" content="width=device-width, initial-scale=1">
      <!-- The above 3 meta tags *must* come first in the head;
           any other head content must come *after* these tags -->
      <title>Bartender</title>
      <!-- Bootstrap core CSS -->
      <link href="/css/bootstrap.min.css" rel="stylesheet">
      <!-- Bootstrap theme -->
      <link href="/css/bootstrap-theme.min.css" rel="stylesheet">
      <link href="/css/theme.css" rel="stylesheet">
    </head>
 
    <body role="document">
 
 
 
      <!-- Bootstrap core JavaScript
      ================================================== -->
      <!-- Placed at the end of the document so the pages load faster -->
      <script src="/js/jquery-1.11.2.min.js"></script>
      <script src="/js/bootstrap.min.js"></script>
      <script src="/js/bartender.js"></script>
      <script type="text/javascript">
        // when we are all done loading the page, hit the data...
        $( document ).ready(reloadAllData());
        // $( window ).load(reloadAllData());
      </script>
    </body>
  </html>

In the middle there, you can put in all the things you want. For me it was a simple menu and a table in a container. All this is on the Bootstrap examples and it's easy to pick-n-choose.

Wiring it in with Ring

The other big thing was to have it all work from the Clojure web project. There are a few things that needed to be done here, but in the end, they really aren't all that hard, and the results are quite nice.

First, make sure you have the necessary ring components in the project.clj:

  [ring/ring-core "1.3.2"]
  [ring/ring-jetty-adapter "1.3.2"]
  [ring.middleware.jsonp "0.1.6"]

and then make sure to require:

  [compojure.route :as route]
  [ring.util.response :as resp]

so that you can add the route:

  (GET "/" []
    (resp/redirect "/index.html"))

and then at the end of all the routes, add:

  (route/resources "/")
  (route/not-found "<h1>Page not Found!</h1>")

where the resources function specifies where to find the static resources, like the index.html.

At this point, restart the web server, and things should just work. Not very interesting, but the log will show the static resources being requested, and that verifies that they are found, and loaded, and you can see this all from the javascript console as well.

Adding the Database Access

At this point, we need to add the useful stuff. To keep things separate, I created a separate bartender.js to hold all the javascript I was going to write, and for now, that is simply:

  /*
   * Function to add a single row to the 'last-import-times' table on the
   * main page. This will be called for each key/value pair returned from
   * the RESTful API on the server, and it's just a simple way to make it
   * easy to see what's loaded up in the service.
   */
  function addLastImportRow(src, vals) {
    var dates = vals.report_date
    var tr = '<tr>';
    tr += '<td>' + src + '</td>';
    tr += '<td>' + vals.generated_at + '</td>';
    tr += '<td>' + dates[dates.length - 1].substring(0,10) + '</td>';
    tr += '<td>' + dates[0].substring(0,10) + '</td>';
    tr += '</tr>';
    $("#last-import-times").append(tr);
  }
 
  /*
   * Function to reload the data in the 'last-import-times' table on the
   * main page. This will clear out what's there, and hit the endpoint that
   * will bring back all the data it needs to properly make the page.
   */
  function refreshLastImportTable() {
    console.log("refreshing the last-import data");
    // remove all the non-header rows in the table
    $("#last-import-times tr").remove();
    // make the call to get the data from Bartender
    $.getJSON("/v1/latest-imports", function(data) {
      $.each(data, function(k, v) {
        addLastImportRow(k, v);
      })
    })
  }
 
  /*
   * Simple function to be called when the page is reloaded and we need to
   * update ALL the data on the page.
   */
  function reloadAllData() {
    refreshLastImportTable();
  }

There are some nice things here that I learned. First, jQuery may be old, but it's awfully nice for simple things like I'm building. Sure, if you want to make a single-page app on the web, it might be weak, but for me, it's very nice. And stable.

It's easy to ask for JSON from the server, and then have a function that simply takes each key/value pair and puts it in the emptied table. In the HTML I put an id on the table to make it easy to find, and that's all that I needed. I cleared it out, then one by one, I added the new rows back in. Works like a charm.

With this, I should be able to start making decent looking web sites. Much more to learn.

The State of Javascript Charting Frameworks

Tuesday, May 5th, 2015

CorePlot.jpg

Since I don't have a lot to do right now, I wanted to see what the current state of the Javascript charting libraries was. I know that D3 is out there, and it's one of the tried-and-true open source tools out there, but it's not necessarily meant for real-time applications, and given that I think in real-time applications, I tend to think of it as the static graphs toolset that allows you to fiddle with every little bit. In fact, you have to build everything yourself. It's not bad... it's just not high-level, and it's not necessarily meant for real-time.

I found Highcharts on a search, and it looks very nice. Now it's commercial, but you get what you pay for, and the very best charting package I've ever used is the Vantage Point Java library. It was, and is, just amazing. But it's Java, and while that's OK, it's still very costly, and you can't embed it in a web page these days because of the Java applet situation.

Highcharts has several packages - simple charting, stock (financial) charting, mapping, etc. It's not a bad system, and the demos they have include the code to generate them, and I have to say, it looks pretty nice. Again, you get what you pay for.

Smoothie Charts looks very nice for real-time strip-chart plotting. The code is very simple, and it's got all the auto-scaling that you can imagine, but it's a strip-chart and not a full-blown charting system. But it is fast.

Fusioncharts is one of the older ones, and it's got a lot of nice functionality, but it's commercial, and a bit on the steep side unless you're really wanting to spend a lot on this.

There were others, but they didn't stack up well against these. Open Source is nice because it's a chance to give back as well as fix problems. Paid is nice in that you have support and someone else fixing the issues - maybe. Lots of trade-offs, and I have no doubt that more are on the way. This is just something that's seen really amazing growth, and I can't believe it's going to stop any time soon.

Microsoft Azure is Getting Very Close

Tuesday, May 5th, 2015

GoogleAppEngine.jpg

This morning I was reading about some recent additions to Microsoft Azure and so I decided to give it a look-see about what it would be like deploying some things I've been noodling on to Azure. What I found was really pretty nice. Almost.

Azure is getting built out as a series of services that you can add on - not unlike Google's App Engine, and Heroku - but in the case of the first two, they are all owned and operated by the parent companies, where as Heroku allows other companies to offer services to customers. What's different with Azure and Google is that they are about hosting more than a web service or app - Azure now has Batch Processing, Hadoop, Hive, Storm, and a host of other services that can all be brought to bear at the problem you are trying to solve.

This is very nice because Azure also has all the nice monitoring tools like NewRelic, as they own it all, so it's easy to see what's happening, and what you might need to do next. Very slick. They even have Machine Learning as an add-on with code you can write in R and Python. Not my favorite languages, but I get that these are the most popular for the platform, so I'm willing to go with it for now.

In all, I was very impressed with Azure - until I got to the technical details. Then I kinda felt a little cheated. Azure is (understandably) written to use Windows Servers as the back-end first. I can believe they might have linux in the plans, but it's kinda tough to work with Windows Servers underneath if you're writing batch jobs. After all, you need to know what you're writing on - and in the case of clojure, that's the JVM, and while the JVM runs on Windows Server, it's not anything I can reproduce easily.

That's not to say Azure is out of the running. Google App Engine can do many of the same things, but doesn't have the breath and depth of the monitoring - or the additional services like Hadoop and Storm. Sure, you can get machines from Google and put Hadoop and Storm on them, but that's forcing you to be in the role of administrator, and Azure is trying to take you out of that role - on purpose.

Google is giving you a little more choice on the lower levels of the stack, but isn't supporting you as well from a platform point of view. If I were going to make a relatively complex system, and didn't want to mess with the administration - because the performance wasn't that big a deal, and the cost of managed services wasn't too much, then I might really pick Azure.

But that's a lot of "ifs". Most of the times I'm going to know exactly what I need, and for those cases, Heroku is far more likely to be the choice.

Atom 0.194.0 is Getting a Lot Better

Thursday, April 30th, 2015

Atom Editor

With all the brew-ha-ha about GitHub's Electron and Atom editors, I wanted to give Atom another look. What I found was that it's amazingly close to an editor that I could use on a daily basis - moving Sublime Text out of the way.

They have solved the "re-opening with old files" issue - that was huge. They have gotten to the point that it even matches the command keys of Sublime Text. It's close.

There are a few things that are annoying... the command palette on the default Atom Light UI is too wide for my 80-column windows, and I can't for the life of me figure out what the CSS is to make it more narrow - like 90% of the display width. Fix that, and I might think about it.

OK... this is cool... get into the developer mode:

  $ atom --dev

and then you can get the standard WebKit Inspector for all the components. Then it's just a matter of making the element visible, inspecting it, seeing what's the dimension settings, and then fixing it.

In Atom -> Open Your Stylesheet, you can then add this little snippet:

  atom-panel.modal, .overlay {
    width: 90%;
    max-width: 500px;
    margin-left: 0;
    transform: translate(-50%, 0);
  }

and then the pop-up modal windows will be 90% of the width, and centered. Now it'd be nice to have this be limited to 500px, but I'll work on that next. For now, it's pretty close to being usable. I'll have to give it a go.

UPDATE: by adding the max-width and the transform tags, as well as setting the default margin-left, I can get it to be 90% and centered, or up to 500px and centered. Very nice!

connect bar

Interesting Tweet about Ruby Development

Thursday, April 16th, 2015

This morning an old co-worker of mine retweeted this very interesting tweet about Ruby development:

It's true, and I think it's one of the reasons I liked moving from C++ to Ruby - the OO was familiar, and the tools and Convention over Configuration made it nice and easy to get things done - and fast.

But then the reality set in. It was always going to be like this. That's when I realized I don't like big cities. Clojure is the way I think with writing code now, and it's the cleanest way I know to write with simplicity.

Acknowledge truth when you hear it. Amen.

Homebrew Really is an Amazing Resource

Thursday, April 9th, 2015

Homebrew

I have been doing a bit of updating in the last 24 hours, which includes getting the latest Apple compilers and support tools - git 2.3.2, even (nice!)... but I've also been checking with Homebrew for the grc command, and updating it based on the new compilers, etc. from Apple. And it just hit me (again), that a tool like Homebrew makes installation of hundreds of Open Source tools so incredibly easy. It's just almost impossible to think that it's necessary to compile many programs from source on a Mac these days.

Lots of great software is in Homebrew - with sections for Science and all kinds of things. Just an amazing community resource. Amazingly all built on Ruby and git. What a way to build something great.

Colorized Tailed Log4j Files

Thursday, April 9th, 2015

log4j.jpg

I've been looking for something to do a decent job of colorizing the output of tail -f for the log files I watch. Nothing really amazing - just something that will highlight the different levels of the log4j file, and maybe something to pick out IP addresses, etc. Nothing fancy. But what it needs to be is simple.

I don't want to have a tool that takes over the tail command - I still want to be able to scroll up on the terminal and see what was there. I don't want something that's got all the search features, or sorting, or filtering... that's all nice, but I have all the other unix tools for those, and that's how I think. No, what I need is a simple, streaming, colorizer of the log4j stream, and then that's it.

After doing a considerable (for me, for this) digging, it looks like a little app called grc is just the thing I need. The source site is here, but there's a fork of the code on GitHub, and given that the savba.sk domain isn't responding this morning, it's nice to have the GitHub version available.

Home-brew has it as well, and so installing it is pretty simple:

  $ brew install grc

and then the info on grace in Homebrew says that it's simple to start colorizing your Bash scripts by simply adding a config file to your ~/.bashrc. But for me, the configuration is a little more complex.

After reading the README at the GitHub fork, it was clear that I needed to make two files, and put them in a new ~/.grc directory: grc.conf for the general configuration of the tool, and conf.log4j for the colorization rules for the log4j files. The contents of the grc.conf file is pretty simple, and just tells grc what config file to use for what source files:

  # log4j files
  \b\w+\b.*log\b
  conf.log4j

the first line is the regex for the name of the file, and the second line is config file to use. Speaking of which, the conf.log4j was created to look like:

  # From http://fahdshariff.blogspot.ca/2011/04/
  #           highlighting-command-output-with.html
  # this configuration file is suitable for displaying log files
  #errors
  regexp=^.*(ERROR|Error|Exception).*$
  colours=bold red
  ======
  #stack trace
  regexp=^\s+at [^:]*:\d*\)$
  colours=red
  ======
  regexp=^.*(FATAL|Fatal).*$
  colours=on_red
  ======
  regexp=^.*(WARNING|Warning).*$
  colours=bold yellow
  ======
  regexp=^.*(DEBUG|Debug).*$
  colours=bold black
  ======
  # this is a time
  regexp=[^\d]*\d\d:\d\d:\d\d[\.,]{0,1}\d{0,1}\d{0,1}\d{0,1}
  colours=bold blue
  count=once
  ======
  # ip number
  regexp=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}
  colours=bold magenta
  count=more

At this point, I should be able to say:

  $ grc tail -f log/cw.log

and grace would see the log in the filename and pick the right config file and then colorize the output of the tail command. To make this even easier, I added a simple alias in my ~/.bashrc:

  alias lt="grc tail -f"

where, to me, the lt stands for Log Tail, and that's so easy for me to remember.

Now if I can just get that site to show up, and download the code. I should be ready to go. Fingers crossed...

UPDATE: I was able to download grc, and sure enough it's working like a champ!

connect bar

Dropbox is an Amazing Tool

Wednesday, April 8th, 2015

Dropbox.jpg

I've been a Dropbox user for quite a while, and I've held off on the storing of Photos for now because of iCloud, and the fact that my iPhone and Mac already have those things handled quite nicely. But for all the other things I might need a cloud storage system for, I have to concede that Dropbox really is the best - by far.

When I need to move a file from one laptop to another - Dropbox is there. When I want to have it available on my phone - Dropbox is there. The seamlessness of Dropbox is really about as frictionless as possible. It really just something that fits in with the Mac user-experience very well. It Just Works.

What more could someone ask for?

I'm in the beginning of helping my niece learn to code, and it's so easy to share a folder with the code files to her. Simple. Easy. Painless. I just can't think of another product that's not made by Apple - that's this frictionless. I'm sure they exist, but none come to mind right now.

What a treat to be able to use great tools like these.

Getting Back into Python for my Niece

Tuesday, April 7th, 2015

python.jpg

This past weekend, I visited my sister in Iowa City, and my niece was there and asked about learning to program in Python. There were no classes in her graduate school (St. Louis University), and so I told her to just pick up a book and go at it. My sister pointed out that a lot of people learn better with the structure of a class, so I told my niece that I'd pick a book, and walk her through it - like a correspondence course in Python.

So I'm back in Python.

I sent her a few book recommendations - an O'Reilly book, and this book from Pragmatic Programmers. I wanted her to be able to read the excerpts of both and see which voice of the author she liked more. It was no surprise to me that she picked the Pragmatic Programmer's book - I have several myself, and they are excellent.

So this morning I'm getting the Pragmatic Programmer book, and scanning the first two chapters. I think it'll be better to use Python 3 from Homebrew as opposed to another source, just because it's self-contained, updatable, and simple. I need to write up a few notes for the first two chapters and then shoot them off to her with a link to a shared Dropbox folder for us to exchange files in - that was her idea, and a good one.

If I can just get these things downloaded on The Shop's WiFi, I'd be in business... but it's a pain in the neck, so it's taking a lot longer than it should. But it'll get there... eventually...