Archive for the ‘Coding’ Category

Interesting Tweet about Ruby Development

Thursday, April 16th, 2015

This morning an old co-worker of mine retweeted this very interesting tweet about Ruby development:

It's true, and I think it's one of the reasons I liked moving from C++ to Ruby - the OO was familiar, and the tools and Convention over Configuration made it nice and easy to get things done - and fast.

But then the reality set in. It was always going to be like this. That's when I realized I don't like big cities. Clojure is the way I think with writing code now, and it's the cleanest way I know to write with simplicity.

Acknowledge truth when you hear it. Amen.

Homebrew Really is an Amazing Resource

Thursday, April 9th, 2015

Homebrew

I have been doing a bit of updating in the last 24 hours, which includes getting the latest Apple compilers and support tools - git 2.3.2, even (nice!)... but I've also been checking with Homebrew for the grc command, and updating it based on the new compilers, etc. from Apple. And it just hit me (again), that a tool like Homebrew makes installation of hundreds of Open Source tools so incredibly easy. It's just almost impossible to think that it's necessary to compile many programs from source on a Mac these days.

Lots of great software is in Homebrew - with sections for Science and all kinds of things. Just an amazing community resource. Amazingly all built on Ruby and git. What a way to build something great.

Colorized Tailed Log4j Files

Thursday, April 9th, 2015

log4j.jpg

I've been looking for something to do a decent job of colorizing the output of tail -f for the log files I watch. Nothing really amazing - just something that will highlight the different levels of the log4j file, and maybe something to pick out IP addresses, etc. Nothing fancy. But what it needs to be is simple.

I don't want to have a tool that takes over the tail command - I still want to be able to scroll up on the terminal and see what was there. I don't want something that's got all the search features, or sorting, or filtering... that's all nice, but I have all the other unix tools for those, and that's how I think. No, what I need is a simple, streaming, colorizer of the log4j stream, and then that's it.

After doing a considerable (for me, for this) digging, it looks like a little app called grc is just the thing I need. The source site is here, but there's a fork of the code on GitHub, and given that the savba.sk domain isn't responding this morning, it's nice to have the GitHub version available.

Home-brew has it as well, and so installing it is pretty simple:

  $ brew install grc

and then the info on grace in Homebrew says that it's simple to start colorizing your Bash scripts by simply adding a config file to your ~/.bashrc. But for me, the configuration is a little more complex.

After reading the README at the GitHub fork, it was clear that I needed to make two files, and put them in a new ~/.grc directory: grc.conf for the general configuration of the tool, and conf.log4j for the colorization rules for the log4j files. The contents of the grc.conf file is pretty simple, and just tells grc what config file to use for what source files:

  # log4j files
  \b\w+\b.*log\b
  conf.log4j

the first line is the regex for the name of the file, and the second line is config file to use. Speaking of which, the conf.log4j was created to look like:

  # From http://fahdshariff.blogspot.ca/2011/04/
  #           highlighting-command-output-with.html
  # this configuration file is suitable for displaying log files
  #errors
  regexp=^.*(ERROR|Error|Exception).*$
  colours=bold red
  ======
  #stack trace
  regexp=^\s+at [^:]*:\d*\)$
  colours=red
  ======
  regexp=^.*(FATAL|Fatal).*$
  colours=on_red
  ======
  regexp=^.*(WARNING|Warning).*$
  colours=bold yellow
  ======
  regexp=^.*(DEBUG|Debug).*$
  colours=bold black
  ======
  # this is a time
  regexp=[^\d]*\d\d:\d\d:\d\d[\.,]{0,1}\d{0,1}\d{0,1}\d{0,1}
  colours=bold blue
  count=once
  ======
  # ip number
  regexp=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}
  colours=bold magenta
  count=more

At this point, I should be able to say:

  $ grc tail -f log/cw.log

and grace would see the log in the filename and pick the right config file and then colorize the output of the tail command. To make this even easier, I added a simple alias in my ~/.bashrc:

  alias lt="grc tail -f"

where, to me, the lt stands for Log Tail, and that's so easy for me to remember.

Now if I can just get that site to show up, and download the code. I should be ready to go. Fingers crossed...

UPDATE: I was able to download grc, and sure enough it's working like a champ!

connect bar

Dropbox is an Amazing Tool

Wednesday, April 8th, 2015

Dropbox.jpg

I've been a Dropbox user for quite a while, and I've held off on the storing of Photos for now because of iCloud, and the fact that my iPhone and Mac already have those things handled quite nicely. But for all the other things I might need a cloud storage system for, I have to concede that Dropbox really is the best - by far.

When I need to move a file from one laptop to another - Dropbox is there. When I want to have it available on my phone - Dropbox is there. The seamlessness of Dropbox is really about as frictionless as possible. It really just something that fits in with the Mac user-experience very well. It Just Works.

What more could someone ask for?

I'm in the beginning of helping my niece learn to code, and it's so easy to share a folder with the code files to her. Simple. Easy. Painless. I just can't think of another product that's not made by Apple - that's this frictionless. I'm sure they exist, but none come to mind right now.

What a treat to be able to use great tools like these.

Getting Back into Python for my Niece

Tuesday, April 7th, 2015

python.jpg

This past weekend, I visited my sister in Iowa City, and my niece was there and asked about learning to program in Python. There were no classes in her graduate school (St. Louis University), and so I told her to just pick up a book and go at it. My sister pointed out that a lot of people learn better with the structure of a class, so I told my niece that I'd pick a book, and walk her through it - like a correspondence course in Python.

So I'm back in Python.

I sent her a few book recommendations - an O'Reilly book, and this book from Pragmatic Programmers. I wanted her to be able to read the excerpts of both and see which voice of the author she liked more. It was no surprise to me that she picked the Pragmatic Programmer's book - I have several myself, and they are excellent.

So this morning I'm getting the Pragmatic Programmer book, and scanning the first two chapters. I think it'll be better to use Python 3 from Homebrew as opposed to another source, just because it's self-contained, updatable, and simple. I need to write up a few notes for the first two chapters and then shoot them off to her with a link to a shared Dropbox folder for us to exchange files in - that was her idea, and a good one.

If I can just get these things downloaded on The Shop's WiFi, I'd be in business... but it's a pain in the neck, so it's taking a lot longer than it should. But it'll get there... eventually...

Great Tool for Selecting the Java Version for OS X

Friday, March 20th, 2015

java-logo-thumb.png

One of the things that I've really missed in the latest version of Mac OS X is the utility to set the default version of the JDK to use. Well, with all the clojure coding I've been doing, I have typically stuck to JDK 1.7, but recently, the auto-updater from Oracle has them shipping JDK 1.8. Yet this doesn't effect the command-line utilities. This is a pain in the neck.

So first, I got JDK 1.7.0_75 on all my machines - it's going to be the last version of JDK 1.7 that I'll get as it's off the auto-updater. Then I was sure that everything was OK, I also got the complete JDK 1.8.0_40 - which is what the auto-updater downloaded for the JRE, but not the JDK. Yes, annoying.

At that point, I started googling about what to do. Turns out, JAVA_HOME is all I need to worry about. The java, jar, java apps in /usr/bin are all respectful of JAVA_HOME. Then I found this function that I added to my ~/.bashrc:

#
# Clever trick to leverage the /usr/bin/java commands to take advantage
# of the JAVA_HOME environment variable and the /usr/libexec/java_home
# executable to change the JDK on-the-fly. This is so easy I'm amazed.
#
function removeFromPath() {
    export PATH=$(echo $PATH | sed -E -e "s;:$1;;" -e "s;$1:?;;")
}
 
function setjdk() {
    if [ $# -ne 0 ]; then
        removeFromPath '/System/Library/Frameworks/JavaVM.framework/Home/bin'
        if [ -n "${JAVA_HOME+x}" ]; then
            removeFromPath $JAVA_HOME
        fi
        export JAVA_HOME=`/usr/libexec/java_home -v $@`
#        export PATH=$JAVA_HOME/bin:$PATH
    fi
}
setjdk 1.7

At this point, I saw that the key command was /usr/libexec/java_home, and it had all the answeres I needed. I didn't need to update my PATH - just JAVA_HOME. I also could verify that I had all the versions of Java I needed:

  $ /usr/libexec/java_home -V
Matching Java Virtual Machines (7):
  1.8.0_40, x86_64:  "Java SE 8"  /Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/
                                   Contents/Home
  1.7.0_75, x86_64:  "Java SE 7"  /Library/Java/JavaVirtualMachines/jdk1.7.0_75.jdk/
                                   Contents/Home
  1.7.0_51, x86_64:  "Java SE 7"  /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/
                                   Contents/Home
  1.7.0_45, x86_64:  "Java SE 7"  /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/
                                   Contents/Home
  1.7.0_13, x86_64:  "Java SE 7"  /Library/Java/JavaVirtualMachines/jdk1.7.0_13.jdk/
                                   Contents/Home
  1.6.0_65-b14-466.1, x86_64: "Java SE 6"  /System/Library/Java/JavaVirtualMachines/
                                            1.6.0.jdk/Contents/Home
  1.6.0_65-b14-466.1, i386:   "Java SE 6"  /System/Library/Java/JavaVirtualMachines/
                                            1.6.0.jdk/Contents/Home

And then with a simple command - in the shell of my choice - I can set - and reset the JDK I use. It's very click:

  $ setjdk 1.7
  $ javac -version
  javac 1.7.0_75
  $ setjdk 1.8
  $ javac -version
  javac 1.8.0_40

Everything else works like a charm. This is a handy little tool to have, and I'm sure I'm going to use it often as I migrate from JDK 1.7 to 1.8.

Did a Sales Simulation Phase – Reloading in Gorilla REPL

Monday, January 5th, 2015

Machine Learning

Today I was talking with Carl, the friend I was doing a little side-project with in Clojure, and it was pretty clear that he didn't understand what the next steps were for the project, and while I tried to explain it to him, he's thinking in terms of cross-sectional probability functions, and I'm thinking this is all a massive Monte Carlo simulation system where the PDFs for the individual events are what you tweak, and then you run the simulation for a year of time, and see what happens. It's very simple, and while it doesn't give you direct control over the things out might want to control, it does give you the ability to see how individual actions - the promotions from one step in the sales pipeline to the next, impacts revenue.

Anyway, so I wanted to make sure that it was all running nicely, and I was using Gorilla REPL for the REPL as it has nice graphing as well as being a generally good REPL, but as I was making changes to the code and refreshing the page, I wasn't seeing what I expected to see.

Then it hit me - Gorilla REPL wasn't reloading the code.

Ouch.

I then switched to using the standard REPL with Leiningen, and everything worked fine - then firing up Gorilla REPL and reloading the page showed that it was now working just fine. Lesson learned - Gorilla doesn't reload the code when you re-execute all the code.

Getting Apache 2.4.9 + PHP 5.5.14 Going on OS X 10.10 Yosemite

Thursday, December 18th, 2014

Yosemite

This morning I thought I'd just check on the status of Apache + PHP + Postgres on OS X 10.10 as it was likely that in the update from 10.9, Apple had changed things, and sure enough, they have. But it wasn't all them - Apache has some changes in it as well, so to be fair, getting this all to run is a little more complex than you might imagine, but it's still not all that hard.

Getting PostgreSQL 9.3

As with the previous post I'm using Homebrew and that's made the entire operation a lot smoother. There's no 32-bit/64-bit issue as Homebrew does both, and it builds in on your box, so again, a lovely customized solution with a simple command:

  $ brew install postgresql

It even has a nice, simple informational blurb about how to start/stop and upgrade. Very nice. But now that it's on the box, and ready to roll, let's add in the PostgreSQL support to the PHP 5.5.14 that's installed with Mac OS X 10.10.

Activating UserDir in Apache 2.4.9

The first thing I noticed is that the UserDir extension is not enabled by default, so we need to get that going right away. This enables the code to be run from the development directories, and that's a big time-saver. First, we need to enable the UserDir module in Apache, and then make a specific config file for the user in question. Start by editing /etc/apache2/httpd.conf and line 166 needs to be uncommented to read:

  LoadModule userdir_module libexec/apache2/mod_userdir.so

and then similarly on line 493 uncomment the line to read:

  Include /private/etc/apache2/extra/httpd-userdir.conf

At this point, you need to make sure you have at least one file in the /etc/apache2/users/ directory for each user, like: drbob.conf:

  <Directory "/Users/drbob/Sites/">
      Options FollowSymLinks Indexes MultiViews
      AllowOverride None
      Order allow,deny
      Allow from all
      Require all granted
  </Directory>

where the last line - Require all granted is new as of Apache 2.4, and without it you will get errors like:

  [Thu Dec 18 10:41:32.385093 2014] [authz_core:error] [pid 55994]
    [client fe80::7a31:c1ff:fed2:ca2c:58108] AH01630: client denied by server
    configuration: /Users/drbob/Sites/info.php

Activating PHP in Apache

The mext thing to do is to activate PHP in the supplied Apache 2 with OS X 10.10. This is line 169 in the file - /etc/apache2/httpd.conf and you need to uncomment it to read:

  LoadModule php5_module libexec/apache2/libphp5.so

and then add a new file called /etc/apache2/other/php5.conf and have it contain:

  <IfModule php5_module>
    AddType application/x-httpd-php .php
    AddType application/x-httpd-php-source .phps
 
    <IfModule dir_module>
      DirectoryIndex index.html index.php
    </IfModule>
  </IfModule>

which does all the other PHP configuration in a separate file to make upgrades easy.

Building in PostgreSQL 9.3 extra for PHP 5.5.14

At this point we need to get the source code for the exact version of PHP that Apple ships. This is pretty easy by getting the PHP 5.5.14 source from a mirror and unpacking it into a directory. We then just run the following commands:

  $ cd php-5.5.14/ext/pgsql/
  $ phpize
  $ ./configure
  $ make

at the end of this process, you'll have a file: php-5.5.14/ext/pgsql/modules/pgsql.so and that needs to be placed in the right directory and referenced in the php.ini file.

For Mac OS X 10.10.1, this is accomplished with:

  $ sudo cp modules/pgsql.so /usr/lib/php/extensions/no-debug-non-zts-20121212/

and then edit the /etc/php.ini file to include this new extension by adding the line:

  extension=pgsql.so

Thankfully, the extension_dir is already set in OS X 10.10.1, and we no longer have to set that.

Finishing Up

At this point, a simple restart of apache:

  $ sudo apachectl restart

and everything should be in order. Hit a URL that's a static file with the contents:

  <?php
    phpinfo();
  ?>

and you should see all the details about the PHP install - including the PostgreSQL section with the version of Postgres indicated:

Postgres + PHP

All is working once again!

You Mean Hadoop Isn’t Perfect?

Thursday, December 18th, 2014

Hadoop

I read with a giggle this article off Slashdot about the problems companies are having making Hadoop actually work in their environment. This is not in the least surprising to me. Hadoop is a nice, distributed, storage environment, but so is a very nice SAN. Both can store a lot of files, and do it very quickly, with redundancy, but one is putting entire computers on each "disk", while the other is allowing the disks to just be... well... disks.

The promise (hype) of Hadoop is that by distributing the computing power like the storage, the map/reduce jobs can be done fast and easily and you get old-style SQL performance with as much online storage as you can muster.

But the truth of the matter is far different than this, as this article in the WSJ attests.

Hadoop is nice, but it's not fast. It's good for lots of storage space, but so is a SAN. It's nice to do small map/reduce jobs, but so is CouchDB. But you can't scale CouchDB to any size you want, either. There will always be limits.

I know folks that are looking at Cassandra, and really like it. It's not as general-purpose as Hadoop, but it's targeted at the problem of massive storage with SQL access. Advocates of Hadoop will say Use Spark - get SQL that way! and that's possible, but then that's not Hadoop, is it?

You can use many different caching schemes to make any storage scheme work faster, but the promise of Hadoop was that you wouldn't need them. That's where it fell over. I have no doubt that Hadoop can do a lot of really good things in a lot of very specialized environments, but it's not the silver bullet worth $2 billion. It's a nice open source map/reduce like CouchDB, but on as many boxes as you want. It's nice... but it's not what people hoped it would be.

Too many distractions in systems development these days. Just too many.

Development Set-Up at Home

Thursday, December 18th, 2014

Thunderbolt

Over break I'm going to be working at home on a few little fun projects, and with my two Thunderbolt displays, I was thinking I might need a hub of some kind, so I went to look for one on the Apple Store web site - just to see what was out there, and what they might cost - if I shopped in town.

I honestly didn't find anything, and then I wondered What's on the back of the monitor? to see if I could chain these together? Turns out, you can. The guy working across from me as two, and they are chained, and I've seen them work just fine. There's even a YouTube video on the set-up.

Very nice!

This will be just what I have when I get started on Saturday morning - if I wait that long. Apple stuff is pricey, yes, but you do really get what you pay for, and this is no exception: one cable to the laptop, driving two 27" displays. Sweet.