{"id":6991,"date":"2013-07-17T08:26:40","date_gmt":"2013-07-17T13:26:40","guid":{"rendered":"http:\/\/bobbeaty.com\/wp\/?p=6991"},"modified":"2013-07-29T14:24:44","modified_gmt":"2013-07-29T19:24:44","slug":"installing-hadoop-on-os-x","status":"publish","type":"post","link":"https:\/\/bobbeaty.com\/wp\/archives\/6991","title":{"rendered":"Installing Hadoop on OS X"},"content":{"rendered":"

\"Hadoop\"<\/p>\n

This morning I finally<\/em> got Hadoop<\/a> installed on my work laptop, and I wanted to write it all down so that I could repeat this when necessary. As I found out, it's not at all<\/em> like installing CouchDB<\/a> which is about as simple as anything could be. No\u2026 Hadoop is a far more difficult beast, and I guess I can understand why, but still, it'd be nice to have a simple Homebrew<\/a> install that set it up in single-node mode and started everything with Launch Control, but that's a wish<\/em>, not a necessity.<\/p>\n

So let's get into it. First, make sure that you have the SSH daemon running on your box. This is controlled in System Preferences -> Sharing -> Remote Login - make sure it's checked, save this, and it should be running just fine. Make sure you can ssh<\/tt> into your box - if necessary, make the SSH keys and put them in your ~\/.ssh<\/tt> directory.<\/p>\n

Next, you certainly need to install Homebrew<\/a>, and once that's all going, you need to install the basic Hadoop<\/a> package:<\/p>\n

$ brew install<\/span> hadoop<\/pre>\n
at this point, you will need to edit a few of the config files, and make a few directories. Let's start by making the directories. These will be the locations for the actual Hadoop data, the Map\/Reduce data, and the NameNode data. I picked to place these next to the Homebrew<\/a> install of Hadoop<\/a> so that it's all in one place:<\/p>\n
  $ cd<\/span> \/<\/span>usr\/<\/span>local\/<\/span>Cellar\/<\/span>hadoop\n  $ mkdir<\/span> data\n  $ cd<\/span> data\n  $ mkdir<\/span> dfs\n  $ mkdir<\/span> mapred\n  $ mkdir<\/span> nn<\/pre>\n
At this point we can go to the directory with the configuration files and update them:<\/p>\n
  $ cd<\/span> \/<\/span>usr\/<\/span>local\/<\/span>Cellar\/<\/span>hadoop\/<\/span>1.1.2\/<\/span>libexec\/<\/span>conf<\/pre>\n
The first update is to handle a Kerberos bug in Hadoop - a known bug. Do this by editing hadoop-env.sh<\/tt> to include:<\/p>\n
  export<\/span> HADOOP_OPTS<\/span>="-Djava.security.krb5.realm= -Djava.security.krb.kdc="<\/span><\/pre>\n
Next, edit the hdfs-site.xml<\/tt> file to include the following:<\/p>\n
  <configuration><\/span><\/span><\/span>\n    <property><\/span><\/span><\/span>\n      <name><\/span><\/span><\/span>dfs.data.dir<\/name><\/span><\/span><\/span>\n      <value><\/span><\/span><\/span>\/usr\/local\/Cellar\/hadoop\/data\/dfs<\/value><\/span><\/span><\/span>\n    <\/property><\/span><\/span><\/span>\n    <property><\/span><\/span><\/span>\n      <name><\/span><\/span><\/span>dfs.name.dir<\/name><\/span><\/span><\/span>\n      <value><\/span><\/span><\/span>\/usr\/local\/Cellar\/hadoop\/data\/nn<\/value><\/span><\/span><\/span>\n    <\/property><\/span><\/span><\/span>\n    <property><\/span><\/span><\/span>\n      <name><\/span><\/span><\/span>dfs.replication<\/name><\/span><\/span><\/span>\n      <value><\/span><\/span><\/span>1<\/value><\/span><\/span><\/span>\n    <\/property><\/span><\/span><\/span>\n    <property><\/span><\/span><\/span>\n      <name><\/span><\/span><\/span>dfs.webhdfs.enabled<\/name><\/span><\/span><\/span>\n      <value><\/span><\/span><\/span>true<\/value><\/span><\/span><\/span>\n    <\/property><\/span><\/span><\/span>\n  <\/configuration><\/span><\/span><\/span><\/pre>\n
Next, edit the core-site.xml<\/tt> file to include the following:<\/p>\n
  <configuration><\/span><\/span><\/span>\n    <property><\/span><\/span><\/span>\n      <name><\/span><\/span><\/span>hadoop.tmp.dir<\/name><\/span><\/span><\/span>\n      <value><\/span><\/span><\/span>\/tmp\/hdfs-${user.name}<\/value><\/span><\/span><\/span>\n    <\/property><\/span><\/span><\/span>\n    <property><\/span><\/span><\/span>\n      <name><\/span><\/span><\/span>fs.default.name<\/name><\/span><\/span><\/span>\n      <value><\/span><\/span><\/span>hdfs:\/\/localhost:9000<\/value><\/span><\/span><\/span>\n      <description><\/span><\/span><\/span>The name of the default file system.  A URI whose\n      scheme and authority determine the FileSystem implementation.  The\n      uri's scheme determines the config property (fs.SCHEME.impl) naming\n      the FileSystem implementation class.  The uri's authority is used to\n      determine the host, port, etc. for a filesystem.<\/description><\/span><\/span><\/span>\n    <\/property><\/span><\/span><\/span>\n  <\/configuration><\/span><\/span><\/span><\/pre>\n
Finally, edit the mapred-site.xml<\/tt> file to include the following:<\/p>\n
  <configuration><\/span><\/span><\/span>\n    <property><\/span><\/span><\/span>\n      <name><\/span><\/span><\/span>mapred.job.tracker<\/name><\/span><\/span><\/span>\n      <value><\/span><\/span><\/span>localhost:9001<\/value><\/span><\/span><\/span>\n      <description><\/span><\/span><\/span>The host and port that the MapReduce job tracker runs\n      at.  If "local", then jobs are run in-process as a single map\n      and reduce task.<\/description><\/span><\/span><\/span>\n    <\/property><\/span><\/span><\/span>\n    <property><\/span><\/span><\/span>\n      <name><\/span><\/span><\/span>mapred.local.dir<\/name><\/span><\/span><\/span>\n      <value><\/span><\/span><\/span>\/usr\/local\/Cellar\/hadoop\/data\/mapred\/<\/value><\/span><\/span><\/span>\n    <\/property><\/span><\/span><\/span>\n  <\/configuration><\/span><\/span><\/span><\/pre>\n
We are finally all configured. At this point, you need to initialize the Name node:<\/p>\n
  $ hadoop namenode -format<\/span><\/pre>\n
and then you can start all the necessary processes on the box:<\/p>\n
  $ start-all.sh<\/pre>\n
At this point, you will be able to hit the endpoints:<\/p>\n