Skip to content

Hadoop

Installing and configuring Hadoop on a single machine: core-site, hdfs-site, mapred-site and SSH setup.

2 min read

References

Prerequisites

  1. Homebrew

Installing Hadoop

Terminal window
$ brew install hadoop

Hadoop will be installed in the following directory /usr/local/Cellar/hadoop

Configuring Hadoop

Edit hadoop-env.sh

The file can be located at /usr/local/Cellar/hadoop/2.8.0/libexec/etc/hadoop/hadoop-env.sh where 2.8.0 is the hadoop version.

Find the line with

export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"

and change it to

export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc="

Edit Core-site.xml

The file can be located at /usr/local/Cellar/hadoop/2.8.0/libexec/etc/hadoop/core-site.xml

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/Cellar/hadoop/hdfs/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>

Edit mapred-site.xml

The file can be located at /usr/local/Cellar/hadoop/2.8.0/libexec/etc/hadoop/mapred-site.xml and by default will be blank.

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9010</value>
</property>
</configuration>

Edit hdfs-site.xml

The file can be located at /usr/local/Cellar/hadoop/2.8.0/libexec/etc/hadoop/hdfs-site.xml .

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

To simplify life edit your ~/.profile using vim or your favorite editor and add the following two commands. By default ~/.profile might not exist. If it doesn’t work after restarting terminal, try to edit ~/.bash_profile.

Terminal window
alias hstart="/usr/local/Cellar/hadoop/2.8.0/sbin/start-dfs.sh;/usr/local/Cellar/hadoop/2.8.0/sbin/start-yarn.sh"
alias hstop="/usr/local/Cellar/hadoop/2.8.0/sbin/stop-yarn.sh;/usr/local/Cellar/hadoop/2.8.0/sbin/stop-dfs.sh"
and execute
Terminal window
$ source ~/.profile

in the terminal to update.

Before we can run Hadoop we first need to format the HDFS using

Terminal window
$ hdfs namenode -format

SSH Localhost

Nothing needs to be done here if you have already generated ssh keys. To verify just check for the existance of ~/.ssh/id_rsa and the ~/.ssh/id_rsa.pub files. If not the keys can be generated using

Terminal window
$ ssh-keygen -t rsa

Enable Remote Login

“System Preferences” -> “Sharing”. Check “Remote Login”

Authorize SSH Keys

To allow your system to accept login, we have to make it aware of the keys that will be used

Terminal window
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Let’s try to login.

Terminal window
$ ssh localhost
Last login: Fri Mar 6 20:30:53 2015
$ exit

Running Hadoop

Now we can run Hadoop just by typing

Terminal window
$ hstart

and stopping using

Terminal window
$ hstop

Using Basic Hadoop Commands

Terminal window
#Copying files from local to Hadoop File System
$ hadoop fs -copyFromLocal test.txt
#Listing files and folders
$ hadoop fs -ls
#Creating a directory named Directory1
$ hadoop fs -mkdir Directory1
#Moving test.txt to Directory1
$ hadoop fs -mv test.txt Directory1
#Running wordcount example for the files under Directory1 and write results under output directory
$ hadoop jar hadoop-mapreduce-examples-2.6.0.jar wordcount Directory1 output
#View the output file
$ hadoop fs -cat output/part-r-00000

Related articles

1 min readTürkçe

ollama

CPU'lu makinelerde Ollama ile yerel LLM çalıştırma ve performansı artıran kritik ayarlar.

  • AI & Data
  • DevOps & Containers
1 min read

Algorithms

Distance measures for data science: Manhattan, Euclidean and Minkowski.

  • AI & Data
5 min read

Algorithms for machine learning

An overview of supervised learning algorithms — regression, classification and the math behind how models learn.

  • AI & Data

Type to search the whole site.