Installing Hadoop: Ubuntu/WSL and a Docker cluster

Two install paths, one working cluster each

Installing Hadoop

Words you'll meet in this lesson

What was tested for this lesson

Installing at three levels

Ways to install Hadoop

How organisations get Hadoop

Path A: install natively on Ubuntu or WSL

Prerequisites from Lesson 4.1 must be done: Java 17, SSH server running, JAVA_HOME in ~/.bashrc. All commands run in the Ubuntu terminal as your normal user; the user here is riya, so replace /home/riya with your own home folder.

1. Download and verify

   cd ~
   wget https://dlcdn.apache.org/hadoop/common/hadoop-3.5.0/hadoop-3.5.0.tar.gz
   wget https://dlcdn.apache.org/hadoop/common/hadoop-3.5.0/hadoop-3.5.0.tar.gz.sha512
   sha512sum -c hadoop-3.5.0.tar.gz.sha512

   hadoop-3.5.0.tar.gz: OK
   Anything other than OK means a broken or altered download: delete it and download again. If a newer 3.5.x exists when you read this, change the version number everywhere.

2. Unpack and set the environment

   tar -xzf hadoop-3.5.0.tar.gz
   cat >> ~/.bashrc <<'EOF'
   export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
   export HADOOP_HOME=$HOME/hadoop-3.5.0
   export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
   EOF
   source ~/.bashrc
   hadoop version | head -1

   Hadoop 3.5.0
   If you added the JAVA_HOME line in Lesson 4.1, don't add it twice. bin holds the user commands (hdfs, yarn, hadoop); sbin holds the start and stop scripts.

3. Tell Hadoop where Java is

   echo "export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64" >> $HADOOP_HOME/etc/hadoop/hadoop-env.sh

   This is the fix for ERROR: JAVA_HOME is not set and could not be found: daemons started over SSH read this file, not your ~/.bashrc.

4. core-site.xml and hdfs-site.xml

   <configuration>
       <property>
           <name>fs.defaultFS</name>
           <value>hdfs://localhost:9000</value>
       </property>
   </configuration>

   <configuration>
       <property>
           <name>dfs.replication</name>
           <value>1</value>
       </property>
       <property>
           <name>dfs.namenode.name.dir</name>
           <value>file:///home/riya/hadoop-data/namenode</value>
       </property>
       <property>
           <name>dfs.datanode.data.dir</name>
           <value>file:///home/riya/hadoop-data/datanode</value>
       </property>
   </configuration>

   The two dir settings move HDFS out of /tmp, which Linux may empty on reboot. Open each file with nano and replace the empty <configuration> block.

5. mapred-site.xml and yarn-site.xml

   <configuration>
       <property>
           <name>mapreduce.framework.name</name>
           <value>yarn</value>
       </property>
       <property>
           <name>mapreduce.application.classpath</name>
           <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
       </property>
   </configuration>

   <configuration>
       <property>
           <name>yarn.nodemanager.aux-services</name>
           <value>mapreduce_shuffle</value>
       </property>
       <property>
           <name>yarn.nodemanager.env-whitelist</name>
           <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
       </property>
       <property>
           <name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>
           <value>99.5</value>
       </property>
   </configuration>

   The first two properties come from the official single-node guide: they run MapReduce on YARN and let containers find the MapReduce jars. The third is for laptops: by default a NodeManager whose disk is over 90% full marks itself unhealthy and runs nothing (see the errors table). Leave it out on real servers.

6. Passwordless SSH to localhost

   ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
   cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
   chmod 0600 ~/.ssh/authorized_keys
   ssh localhost "echo connected"

   Warning: Permanently added 'localhost' (ED25519) to the list of known hosts. connected
   The first connection asks to trust the host key (type yes). After that, no questions and no password: exactly what start-dfs.sh needs.

7. Format the NameNode (once)

   hdfs namenode -format

   … INFO common.Storage: Storage directory /home/riya/hadoop-data/namenode has been successfully formatted. …
   Look for “successfully formatted” in the long output. Never run this again on a cluster with data you need.

8. Start HDFS and check it

   start-dfs.sh
   jps
   hdfs dfsadmin -safemode wait
   hdfs dfs -mkdir -p /user/riya
   hdfs dfs -mkdir input
   hdfs dfs -put $HADOOP_HOME/etc/hadoop/*.xml input

   Starting namenodes on [localhost] Starting datanodes Starting secondary namenodes [vm] vm: Warning: Permanently added 'vm' (ED25519) to the list of known hosts. 4865 NameNode 4966 DataNode 5304 Jps 5166 SecondaryNameNode Safe mode is OFF
   /user/riya is your HDFS home: relative paths like input mean /user/riya/input. The NameNode web UI is now at http://localhost:9870 (WSL forwards it to Windows).

9. Start YARN and run a job

   start-yarn.sh
   yarn node -list
   hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.5.0.jar \
       grep input output 'dfs[a-z.]+'
   hdfs dfs -cat output/*

   Starting resourcemanager Starting nodemanagers Total Nodes:1 Node-Id Node-State Node-Http-Address Number-of-Running-Containers localhost:40123 RUNNING localhost:8042 0 … INFO mapreduce.Job: Job job_1792562428511_0001 completed successfully … 1 dfsadmin 1 dfs.replication 1 dfs.namenode.name.dir 1 dfs.datanode.data.dir
   The ResourceManager UI is at http://localhost:8088. The result lists the dfs… property names found in the XML files you uploaded, including the ones you just wrote.

10. Stop, and start again later

    stop-yarn.sh
    stop-dfs.sh
    # next time:
    start-dfs.sh
    start-yarn.sh
    hdfs dfs -cat output/*

    Stopping nodemanagers Stopping resourcemanager Stopping namenodes on [localhost] Stopping datanodes Stopping secondary namenodes [vm] … 1 dfsadmin 1 dfs.replication 1 dfs.namenode.name.dir 1 dfs.datanode.data.dir
    The output is still there after a full stop and start, because HDFS lives in ~/hadoop-data. On WSL, also run sudo service ssh start after reopening Ubuntu.

Path B: a 3-DataNode cluster with Docker Compose

Install Docker Desktop (Windows/macOS, with the WSL 2 back end on Windows) or Docker Engine (Linux), and give it at least 6 GB of memory. Then make a folder hadoop-cluster with these two files.

1. docker-compose.yaml

   services:
     namenode:
       image: apache/hadoop:3.5.0
       hostname: namenode
       command: ["hdfs", "namenode"]
       ports:
         - 9870:9870
       env_file:
         - ./config
       environment:
         ENSURE_NAMENODE_DIR: "/tmp/hadoop-hadoop/dfs/name"
     datanode:
       image: apache/hadoop:3.5.0
       command: ["hdfs", "datanode"]
       env_file:
         - ./config
     resourcemanager:
       image: apache/hadoop:3.5.0
       hostname: resourcemanager
       command: ["yarn", "resourcemanager"]
       ports:
         - 8088:8088
       env_file:
         - ./config
     nodemanager:
       image: apache/hadoop:3.5.0
       command: ["yarn", "nodemanager"]
       env_file:
         - ./config

   One service per daemon. ENSURE_NAMENODE_DIR: if this folder does not exist, the image formats the NameNode before starting it. It must be the NameNode's real folder; the image runs as user hadoop, so that is /tmp/hadoop-hadoop/dfs/name. (With /tmp/hadoop-root/…, as in some older examples, the NameNode is re-formatted on every start; this happened during testing.) Only the NameNode and ResourceManager UIs are published to your PC.

2. config (the settings)

   CORE-SITE.XML_fs.defaultFS=hdfs://namenode
   HDFS-SITE.XML_dfs.namenode.rpc-address=namenode:8020
   HDFS-SITE.XML_dfs.replication=3
   MAPRED-SITE.XML_mapreduce.framework.name=yarn
   MAPRED-SITE.XML_yarn.app.mapreduce.am.env=HADOOP_MAPRED_HOME=/opt/hadoop
   MAPRED-SITE.XML_mapreduce.map.env=HADOOP_MAPRED_HOME=/opt/hadoop
   MAPRED-SITE.XML_mapreduce.reduce.env=HADOOP_MAPRED_HOME=/opt/hadoop
   YARN-SITE.XML_yarn.resourcemanager.hostname=resourcemanager
   YARN-SITE.XML_yarn.nodemanager.pmem-check-enabled=false
   YARN-SITE.XML_yarn.nodemanager.vmem-check-enabled=false
   YARN-SITE.XML_yarn.nodemanager.aux-services=mapreduce_shuffle
   YARN-SITE.XML_yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage=99.5
   CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.maximum-am-resource-percent=0.5
   CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.queues=default
   CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.capacity=100
   CAPACITY-SCHEDULER.XML_yarn.scheduler.capacity.root.default.maximum-capacity=100

   Each line is FILE_property=value; the image writes them into the matching XML file when a container starts. Write /opt/hadoop literally: this file does not expand variables like $HADOOP_HOME.

3. Start with three DataNodes

   docker compose up -d --scale datanode=3
   docker compose ps --format "table {{.Name}}\t{{.Status}}"

   NAME STATUS hadoop-cluster-datanode-1 Up 35 seconds hadoop-cluster-datanode-2 Up 35 seconds hadoop-cluster-datanode-3 Up 35 seconds hadoop-cluster-namenode-1 Up 36 seconds hadoop-cluster-nodemanager-1 Up 36 seconds hadoop-cluster-resourcemanager-1 Up 36 seconds
   The first run downloads the image (about 2.2 GB unpacked). Six containers, six “machines”, one network.

4. Check the cluster

   docker compose exec namenode hdfs dfsadmin -printTopology
   docker compose exec resourcemanager yarn node -list

   Rack: /default-rack 172.18.0.6:9866 (hadoop-cluster-datanode-3.hadoop-cluster_default) In Service 172.18.0.5:9866 (hadoop-cluster-datanode-1.hadoop-cluster_default) In Service 172.18.0.7:9866 (hadoop-cluster-datanode-2.hadoop-cluster_default) In Service Total Nodes:1 Node-Id Node-State Node-Http-Address Number-of-Running-Containers 8f368ea04df2:45495 RUNNING 8f368ea04df2:8042 0
   Three DataNodes in HDFS, one NodeManager in YARN. The NodeManager's name is its container ID.

5. Store a file and see its three copies

   docker compose exec namenode hdfs dfs -mkdir -p /user/hadoop/data
   docker compose cp canteen_orders.csv namenode:/tmp/canteen_orders.csv
   docker compose exec namenode hdfs dfs -put /tmp/canteen_orders.csv data/
   docker compose exec namenode hdfs fsck /user/hadoop/data/canteen_orders.csv -files -blocks -locations

   /user/hadoop/data/canteen_orders.csv 4692009 bytes, replicated: replication=3, 1 block(s): OK 0. BP-1055690327-172.18.0.2-1792562685335:blk_1073741825_1001 len=4692009 Live_repl=3 [DatanodeInfoWithStorage[172.18.0.6:9866,…], DatanodeInfoWithStorage[172.18.0.7:9866,…], DatanodeInfoWithStorage[172.18.0.5:9866,…]] Status: HEALTHY Number of data-nodes: 3
   canteen_orders.csv is a 3 lakh-row practice file (order_id, item, amount); any CSV works. One 4.5 MB block, stored on all three DataNodes.

6. Run a job, then see what down does

   docker compose exec namenode yarn jar \
       /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.5.0.jar \
       grep data out "vada pav|samosa|dosa"
   docker compose exec namenode hdfs dfs -cat "out/*"
   docker compose down
   docker compose up -d --scale datanode=3
   docker compose exec namenode hdfs dfs -ls data

   INFO Job:1765 - Job job_1792562696287_0001 completed successfully Rack-local map tasks=1 … 50021 samosa 49986 vada pav 49948 dosa ls: `data': No such file or directory
   down deletes the containers, and with them the HDFS data. The map task is “rack-local”, not “data-local”, because DataNodes and the NodeManager are separate containers here.

7. Pause and resume without losing data

   docker compose stop
   docker compose start
   docker compose exec namenode hdfs dfsadmin -safemode wait
   docker compose exec namenode hdfs dfs -ls data

   Safe mode is OFF Found 1 items -rw-r--r-- 3 hadoop supergroup 4692009 2026-10-21 06:15 data/canteen_orders.csv
   stop keeps the containers and their disks, so the file is still there (after uploading it again following the down above). Use stop/start at the end of each practice session.

Errors you will meet (all seen during testing)

What docker compose up builds, and what down destroys

Kabir's cluster from Path B, step by step. Every terminal line is from the real test run.

Installs at work

Simplified; organisations differ in tools and policies.

Turn the Docker config file into Hadoop XML

The apache/hadoop image reads lines like CORE-SITE.XML_fs.defaultFS=hdfs://namenode and writes them into the right XML file. This program does the same, so you can see the XML your containers really use.

config = """CORE-SITE.XML_fs.defaultFS=hdfs://namenode
HDFS-SITE.XML_dfs.replication=3
YARN-SITE.XML_yarn.resourcemanager.hostname=resourcemanager
YARN-SITE.XML_yarn.nodemanager.aux-services=mapreduce_shuffle"""

files = {}
for line in config.splitlines():
    key, value = line.split("=", 1)
    filename, prop = key.split("_", 1)
    files.setdefault(filename.lower(), []).append((prop, value))

for filename, props in files.items():
    print(f"--- {filename}")
    print("<configuration>")
    for name, value in props:
        print(f"  <property><name>{name}</name><value>{value}</value></property>")
    print("</configuration>")

• key, value = line.split("=", 1)Split only at the first =: values such as HADOOP_MAPRED_HOME=/opt/hadoop contain another one.
• filename, prop = key.split("_", 1)The file name is before the first underscore. Property names contain dots and sometimes underscores, so split only once.
• files.setdefault(filename.lower(), []).append((prop, value))setdefault creates an empty list the first time a file name appears, then keeps adding to it.

Run it on your PC: make the practice file and check Hadoop's answer

Path B uses canteen_orders.csv. This program creates it (the same file on every computer, because the random generator is seeded) and then counts the items with plain Python, so you can check the answer your Hadoop job gives. Save it as make_canteen.py in the hadoop-cluster folder and run it.

# make_canteen.py: create the practice file canteen_orders.csv (3 lakh orders)
import os
import random

random.seed(7)                       # same file on every computer
prices = {"chai": 15, "vada pav": 20, "samosa": 18, "poha": 25, "idli": 30, "dosa": 45}
items = list(prices)

with open("canteen_orders.csv", "w") as f:
    f.write("order_id,item,amount\n")
    for order_id in range(1001, 1001 + 300_000):
        item = random.choice(items)
        f.write(f"{order_id},{item},{prices[item]}\n")

print(f"Wrote canteen_orders.csv: 300,000 orders, {os.path.getsize('canteen_orders.csv'):,} bytes")

# Check Hadoop's answer with plain Python
counts = {}
with open("canteen_orders.csv") as f:
    next(f)                          # skip the header
    for line in f:
        item = line.split(",")[1]
        counts[item] = counts.get(item, 0) + 1
for item in ["samosa", "vada pav", "dosa"]:
    print(f"{counts[item]:>6}  {item}")

# In a terminal (in the hadoop-cluster folder):
#   python3 make_canteen.py

One config line becomes one XML property

Follow a single line of the Docker config file through the conversion and watch each variable.

Worked examples: real problems, solved and traced in memory

Three things people script during real installs: writing the site files from one table, checking an install stage by stage, and catching configuration mistakes before starting a cluster. Each runs live and is traced in memory.

core: fs.defaultFS · hdfs: dfs.replication · mapred: mapreduce.framework.name · yarn: aux-services

== core-site.xml
<configuration>
    <property><name>fs.defaultFS</name><value>hdfs://localhost:9000</value></property>
</configuration>
== hdfs-site.xml
<configuration>
    <property><name>dfs.replication</name><value>1</value></property>
</configuration>
== mapred-site.xml
<configuration>
    <property><name>mapreduce.framework.name</name><value>yarn</value></property>
</configuration>
== yarn-site.xml
<configuration>
    <property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property>
</configuration>

version OK · jps has NameNode, DataNode, SecondaryNameNode · safe mode OFF · yarn nodes 0 · job not run

PASS  hadoop version
PASS  HDFS daemons in jps
PASS  safe mode OFF
FAIL  yarn node -list > 0
      fix: NodeManager unhealthy? check disk space and NM log

replication 3 · 1 DataNode · name.dir /tmp/… · aux-services missing · framework yarn

- replication is higher than the number of DataNodes
- NameNode metadata under /tmp may vanish on reboot
- YARN has no mapreduce_shuffle service

Check yourself

Quick revision

Interview and viva questions

Practice problems

Exercises on the install itself: paths, configuration, checks and errors. All runnable in the browser.

version = "3.5.0"

https://dlcdn.apache.org/hadoop/common/hadoop-3.5.0/hadoop-3.5.0.tar.gz
https://dlcdn.apache.org/hadoop/common/hadoop-3.5.0/hadoop-3.5.0.tar.gz.sha512

v = "3.5.0"
url = f"https://dlcdn.apache.org/hadoop/common/hadoop-{v}/hadoop-{v}.tar.gz"
print(url)
print(url + ".sha512")

user riya · input · output/part-r-00000

input -> /user/riya/input
output/part-r-00000 -> /user/riya/output/part-r-00000

user = "riya"
for p in ["input", "output/part-r-00000"]:
    print(p, "->", f"/user/{user}/{p}")

NameNode 9870 · ResourceManager 8088

NameNode         http://localhost:9870
ResourceManager  http://localhost:8088

for name, port in [("NameNode", 9870), ("ResourceManager", 8088)]:
    print(f"{name:16} http://localhost:{port}")

len=4692009 Live_repl=3

4.5 MB, 3 live replicas

line = "len=4692009 Live_repl=3"
fields = dict(part.split("=") for part in line.split())
print(f"{int(fields['len']) / 1024 / 1024:.1f} MB, {fields['Live_repl']} live replicas")

465 GB disk · 430 GB used · 90% and 99.5%

used 92.5% vs limit 90% -> UNHEALTHY
used 92.5% vs limit 99.5% -> healthy

used_pct = 430 / 465 * 100
for limit in [90, 99.5]:
    print(f"used {used_pct:.1f}% vs limit {limit}% -> {'healthy' if used_pct <= limit else 'UNHEALTHY'}")

6 × 0.3 GB · 4 GB · 1 GB per job container

daemons use 1.8 GB, 2.2 GB left -> 2 job container(s)

used = 6 * 0.3
left = 4 - used
print(f"daemons use {used:.1f} GB, {left:.1f} GB left -> {int(left // 1)} job container(s)")

12 installs · 45 min vs 5 min · 3 h setup

manual 9 h, compose 4 h, saved 5 h

manual = 12 * 45 / 60
compose = 3 + 12 * 5 / 60
print(f"manual {manual:.0f} h, compose {compose:.0f} h, saved {manual - compose:.0f} h")

JAVA_HOME … · Permission denied (publickey,password) · Output directory … already exists

set JAVA_HOME in etc/hadoop/hadoop-env.sh
set up passwordless SSH to the host
hdfs dfs -rm -r <output> or choose a new folder

fixes = [("JAVA_HOME is not set", "set JAVA_HOME in etc/hadoop/hadoop-env.sh"),
         ("Permission denied (publickey", "set up passwordless SSH to the host"),
         ("already exists", "hdfs dfs -rm -r <output> or choose a new folder"),
         ("safe mode", "hdfs dfsadmin -safemode wait"),
         ("MRAppMaster", "set HADOOP_MAPRED_HOME / mapreduce.application.classpath")]
def diagnose(msg):
    return next((fix for key, fix in fixes if key in msg), "read the daemon log in $HADOOP_HOME/logs")
for m in ["localhost: ERROR: JAVA_HOME is not set and could not be found.",
          "riya@localhost: Permission denied (publickey,password).",
          "Output directory hdfs://localhost:9000/user/riya/output already exists"]:
    print(diagnose(m))

Real-world scenario questions (asked in company interviews)

Installation questions test whether you have really done it. Answer with the order of steps, the checks and the errors you met.