Prerequisites: Linux, Java and SSH

Three things Hadoop needs before it will even start

Hadoop's prerequisites

Words you'll meet in this lesson

What you need (Hadoop 3.5.0, practice setup)

Source for the Java requirement: the Apache Hadoop “Hadoop Java Versions” wiki page. Always check it before installing a new release.

Prerequisites at three levels

Four ways to get a Linux environment for Hadoop

Which Java for which Hadoop?

Prepare a fresh Ubuntu (or WSL) for Hadoop

1. Windows only: install WSL 2 with Ubuntu

   wsl --install -d Ubuntu-24.04

   Restart when asked, open “Ubuntu” from the Start menu and create a Linux user name and password. From now on, every command runs in that Ubuntu terminal.

2. Install Java 17 and SSH

   sudo apt update
   sudo apt install -y openjdk-17-jdk-headless openssh-server openssh-client wget

   headless means no graphical parts, which servers don't need. openssh-server provides sshd, which the start scripts connect to.

3. Find where Java lives

   java -version
   readlink -f $(which java)

   openjdk version "17.0.20.1" 2026-08-18 OpenJDK Runtime Environment (build 17.0.20.1+1-1-24.04-Ubuntu) OpenJDK 64-Bit Server VM (build 17.0.20.1+1-1-24.04-Ubuntu, mixed mode, sharing) /usr/lib/jvm/java-17-openjdk-amd64/bin/java
   JAVA_HOME is that path without /bin/java: /usr/lib/jvm/java-17-openjdk-amd64. The exact update number on your machine will differ.

4. Set variables permanently

   export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
   export HADOOP_HOME=$HOME/hadoop-3.5.0
   export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

   Then run source ~/.bashrc. HADOOP_HOME points to where you'll unpack Hadoop in Lesson 4.3; adding bin and sbin to PATH lets you type hdfs and start-dfs.sh from any folder.

5. Start the SSH server

   sudo service ssh start

   On WSL, services don't always start automatically: run this each time you open Ubuntu (or enable systemd in WSL). Passwordless keys are set up in Lesson 4.3, right before the first start-dfs.sh.

Kabir's three errors: what he sees, and what is really missing

Every message on the left is real output from Hadoop 3.5.0 on Ubuntu 24.04. On the right: which leg was missing.

Prerequisites at work

Simplified; organisations differ in tools and policies.

A prerequisite checker in Python

Real tools such as Ambari and Cloudera Manager run host checks before installing anything. This small version reads facts about a machine (here typed in as a dictionary) and reports what is missing. Run it, then try the challenges.

machine = {
    "os": "Ubuntu 24.04.5 LTS",
    "java": 'openjdk version "17.0.20.1" 2026-08-18',
    "java_home": "/usr/lib/jvm/java-17-openjdk-amd64",
    "ram_gb": 7.8,
    "disk_free_gb": 19,
    "ssh_server": True,
    "passwordless_ssh": False,
}

def java_major(line):
    version = line.split('"')[1]          # 17.0.20.1
    first = version.split(".")[0]          # 17
    return int(first)

problems = []
if java_major(machine["java"]) < 17:
    problems.append("Hadoop 3.5 needs Java 17")
if not machine["java_home"]:
    problems.append("JAVA_HOME is not set")
if machine["ram_gb"] < 8:
    problems.append("less than 8 GB RAM: run fewer daemons")
if machine["disk_free_gb"] < 10:
    problems.append("less than 10 GB free disk")
if not machine["ssh_server"]:
    problems.append("install openssh-server")
if not machine["passwordless_ssh"]:
    problems.append("set up passwordless SSH keys")

print("Java major version:", java_major(machine["java"]))
for p in problems:
    print("FIX:", p)
print("Ready!" if not problems else f"{len(problems)} thing(s) to fix")

• "java": 'openjdk version "17.0.20.1" 2026-08-18',This is the exact first line that java -version printed on our test machine.
• version = line.split('"')[1] # 17.0.20.1Splitting on the double quote gives three pieces; the version is the middle one.
• first = version.split(".")[0] # 17The major version is the part before the first dot. (Old Java 8 printed 1.8.0_…; challenge 3 handles that.)
• if machine["ram_gb"] < 8:7.8 GB shows up as “less than 8” because the operating system reserves a little memory. Hadoop still runs; you just run fewer things at once.
• print("Ready!" if not problems else f"{len(problems)} thing(s) to fix")An empty list counts as False in Python, so not problems means “nothing to fix”.

Run it on your machine: is it ready for Hadoop?

This shell script checks everything from Part 4 in one go. Save it in your Ubuntu or WSL home folder as check_prereqs.sh and run it before and after the install steps. Both outputs below are from a real run on a fresh Ubuntu 24.04 machine.

#!/usr/bin/env bash
# check_prereqs.sh: is this Linux / WSL machine ready for Hadoop 3.5?
echo "== Operating system";  grep PRETTY_NAME /etc/os-release | cut -d= -f2
echo "== CPU cores";         nproc
echo "== Memory";            free -h | awk '/Mem:/ {print $2 " total, " $7 " available"}'
echo "== Free disk in HOME"; df -h "$HOME" | awk 'NR==2 {print $4 " free"}'
echo "== Java";              if command -v java >/dev/null; then java -version 2>&1 | head -1; else echo "NOT INSTALLED"; fi
echo "== JAVA_HOME";         echo "${JAVA_HOME:-not set}"
echo "== SSH client";        command -v ssh >/dev/null && echo "installed" || echo "NOT INSTALLED"
echo "== SSH server";        (command -v sshd >/dev/null || [ -x /usr/sbin/sshd ]) && echo "installed" || echo "NOT INSTALLED"
echo "== Passwordless SSH to localhost"
if ssh -o BatchMode=yes -o ConnectTimeout=3 localhost true 2>/dev/null; then echo "OK"; else echo "NOT YET (set up keys in Lesson 4.3)"; fi

# In the Ubuntu / WSL terminal:
#   bash check_prereqs.sh

sudo apt install -y openjdk-17-jdk-headless openssh-server openssh-client
source ~/.bashrc
sudo service ssh start
# keys from Lesson 4.3, then:
bash check_prereqs.sh

Reading a version string, one step at a time

Follow the real java -version line through the parser and watch each variable change in memory.

Worked examples: real problems, solved and traced in memory

Three jobs a Hadoop admin really does before installing: checking a list of machines, sizing memory for the daemons, and generating the environment lines every node needs. Each runs live and is traced in memory.

lab1: Java 17, 16 GB, 120 GB · lab2: Java 11, 16 GB, 80 GB · lab3: Java 17, 4 GB, 50 GB · lab4: Java 21, 32 GB, 10 GB

lab1: ready
lab2: needs Java 17
lab3: needs RAM
lab4: needs disk

8 GB RAM · 6 daemons × 1 GB · OS 2 GB · 1 GB per container

Daemons + OS use 8 GB, 0 GB left -> 0 containers at a time

w1–w3: amd64 · w4–w5: arm64 · Hadoop in /opt/hadoop-3.5.0

# w1
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
export HADOOP_HOME=/opt/hadoop-3.5.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
# w2
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
export HADOOP_HOME=/opt/hadoop-3.5.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
# w3
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
export HADOOP_HOME=/opt/hadoop-3.5.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
# w4
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-arm64
export HADOOP_HOME=/opt/hadoop-3.5.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
# w5
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-arm64
export HADOOP_HOME=/opt/hadoop-3.5.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Check yourself

Quick revision

Interview and viva questions

Practice problems

Small, practical exercises around versions, paths, memory and checks. Each can be run in the browser.

/usr/lib/jvm/java-17-openjdk-amd64/bin/java

/usr/lib/jvm/java-17-openjdk-amd64

p = "/usr/lib/jvm/java-17-openjdk-amd64/bin/java"
print(p.removesuffix("/bin/java"))

openjdk version "17.0.20.1" 2026-08-18

17

line = 'openjdk version "17.0.20.1" 2026-08-18'
print(int(line.split('"')[1].split(".")[0]))

"3.4" · "3.5"

3.4 -> [8, 11]
3.5 -> [17]

def java_for(hadoop):
    return {"3.3": [8, 11], "3.4": [8, 11], "3.5": [17]}[hadoop]
print("3.4 ->", java_for("3.4"))
print("3.5 ->", java_for("3.5"))

Mem: 7.8Gi 512Mi 6.9Gi 2.0Mi 418Mi 7.1Gi

total 7.8Gi available 7.1Gi

cols = "Mem: 7.8Gi 512Mi 6.9Gi 2.0Mi 418Mi 7.1Gi".split()
print("total", cols[1], "available", cols[6])

export JAVA_HOME=… · alias ll=…

JAVA_HOME set
HADOOP_HOME MISSING
PATH MISSING

bashrc = ["alias ll='ls -l'",
          "export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64"]
for var in ["JAVA_HOME", "HADOOP_HOME", "PATH"]:
    found = any(l.startswith(f"export {var}=") for l in bashrc)
    print(var, "set" if found else "MISSING")

19 GB free · 0.6 + 1.3 + 10 GB

need 11.9 GB, enough: True, spare 7.1 GB

need = 0.6 + 1.3 + 10
print(f"need {need:.1f} GB, enough: {19 >= need}, spare {19 - need:.1f} GB")

45 min/node by hand · 2 h + 5 min/node automated · 40 nodes

by hand 30.0 h, automated 5.3 h, saved 24.7 h

by_hand = 40 * 45 / 60
auto = 2 + 40 * 5 / 60
print(f"by hand {by_hand:.1f} h, automated {auto:.1f} h, saved {by_hand - auto:.1f} h")

nn1: 17 · dn1: 11 · dn2: 17 · edge1: 21

nn1    server Java 17: OK
dn1    server Java 11: UPGRADE JAVA
dn2    server Java 17: OK
edge1  client Java 21: OK

nodes = {"nn1": ("server", 17), "dn1": ("server", 11),
         "dn2": ("server", 17), "edge1": ("client", 21)}
for name, (role, java) in nodes.items():
    ok = java == 17 if role == "server" else java in (17, 21)
    print(f"{name:6} {role:6} Java {java}: {'OK' if ok else 'UPGRADE JAVA'}")

Real-world scenario questions (asked in company interviews)

Setup problems are a favourite in support and admin interviews: the interviewer shows an error and asks what you would check.