Standalone, pseudo-distributed and fully distributed modes

One download, three ways to run it

Hadoop's three run modes

Words you'll meet in this lesson

The settings that decide the mode (Hadoop 3.5.0)

Modes at three levels

The three modes side by side

Ways people get each mode

Run the same job in all three modes

Every output below is from a real run of Hadoop 3.5.0 on Ubuntu 24.04 with OpenJDK 17. Full installation commands are in Lesson 4.3; here we focus on what changes between the modes.

1. Standalone: run an example job on local files

   cd ~/hadoop-3.5.0
   mkdir input && cp etc/hadoop/*.xml input
   hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.5.0.jar \
       grep input output 'dfs[a-z.]+'
   cat output/*
   ls output

   1 dfsadmin _SUCCESS part-r-00000
   The grep example counts words matching dfs[a-z.]+ in Hadoop's own XML files. input and output are ordinary Linux folders.

2. Standalone: prove there is no cluster

   jps
   hdfs getconf -confKey fs.defaultFS
   hdfs getconf -confKey mapreduce.framework.name
   hadoop fs -ls /

   9241 Jps file:/// local drwxr-xr-x - root root 4096 2026-10-21 05:12 /bin drwxr-xr-x - root root 4096 2026-10-21 05:12 /etc drwxr-xr-x - root root 4096 2026-10-21 05:12 /home …
   No daemons. The default file system is the local disk, so hadoop fs -ls / shows the Linux root folder, owned by root, not an HDFS folder.

3. Pseudo-distributed: change the settings

   <configuration>
       <property>
           <name>fs.defaultFS</name>
           <value>hdfs://localhost:9000</value>
       </property>
   </configuration>

   <configuration>
       <property>
           <name>dfs.replication</name>
           <value>1</value>
       </property>
   </configuration>

   These two files are exactly what the official Apache single-node guide uses. mapred-site.xml and yarn-site.xml (to run jobs on YARN) are shown in Lesson 4.3 and explained in Lesson 4.4.

4. Pseudo-distributed: format and start

   hdfs namenode -format
   start-dfs.sh
   start-yarn.sh
   jps

   Storage directory /tmp/hadoop-riya/dfs/name has been successfully formatted. Starting namenodes on [localhost] Starting datanodes Starting secondary namenodes [vm] Starting resourcemanager Starting nodemanagers 5584 SecondaryNameNode 5283 NameNode 6567 Jps 6120 ResourceManager 5384 DataNode 6221 NodeManager
   Five daemons, five JVMs, one machine (here named vm). The numbers are process IDs and will differ on your machine.

5. Pseudo-distributed: same job, now on HDFS and YARN

   hdfs getconf -confKey fs.defaultFS
   hdfs getconf -confKey dfs.replication
   hdfs dfs -mkdir -p /user/riya/input
   hdfs dfs -put etc/hadoop/*.xml input
   hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.5.0.jar \
       grep input output 'dfs[a-z.]+'
   hdfs dfs -cat output/*

   hdfs://localhost:9000 1 … INFO mapreduce.Job: Job job_1792559666229_0001 completed successfully … 1 dfsadmin 1 dfs.replication
   The job now runs through YARN (it has a job ID). The result has a second line, dfs.replication, because the hdfs-site.xml you just edited is one of the input files: the data changed, not the program.

6. Fully distributed: many nodes

   docker compose up -d --scale datanode=3
   docker compose exec namenode hdfs dfsadmin -printTopology
   docker compose exec namenode hdfs getconf -confKey dfs.replication

   Rack: /default-rack 172.18.0.4:9866 (mpcluster-datanode-1.mpcluster_default) In Service 172.18.0.6:9866 (mpcluster-datanode-3.mpcluster_default) In Service 172.18.0.7:9866 (mpcluster-datanode-2.mpcluster_default) In Service 3
   Three DataNodes, each a separate container with its own IP address, and replication back to 3. On real servers the only difference is that each DataNode is a separate computer, listed in the workers file.

From one process to a cluster

On the left, the real terminal output at each step. On the right, the processes and machines behind it.

Try it yourself: from one node to many

The playground starts as a one-node cluster, like pseudo-distributed mode, with replication 1. Break it, then grow it and see what changes.

Which mode, where?

Simplified; organisations differ in tools and policies.

Work out the mode from the settings

The mode is decided by a few configuration values. This function reads them (here typed in as dictionaries, exactly as hdfs getconf printed them on our three test machines) and names the mode.

def mode(conf):
    fs = conf.get("fs.defaultFS", "file:///")
    if fs.startswith("file:"):
        return "standalone"
    host = fs.split("//")[1].split(":")[0]
    if host in ("localhost", "127.0.0.1"):
        return "pseudo-distributed"
    return "fully distributed (NameNode on " + host + ")"

machines = {
    "fresh unpack": {},
    "Riya's WSL": {"fs.defaultFS": "hdfs://localhost:9000", "dfs.replication": "1"},
    "Docker cluster": {"fs.defaultFS": "hdfs://namenode", "dfs.replication": "3"},
}
for name, conf in machines.items():
    print(f"{name:15} -> {mode(conf)}")

• fs = conf.get("fs.defaultFS", "file:///")If the setting is missing, Hadoop uses its default: file:///, the local disk. That is why a fresh unpack is standalone.
• host = fs.split("//")[1].split(":")[0]hdfs://localhost:9000 → after // is localhost:9000 → before : is localhost.
• "Docker cluster": {"fs.defaultFS": "hdfs://namenode", "dfs.replication": "3"},No port means the default NameNode RPC port, 8020.

Run it on your machine: which mode am I in?

This script asks Hadoop for its settings and lists the running daemons, then names the mode. Save it as which_mode.sh (Hadoop's bin must be on your PATH). The three outputs below are from real runs.

#!/usr/bin/env bash
# which_mode.sh: which Hadoop mode is this machine in?
fs=$(hdfs getconf -confKey fs.defaultFS 2>/dev/null)
fw=$(hdfs getconf -confKey mapreduce.framework.name 2>/dev/null)
daemons=$(jps | awk '$2 != "Jps" {print $2}' | sort | tr '\n' ' ')
echo "fs.defaultFS             : $fs"
echo "mapreduce.framework.name : $fw"
echo "daemons on this machine  : ${daemons:-none}"
case "$fs" in
  file:*)                              echo "MODE: standalone (local files, no daemons)" ;;
  hdfs://localhost*|hdfs://127.0.0.1*) echo "MODE: pseudo-distributed (every daemon on this machine)" ;;
  hdfs://*)                            echo "MODE: fully distributed (NameNode at ${fs#hdfs://})" ;;
esac

# In the terminal:
#   bash which_mode.sh

start-dfs.sh
start-yarn.sh
bash which_mode.sh

docker cp which_mode.sh mpcluster-namenode-1:/tmp/
docker exec mpcluster-namenode-1 bash /tmp/which_mode.sh
docker cp which_mode.sh mpcluster-datanode-1:/tmp/
docker exec mpcluster-datanode-1 bash /tmp/which_mode.sh

Deciding the mode, one line at a time

Follow Riya's pseudo-distributed setting through the function and watch how the host name is cut out of the address.

Worked examples: real problems, solved and traced in memory

Three questions people really answer when choosing or checking a mode: what fits on a laptop, what a one-node cluster risks, and how to check that every node of a real cluster has the same configuration. Each runs live and is traced in memory.

5 daemons × 1 GB · OS 2 GB · 1 GB per container · 8 GB and 16 GB

8 GB laptop: 1 GB free -> 1 container(s) at once
16 GB laptop: 9 GB free -> 9 container(s) at once

2% yearly disk failure · one disk vs three copies · repair within 1 day

replication 1: 2.00% chance of loss per year
replication 3: 0.00000002% chance of loss per year
about 111,020,833 times safer

nn1, dn1, dn3: hdfs://nn1:8020, 3 · dn2: hdfs://localhost:9000, 1

expected: ('hdfs://nn1:8020', '3') (3 of 4 nodes)
MISMATCH on dn2 -> ('hdfs://localhost:9000', '1')

Check yourself

Quick revision

Interview and viva questions

Practice problems

Short exercises on reading settings, counting daemons and checking configurations, all runnable in the browser.

file:/// · hdfs://localhost:9000 · hdfs://namenode

file:/// -> local files
hdfs://localhost:9000 -> HDFS
hdfs://namenode -> HDFS

for fs in ["file:///", "hdfs://localhost:9000", "hdfs://namenode"]:
    print(fs, "->", "local files" if fs.startswith("file:") else "HDFS")

hdfs://localhost:9000 · hdfs://namenode

hdfs://localhost:9000 -> port 9000
hdfs://namenode -> port 8020

for fs in ["hdfs://localhost:9000", "hdfs://namenode"]:
    hostport = fs.split("//")[1]
    port = hostport.split(":")[1] if ":" in hostport else "8020"
    print(fs, "-> port", port)

5584 SecondaryNameNode · 5283 NameNode · 6567 Jps · 6120 ResourceManager · 5384 DataNode · 6221 NodeManager

5 daemons: DataNode, NameNode, NodeManager, ResourceManager, SecondaryNameNode

jps = """5584 SecondaryNameNode
5283 NameNode
6567 Jps
6120 ResourceManager
5384 DataNode
6221 NodeManager"""
names = sorted(line.split()[1] for line in jps.splitlines() if not line.endswith(" Jps"))
print(len(names), "daemons:", ", ".join(names))

{} · {NameNode, DataNode, …} · {DataNode, NodeManager}

[] -> standalone
['DataNode', 'NameNode', 'ResourceManager'] -> pseudo-distributed
['DataNode', 'NodeManager'] -> node of a fully distributed cluster

def guess(daemons):
    if not daemons:
        return "standalone"
    if {"NameNode", "DataNode"} <= daemons:
        return "pseudo-distributed"
    return "node of a fully distributed cluster"
for d in [set(), {"NameNode", "DataNode", "ResourceManager"}, {"DataNode", "NodeManager"}]:
    print(sorted(d), "->", guess(d))

4 TB · replication 1 and 3 · 25% free

replication 1: 5.3 TB raw disk
replication 3: 16.0 TB raw disk

data = 4
for repl in [1, 3]:
    raw = data * repl / 0.75
    print(f"replication {repl}: {raw:.1f} TB raw disk")

fs.defaultFS=hdfs://localhost:9000 · dfs.replication=1 · workers=localhost

fs.defaultFS    : hdfs://localhost:9000  ->  hdfs://nn1:8020
dfs.replication : 1  ->  3
workers         : ['localhost']  ->  ['worker1', 'worker2', 'worker3']

pseudo = {"fs.defaultFS": "hdfs://localhost:9000", "dfs.replication": "1", "workers": ["localhost"]}
fully = dict(pseudo)
fully["fs.defaultFS"] = "hdfs://nn1:8020"
fully["dfs.replication"] = "3"
fully["workers"] = ["worker1", "worker2", "worker3"]
for k in pseudo:
    print(f"{k:16}: {pseudo[k]}  ->  {fully[k]}")

3 failed attempts · 15 min on cluster · 1 min local · 1 final cluster run

cluster only 60 min, local first 18 min, saved 42 min

cluster_only = 4 * 15
local_first = 3 * 1 + 15
print(f"cluster only {cluster_only} min, local first {local_first} min, saved {cluster_only - local_first} min")

w1: hdfs://nn1:8020 · w2: hdfs://localhost:9000 · w3: hdfs://nn1:8020 · w4: file:///

will not join: ['w2', 'w4']

expected = "hdfs://nn1:8020"
workers = {"w1": "hdfs://nn1:8020", "w2": "hdfs://localhost:9000",
           "w3": "hdfs://nn1:8020", "w4": "file:///"}
bad = [w for w, fs in workers.items() if fs != expected]
print("will not join:", bad)

Real-world scenario questions (asked in company interviews)

Interviewers use the modes to check whether you understand what a cluster really is, not just the names.