Safe mode

“Name node is in safe mode”

Safe mode

Words you'll meet in this lesson

Safe mode settings (Hadoop 3.5.0 defaults)

Safe mode at three levels

Ways into and out of safe mode

The four safemode commands

Restart the NameNode and watch safe mode

On the Module 5 lab cluster (8 blocks, 4 DataNodes). The NameNode's own explanation is read from its /jmx “Safemode” field.

1. Read the settings

   for k in dfs.namenode.safemode.threshold-pct dfs.namenode.safemode.extension \
            dfs.namenode.safemode.min.datanodes dfs.namenode.replication.min; do
     echo "$k = $(docker compose exec namenode hdfs getconf -confKey $k)"
   done

   dfs.namenode.safemode.threshold-pct = 0.999f dfs.namenode.safemode.extension = 30000 dfs.namenode.safemode.min.datanodes = 0 dfs.namenode.replication.min = 1
   The f in 0.999f just means a floating-point number in the defaults file.

2. Restart the NameNode with all DataNodes up

   docker restart lab5-namenode-1
   # every few seconds:
   docker compose exec namenode hdfs dfsadmin -safemode get

   t=5s Call From namenode/172.19.0.3 to namenode:8020 failed on connection exception: java.net.ConnectException: Connection refused t=10s Safe mode is ON | The reported blocks 8 has reached the threshold 0.9990 of total blocks 8. … In safe mode extension. Safe mode will be turned off automatically in 28 seconds. t=15s Safe mode is ON | … Safe mode will be turned off automatically in 23 seconds. t=28s Safe mode is ON | … Safe mode will be turned off automatically in 10 seconds. t=38s Safe mode is ON | … Safe mode will be turned off automatically in 0 seconds. t=43s Safe mode is OFF
   For the first seconds the NameNode isn't even listening yet (loading fsimage and edits). Then the four DataNodes report within moments, the threshold is reached, and the 30 s extension counts down.

3. Try to write during safe mode

   docker compose exec namenode hdfs dfs -put /tmp/canteen_orders.csv data/during_safemode.csv

   put: Cannot create file/user/hadoop/data/during_safemode.csv._COPYING_. Name node is in safe mode.
   Writes are refused (SafeModeException). put first writes to a temporary ._COPYING_ name, which is what it tried to create.

4. Restart with no DataNodes at all

   docker stop lab5-datanode-1 lab5-datanode-2 lab5-datanode-3 lab5-datanode-4
   docker restart lab5-namenode-1
   docker compose exec namenode hdfs dfsadmin -safemode get
   docker compose exec namenode hdfs dfs -ls data
   docker compose exec namenode hdfs dfs -cat data/canteen_orders.csv

   Safe mode is ON NameNode says: The minimum number of live datanodes is not required. Safe mode will be turned off automatically once the thresholds have been reached. Found 2 items -rw-r--r-- 3 hadoop supergroup 4692009 2026-10-21 06:49 data/canteen_1mb.csv -rw-r--r-- 3 hadoop supergroup 4692009 2026-10-21 06:49 data/canteen_orders.csv cat: Zero blocklocations for /user/hadoop/data/canteen_orders.csv. Name node is in safe mode.
   The namespace is all there (from the fsimage), so ls works. But no block has a known location, so the data can't be read. The threshold can't be reached, so it waits.

5. Bring the DataNodes back

   docker start lab5-datanode-1 lab5-datanode-2 lab5-datanode-3 lab5-datanode-4
   docker compose logs namenode | grep "STATE\*"

   t=13s Safe mode is ON | … Safe mode will be turned off automatically once the thresholds have been reached. t=19s Safe mode is ON | The reported blocks 8 has reached the threshold 0.9990 of total blocks 8. … turned off automatically in 24 seconds. t=37s Safe mode is ON | … turned off automatically in 5 seconds. t=49s Safe mode is OFF INFO StateChange:406 - STATE* Safe mode is OFF INFO StateChange:411 - STATE* Leaving safe mode after 70 secs
   As soon as the DataNodes register and report, the threshold is met, the extension runs, and the NameNode leaves safe mode on its own: 70 s after its start in total.

6. A missing block: what safe mode does and doesn't catch

   docker compose exec namenode hdfs dfs -D dfs.replication=1 -put /tmp/canteen_orders.csv data/scratch_r1.csv
   # its only replica is on lab5-datanode-1: stop it, restart the NameNode
   docker stop lab5-datanode-1
   docker restart lab5-namenode-1
   docker compose logs namenode | grep -A1 "Safe mode ON"
   docker compose exec namenode hdfs fsck / | grep -E "MISSING|Missing blocks|Status"

   STATE* Safe mode ON, in safe mode extension. The reported blocks 8 has reached the threshold 0.9990 of total blocks 9. … /user/hadoop/data/scratch_r1.csv: MISSING 1 blocks of total size 4692009 B. Status: CORRUPT Missing blocks: 1
   The threshold is a whole number: int(9 × 0.999) = 8, so 8 of 9 blocks was “enough” and the NameNode left safe mode with a block missing. fsck reports it. Starting datanode-1 again made the file HEALTHY. On a big cluster the threshold is tighter (1,000 missing blocks out of 10 lakh), but safe mode is not a data-integrity check: fsck is.

A NameNode restart with no DataNodes, then four

Step 4 and step 5 from the real lab, frame by frame.

Safe mode at work

Simplified; organisations differ in tools and policies.

Will the NameNode leave safe mode?

This function applies the safe mode rules: the needed count is the whole-number part of total blocks × threshold, plus the minimum number of live DataNodes. Run it on the real cases from Part 4.

def safe_mode(total_blocks, reported, live_dns, threshold=0.999, min_dns=0):
    needed = int(total_blocks * threshold)
    blocks_ok = reported >= needed
    dns_ok = live_dns >= min_dns
    if blocks_ok and dns_ok:
        return f"reported {reported}/{total_blocks} (needed {needed}): leaves after 30 s extension"
    return f"reported {reported}/{total_blocks} (needed {needed}): STAYS in safe mode"

print(safe_mode(8, 8, 4))       # restart, all DataNodes up
print(safe_mode(8, 0, 0))       # restart, no DataNodes
print(safe_mode(9, 8, 3))       # replication-1 file on a stopped DataNode
print(safe_mode(10_00_000, 998_000, 400))

• needed = int(total_blocks * threshold)int() keeps only the whole-number part: int(9 × 0.999) = int(8.991) = 8. That is why 8 of 9 was enough in our lab.
• dns_ok = live_dns >= min_dnsdfs.namenode.safemode.min.datanodes is 0 by default, so this never blocks unless an admin sets it.

Run it on your PC: follow a NameNode restart

This script restarts the lab NameNode, then every few seconds asks for the safe mode state and reads the NameNode's own explanation from /jmx. Save it as safemode_watch.py in the hdfs-lab folder and run it while the DataNodes are up.

# safemode_watch.py: restart the NameNode and follow safe mode until it ends.
# Run in the hdfs-lab folder while the cluster is up.
import json
import re
import subprocess
import time

URL = "http://namenode:9870/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo"

def nn(*cmd):
    return subprocess.run(["docker", "compose", "exec", "-T", "namenode", *cmd],
                          capture_output=True, text=True).stdout.strip()

def explanation():
    try:
        return json.loads(nn("curl", "-s", URL))["beans"][0]["Safemode"] or "(off)"
    except (ValueError, KeyError, IndexError):
        return "(NameNode not answering yet)"

subprocess.run(["docker", "restart", "lab5-namenode-1"], capture_output=True)
start = time.time()
while time.time() - start < 120:
    state = nn("hdfs", "dfsadmin", "-safemode", "get").splitlines()
    state = state[-1] if state else "(no answer)"
    why = explanation()
    blocks = re.search(r"reported blocks (\d+).*?total blocks (\d+)", why)
    left = re.search(r"in (\d+) seconds", why)
    detail = (f"reported {blocks.group(1)}/{blocks.group(2)} blocks" if blocks else why[:40]) + \
             (f", off in {left.group(1)} s" if left else "")
    print(f"t={time.time() - start:3.0f}s  {state:24} {detail}")
    if "OFF" in state:
        break
    time.sleep(5)

# In the hdfs-lab folder:
#   python3 safemode_watch.py

The NameNode counting block reports

Follow the safe-block count as three DataNodes report, and see the moment the threshold is reached.

Worked examples: real problems, solved and traced in memory

Three safe mode questions that come up in operations: how long a restart will keep the cluster read-only, how much missing data the threshold tolerates, and when forcing safe mode off is reasonable. Each runs live and is traced in memory.

load 6 min · threshold at 3 min 50 s after reports start · extension 30 s

writes allowed after 10 min 20 s

2,00,00,000 blocks · 100 MB each · 0.999

up to 20,000 blocks, about 1.9 TB

case A: 1,000,000 / 990,000, 400 expected / 380 up, not lost · case B: same, nodes lost

A: investigate: 20 DataNodes not registered yet
B: leave safe mode, then run fsck and restore/regenerate missing data

Check yourself

Quick revision

Interview and viva questions

Practice problems

Safe mode exercises built on the real restarts from the lab.

8 · 9 · 1000

8 -> 7
9 -> 8
1000 -> 999

for t in [8, 9, 1000]:
    print(t, "->", int(t * 0.999))

07:27:01 + 9 s

07:27:10

from datetime import datetime, timedelta
print((datetime(2026, 10, 21, 7, 27, 1) + timedelta(seconds=9)).time())

70 · 25 · 30

15 s

print(70 - 25 - 30, "s")

The reported blocks 8 has reached the threshold 0.9990 of total blocks 8. … in 24 seconds.

8/8 reported, 24 s left

import re
msg = ("The reported blocks 8 has reached the threshold 0.9990 of total blocks 8. "
       "The minimum number of live datanodes is not required. In safe mode extension. "
       "Safe mode will be turned off automatically in 24 seconds.")
r, t = re.search(r"reported blocks (\d+).*total blocks (\d+)", msg).groups()
left = re.search(r"in (\d+) seconds", msg).group(1)
print(f"{r}/{t} reported, {left} s left")

600,000 blocks · 3 replicas · 300 nodes

threshold reached after 270 of 300 DataNodes

import random
random.seed(1)
nodes = 300
blocks = [random.sample(range(nodes), 3) for _ in range(60_000)]   # 1/10 sample
order = random.sample(range(nodes), nodes)
reported = set()
for i, n in enumerate(order, 1):
    reported.add(n)
    safe = sum(1 for b in blocks if reported.intersection(b))
    if safe >= int(len(blocks) * 0.999):
        print("threshold reached after", i, "of", nodes, "DataNodes")
        break

restart · enter · disk low

after restart          -> ends by itself
after -safemode enter  -> needs -safemode leave
metadata disk low      -> needs -safemode leave

cases = {"after restart": True, "after -safemode enter": False, "metadata disk low": False}
for c, auto in cases.items():
    print(f"{c:22} -> {'ends by itself' if auto else 'needs -safemode leave'}")

12 vs 4 min · 40 jobs · Rs 300/min after 5 min

12 min safe mode -> Rs 84,000
 4 min safe mode -> Rs 0

for minutes in [12, 4]:
    late = max(0, minutes - 5)
    print(f"{minutes:>2} min safe mode -> Rs {late * 300 * 40:,}")

file → blocks · reported replicas

orders.csv ok
scratch_r1.csv MISSING b9
users.csv ok

files = {"orders.csv": ["b1", "b2"], "scratch_r1.csv": ["b9"], "users.csv": ["b5"]}
reported = {"b1": ["dn2"], "b2": ["dn3", "dn4"], "b5": ["dn2", "dn4"], "b9": []}
for f, blocks in files.items():
    missing = [b for b in blocks if not reported.get(b)]
    print(f, "MISSING " + ", ".join(missing) if missing else "ok")

Real-world scenario questions (asked in company interviews)

Safe mode questions check both knowledge and judgement: do you force it, or do you find out why?