Admin commands: dfsadmin, fsck and balancer

Retire one server, welcome another

HDFS administration tools

Words you'll meet in this lesson

Defaults and limits (Hadoop 3.5.0, from the lab)

Admin tools at three levels

The commands you will actually use

Ways to take a DataNode out

A morning of real admin work

All commands ran in the namenode container of the lab: 5 DataNodes, two racks (/rack1: datanode-1, -2; /rack2: datanode-3, -4, -5).

1. Morning check: dfsadmin -report

   hdfs dfsadmin -report -dead

   Configured Capacity: 1352765870080 (1.23 TB) Present Capacity: 92735646762 (86.37 GB) DFS Remaining: 89697912832 (83.54 GB) DFS Used: 3037733930 (2.83 GB) DFS Used%: 3.28% Replicated Blocks: Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 ... ------------------------------------------------- Dead datanodes (0):
   The three numbers to read first: dead datanodes, missing blocks, under-replicated blocks. Capacity looks odd here (1.23 TB configured, 86 GB present) because all five lab DataNodes sit on one shared laptop disk; on real servers each node has its own disks.

2. Health check: fsck

   hdfs fsck / -list-corruptfileblocks
   hdfs fsck /user/hadoop/canteen -files -blocks -racks

   The filesystem under path '/' has 0 CORRUPT blocks /user/hadoop/canteen/raw/day1.csv 47 bytes, replicated: replication=3, 1 block(s): OK 0. BP-1507681208-172.19.0.3-1792565333483:blk_1073741852_1029 len=47 Live_repl=3 [/rack1/172.19.0.3:9866, /rack1/172.19.0.4:9866, /rack2/172.19.0.6:9866] /user/hadoop/canteen/raw/day2.csv 34 bytes, replicated: replication=3, 1 block(s): OK 0. BP-1507681208-172.19.0.3-1792565333483:blk_1073741855_1033 len=34 Live_repl=3 [/rack2/172.19.0.6:9866, /rack1/172.19.0.4:9866, /rack1/172.19.0.3:9866] Status: HEALTHY
   fsck asks the NameNode only; it reads no data. It shows each block's live replicas with their racks: two in one rack and one in the other, exactly the placement rule from Lesson 5.3.

3. Who is where: topology and node info

   hdfs dfsadmin -printTopology
   hdfs dfsadmin -getDatanodeInfo lab5-datanode-1:9867

   Rack: /rack1 172.19.0.4:9866 (lab5-datanode-1.lab5_default) In Service 172.19.0.3:9866 (lab5-datanode-2.lab5_default) In Service Rack: /rack2 172.19.0.8:9866 (lab5-datanode-5.lab5_default) In Service 172.19.0.7:9866 (lab5-datanode-4.lab5_default) In Service 172.19.0.6:9866 (lab5-datanode-3.lab5_default) In Service Uptime: 945, Software version: 3.5.0, Config version: core-3.0.0,hdfs-1
   9867 is the DataNode's IPC port (9866 is its data port). Uptime is in seconds.

4. Decommission datanode-4

   # once: hdfs-site.xml of the NameNode has
   #   dfs.hosts.exclude = /opt/hadoop/etc/hadoop/dfs.exclude
   echo 172.19.0.7 > /opt/hadoop/etc/hadoop/dfs.exclude
   hdfs dfsadmin -refreshNodes
   hdfs dfsadmin -report | grep -A3 'Name: 172.19.0.7'     # repeated every few seconds

   Refresh nodes successful t=2s Decommission Status : Decommission in progress t=8s Decommission Status : Decommission in progress ... t=46s Decommission Status : Decommission in progress t=52s Decommission Status : Decommissioned
   During decommission the node still serves reads, so nothing breaks. The NameNode copies each of its blocks from any live replica to another node, checking progress every 30 seconds. Then it is safe to switch it off.

5. Prove it, then bring it back

   hdfs fsck / | grep -E 'Status|Under-replicated|Number of data-nodes'
   > /opt/hadoop/etc/hadoop/dfs.exclude          # empty the exclude file
   hdfs dfsadmin -refreshNodes

   Status: HEALTHY Number of data-nodes: 4 Under-replicated blocks: 0 (0.0 %) Refresh nodes successful Decommission Status : Normal
   “4 data-nodes” and 0 under-replicated: every block had 3 replicas without datanode-4. After recommissioning, its replicas became extra copies (4 instead of 3), and the NameNode deleted excess ones. Datanode-4 had 11 blocks before; a minute after coming back it had 0. A recommissioned node often returns nearly empty.

6. A new, empty node and the balancer

   docker run -d --name lab5-datanode-6 ... --tmpfs /tmp/hadoop-hadoop:size=1536m apache/hadoop:3.5.0 hdfs datanode
   hdfs balancer -threshold 5

   INFO Balancer:568 - 1 underutilized: [172.19.0.9:9866:DISK] INFO Balancer:810 - Need to move 1.32 GB to make the cluster balanced. INFO Balancer:663 - Decided to move 340.33 MB bytes from 172.19.0.7:9866:DISK to 172.19.0.9:9866:DISK INFO Balancer:663 - Decided to move 340.38 MB bytes from 172.19.0.4:9866:DISK to 172.19.0.9:9866:DISK ... INFO Dispatcher:407 - Successfully moved blk_1073741844_1020 with size=134217728 from 172.19.0.4:9866:DISK to 172.19.0.9:9866:DISK through 172.19.0.6:9866 ... INFO Dispatcher:1289 - Total bytes (blocks) moved in this iteration 914.72 MB (8)
   Datanode-6 got its own 1.5 GB disk, so it was the only empty one (the other five share one disk and look equally full). The balancer planned moves, then copied 8 blocks in 16 seconds. “through 172.19.0.6”: the bytes came from a replica in the target's own rack, not necessarily from the node being emptied. Afterwards the source replica is deleted, so replication stays 3.

7. When the balancer gives up

   # same run, iterations 1 to 5

   Need to move 430.09 MB to make the cluster balanced. ... Out of space: The volume with the most available space (=643919872 B) is less than the block size (=134217728 B). ... (the same in iterations 2, 3, 4 and 5) No block has been moved for 5 iterations. Exiting... Oct 21, 2026, 10:04:11 AM Balancing took 1.1124 minutes $ hdfs dfsadmin -report (DFS Used%) 172.19.0.3 0.27% 172.19.0.4 0.22% 172.19.0.6 0.27% 172.19.0.7 0.00% 172.19.0.8 0.00% 172.19.0.9 60.02%
   The tiny lab disk started refusing blocks with “Out of space”. A failed move is just skipped: the block stays where it was, with all its replicas. After 5 rounds without progress, the balancer stops. On a real cluster, check the target's disks and rerun.

8. Speed limits and mistakes

   hdfs dfsadmin -setBalancerBandwidth 52428800
   hdfs balancer -threshold 0.5
   hdfs dfsadmin -saveNamespace

   Balancer bandwidth is set to 52428800 Expecting a number in the range of [1.0, 100.0]: 0.5 saveNamespace: Safe mode should be turned ON in order to create namespace image.
   52428800 bytes = 50 MB/s per DataNode, applied at once. The threshold must be at least 1%. -saveNamespace is only allowed in safe mode, so the namespace doesn't change while it is written.

Decommission, then balance

Block A lives on dn-2, dn-4 and dn-5. We retire dn-4, then add an empty dn-6.

Admin tools at work

Simplified; organisations differ in tools and policies.

A balancer in 30 lines

This program does what hdfs balancer does, simplified: compute each node's utilisation (used ÷ capacity) and the cluster average, then move 128 MB blocks from the fullest node to the emptiest until every node is within the threshold of the average.

cap = {"dn-1": 4000, "dn-2": 4000, "dn-3": 4000, "dn-6": 4000}    # MB
used = {"dn-1": 3200, "dn-2": 3072, "dn-3": 2944, "dn-6": 0}
threshold = 10          # percent
block = 128             # MB

def util(n):
    return 100 * used[n] / cap[n]

def average():
    return 100 * sum(used.values()) / sum(cap.values())

moves = 0
while True:
    avg = average()
    src = max(used, key=util)
    dst = min(used, key=util)
    if util(src) - avg <= threshold and avg - util(dst) <= threshold:
        break
    used[src] -= block
    used[dst] += block
    moves += 1

print(f"average {average():.1f}%  moves {moves}  ({moves * block} MB)")
for n in cap:
    print(f"  {n}: {util(n):5.1f}%")

• return 100 * used[n] / cap[n]Utilisation is a percentage of capacity, not a block count. That is why a big new disk with 0 blocks is “underutilised”.
• if util(src) - avg <= threshold and avg - util(dst) <= threshold:Balanced = every node within ±threshold of the average. Here 10%, the default; the real tool accepts 1 to 100.
• used[src] -= blockA real move copies the block to the target and then deletes the source replica, so replication never drops.

Run it on your PC: retire a node, fill a new one

Two scripts for the hdfs-lab folder. The first decommissions a DataNode and prints its status every few seconds until it is Decommissioned, checks fsck and recommissions it. The second starts a sixth DataNode with its own small disk and runs the balancer. First add HDFS-SITE.XML_dfs.hosts.exclude=/opt/hadoop/etc/hadoop/dfs.exclude to the config file and recreate the namenode (docker compose up -d).

#!/usr/bin/env bash
# decommission_watch.sh: retire one DataNode safely and watch HDFS copy its blocks away.
# Needs dfs.hosts.exclude=/opt/hadoop/etc/hadoop/dfs.exclude in the NameNode's hdfs-site.xml.
# Run from the hdfs-lab folder:  bash decommission_watch.sh 172.19.0.4
IP=${1:?give the DataNode IP}
NN="docker exec lab5-namenode-1"
status() { $NN hdfs dfsadmin -report 2>/dev/null | grep -m1 -A20 "^Name: $IP:" |
           grep -E "Decommission Status|Num of Blocks" | sed 's/ : /: /' | paste -sd'|' | sed 's/|/,  /'; }
echo "before:  $(status)"
$NN bash -c "echo $IP > /opt/hadoop/etc/hadoop/dfs.exclude"
$NN hdfs dfsadmin -refreshNodes
start=$(date +%s)
while true; do
  s=$(status); echo "t=$(( $(date +%s) - start ))s  $s"
  [[ $s == *"Status: Decommissioned"* ]] && break
  sleep 5
done
$NN bash -c "hdfs fsck / 2>/dev/null | grep -E 'Status:|Under-replicated|Number of data-nodes'"
echo "### recommission"
$NN bash -c "> /opt/hadoop/etc/hadoop/dfs.exclude"
$NN hdfs dfsadmin -refreshNodes
sleep 10; echo "after:   $(status)"

# In the hdfs-lab folder (find the IP with: hdfs dfsadmin -printTopology):
#   bash decommission_watch.sh 172.19.0.7

#!/usr/bin/env bash
# add_node_and_balance.sh: add a new, empty DataNode with its own small disk, then run the balancer.
# Run from the hdfs-lab folder (compose project "lab5", network lab5_default).
NN="docker exec lab5-namenode-1"
show() {
  $NN hdfs dfsadmin -report 2>/dev/null |
    awk '/^Name:/{ip=$2; sub(":9866","",ip)} /^DFS Used%/{if(ip) printf "  %-12s used %6s\n", ip, $3; ip=""}'
}
echo "### 1. Start datanode-6 with a 1.5 GB disk of its own (a RAM disk)"
docker run -d --name lab5-datanode-6 --hostname datanode-6 --network lab5_default \
  --env-file config --tmpfs /tmp/hadoop-hadoop:size=1536m,mode=1777 \
  apache/hadoop:3.5.0 hdfs datanode > /dev/null
sleep 25; show
echo "### 2. Balancer, threshold 5%"
$NN hdfs balancer -threshold 5 2>&1 |
  grep -E "underutilized|Need to move|Decided to move|Successfully moved|Out of space|^[A-Z][a-z]{2} [0-9]+, |No block|Iteration#" |
  sed -E 's/^2026-[0-9-]+ ([0-9:]+) (INFO|WARN)  [A-Za-z]+:[0-9]+ - /\1 /; s/java.io.IOException: Got error.*(Out of space)/  ... \1/; s/:9866:DISK//g' | cut -c1-150
echo "### 3. After"
show

# In the hdfs-lab folder:
#   bash add_node_and_balance.sh
# Remove the extra node afterwards:  docker rm -f lab5-datanode-6

Deciding what a decommission must copy

For each block on the retiring node, count the replicas on other live nodes and decide whether a new copy is needed.

Worked examples: real problems, solved and traced in memory

Three admin calculations: how long a decommission takes, how much data a balancer must move, and how much faster it runs with more bandwidth. Each runs live and is traced in memory.

12 TB · 1 GB/s

about 3.4 hours

10 × 40 TB at 75% · 5 new × 40 TB · threshold 10%

average 50.0%, each new node needs 16.0 TB, total 80 TB

80 TB · 5 receivers · 100 or 300 MB/s

100 MB/s: 47 hours (1.9 days)
300 MB/s: 16 hours (0.6 days)

Check yourself

Quick revision

Interview and viva questions

Practice problems

Admin exercises using real reports, logs and timings from the lab.

Configured Capacity 1352765870080 · DFS Used 3037733930

0.22%

print(f"{100 * 3037733930 / 1352765870080:.2f}%")

2 8 14 20 28 34 40 46 52

in progress: 8  average gap: 6.25 s

t = [2, 8, 14, 20, 28, 34, 40, 46, 52]
gaps = [b - a for a, b in zip(t, t[1:])]
print("in progress:", len(t) - 1, " average gap:", sum(gaps) / len(gaps), "s")

52428800 · 104857600 bytes/s

52428800 = 50.0 MB/s
104857600 = 100.0 MB/s

for b in [52428800, 104857600]:
    print(b, "=", b / 1024 / 1024, "MB/s")

three log lines

274.71836280822754 MB {'172.19.0.4': 2, '172.19.0.7': 1}

from collections import Counter
lines = ["Successfully moved blk_1073741846_1022 with size=19627626 from 172.19.0.4 to 172.19.0.9",
         "Successfully moved blk_1073741860_1038 with size=134217728 from 172.19.0.7 to 172.19.0.9",
         "Successfully moved blk_1073741844_1020 with size=134217728 from 172.19.0.4 to 172.19.0.9"]
total, src = 0, Counter()
for l in lines:
    w = l.split()
    total += int(w[4].split("=")[1]); src[w[6]] += 1
print(total / 1024 / 1024, "MB", dict(src))

utilisations 92, 81, 70, 64, 12 %

dn-1 92 over
dn-2 81 over
dn-3 70 above
dn-4 64 above
dn-6 12 under
average 63.8

util = {"dn-1": 92, "dn-2": 81, "dn-3": 70, "dn-4": 64, "dn-6": 12}
avg, t = sum(util.values()) / len(util), 10
for n, u in util.items():
    kind = "over" if u > avg + t else "above" if u > avg else "under" if u < avg - t else "below"
    print(n, u, kind)
print("average", avg)

block → nodes

A live replicas left: 1 
B live replicas left: 1 
C live replicas left: 2

blocks = {"A": {"dn-2", "dn-4", "dn-5"}, "B": {"dn-2", "dn-5", "dn-3"}, "C": {"dn-1", "dn-2", "dn-3"}}
stop = {"dn-2", "dn-5"}
for b, nodes in blocks.items():
    left = nodes - stop
    print(b, "live replicas left:", len(left), "" if left else "<- UNAVAILABLE")

20 × Rs 4.5 lakh · 5% used

Rs 90 lakh of hardware, about Rs 85.5 lakh of it unused

cost = 20 * 4.5
print(f"Rs {cost:.0f} lakh of hardware, about Rs {cost * 0.95:.1f} lakh of it unused")

4 blocks · 2 racks · free space per node

A copy to dn-1 racks [1, 2]
B copy to dn-2 racks [1, 2]
D copy to dn-2 racks [1, 2]

rack = {"dn-1": 1, "dn-2": 1, "dn-3": 2, "dn-4": 2, "dn-5": 2}
free = {"dn-1": 50, "dn-2": 80, "dn-3": 30, "dn-5": 60}
blocks = {"A": ["dn-2", "dn-4", "dn-5"], "B": ["dn-4", "dn-1", "dn-3"], "D": ["dn-4", "dn-3", "dn-5"]}
for b, nodes in blocks.items():
    if "dn-4" not in nodes:
        continue
    keep = [n for n in nodes if n != "dn-4"]
    racks = {rack[n] for n in keep}
    cands = [n for n in free if n not in keep]
    if len(racks) == 1:                       # all others on one rack: need the other rack
        cands = [n for n in cands if rack[n] not in racks]
    target = max(cands, key=free.get)
    free[target] -= 1
    print(b, "copy to", target, "racks", sorted(racks | {rack[target]}))

Real-world scenario questions (asked in company interviews)

Admin questions separate people who have run a cluster from people who have only read about one.