Troubleshooting playbook

Four alerts, one morning

A troubleshooting playbook

Words you'll meet in this lesson

From our lab (Hadoop 3.5.0, four incidents reproduced)

Troubleshooting at three levels

The layered checklist

Architecture at a glance

Start at the top with the exact message, then walk down the layers in dependency order. A layer can only work if the ones above it work. Each box lists the first commands to run there.

Key commands for each layer

Symptom → likely cause → first command (real failures from this course)

Animated example: the 6 AM report that never came

Four incidents on our lab

Run with trouble_lab.sh on the Module 8 lab (in “Run it on your PC”). Each step creates one incident, shows the exact message, and undoes it.

1. A. Connection refused

   hdfs dfs -ls hdfs://namenode:9000/

   Call From namenode/172.22.0.2 to namenode:9000 failed on connection exception: java.net.ConnectException: Connection refused
   The machine answered, but nothing listens on port 9000. “Refused” means the host exists and the port is wrong or the daemon is down; “UnknownHost” would mean the name itself is wrong. Check hdfs getconf -confKey fs.defaultFS (8020 here) and jps.

2. B. Safe mode

   hdfs dfsadmin -safemode enter
   hdfs dfs -mkdir /incident_b
   hdfs dfs -cat /user/hadoop/data/menu.csv | head -1
   hdfs dfsadmin -safemode leave

   Safe mode is ON Cannot create directory /incident_b. Name node is in safe mode. chai,drink,15,A Safe mode is OFF
   Writes fail, reads work: HDFS is read-only. Before leaving safe mode by hand, find out why it's on (-safemode get, NameNode log): after a restart it waits for enough block reports, and forcing it off with missing DataNodes can expose missing blocks.

3. C. The only replica is on a stopped DataNode

   hdfs dfs -D dfs.replication=1 -put -f menu.csv /incident_c.csv
   hdfs fsck /incident_c.csv -files -blocks -locations
   docker exec lab8-worker2-1 hdfs --daemon stop datanode
   hdfs dfs -cat /incident_c.csv

   DatanodeInfoWithStorage[172.22.0.6 the only replica is on worker2; stopping its DataNode No live nodes contain current block the read gave up after 16 s DataNode back: chai,drink,15,A
   fsck … -locations showed the one replica's address. With replication 1 there is no second copy, so the client tries, waits and gives up. Starting the DataNode made the block available again: no data was lost, only unavailable. With replication 3 this read would have succeeded.

4. D. A job stuck in ACCEPTED on a half-empty cluster

   yarn application -list
   yarn application -status application_…_0019
   curl -s http://resourcemanager:8088/ws/v1/cluster/scheduler

   application_1792661456575_0018 RUNNING application_1792661456575_0019 ACCEPTED Diagnostics : … Application is added to the scheduler and is not yet activated. Queue's AM resource limit exceeded. queue default | AM limit 768 MB | AM used 512 MB
   yarn.scheduler.capacity.maximum-am-resource-percent is 0.1 by default: 10% of 6,144 MB ≈ 614 MB, which the scheduler rounds up to 768 MB. One 512 MB ApplicationMaster fits; a second would need 1,024 MB. The cluster had memory free, but the queue had no ApplicationMaster room.

5. Write it down

   Title       Nightly report stuck in ACCEPTED (incident D)
   Impact      report 48 min late, 1 queue, no data loss
   Timeline    01:00 submitted · 06:05 noticed · 06:09 diagnosed · 06:16 mitigated
   Root cause  queue AM limit (10%) filled by a long-running job
   Detection   no alert for "ACCEPTED too long"  ← gap
   Actions     alert on ACCEPTED > 15 min · AM percent 0.1 → 0.3 for reports · job timeout

   Blameless: it records what the system allowed, not who submitted the long job. The detection gap and the actions are the most valuable lines.

From message to layer to fix

Six steps through the four real incidents and the method that solved each one.

Playbooks at work

Simplified; organisations differ in tools and policies.

A symptom matcher for real error messages

The first step of every incident is reading the message. This matcher holds a few playbook rules (a pattern, the layer, the likely cause and the first command) and applies them to real messages from our lab.

import re
RULES = [
    (r"Connection refused",              "network", "wrong port or daemon down",       "getent hosts; jps"),
    (r"UnknownHostException",            "network", "name not resolvable",             "getent hosts <name>"),
    (r"in safe mode",                    "HDFS",    "NameNode is read-only",           "hdfs dfsadmin -safemode get"),
    (r"No live nodes contain",           "HDFS",    "every replica's DataNode is down", "hdfs fsck <path> -locations"),
    (r"AM resource limit exceeded",      "YARN",    "queue's ApplicationMaster limit",  "yarn application -status"),
    (r"Java heap space",                 "job",     "task heap too small",             "yarn logs (attempt)"),
    (r"cannot authenticate via",         "security","no Kerberos ticket",              "klist"),
]

def diagnose(message):
    for pattern, layer, cause, first in RULES:
        if re.search(pattern, message, re.IGNORECASE):
            return layer, cause, first
    return "unknown", "not in the playbook yet", "read the full log; add a rule"

messages = [
    "Call From namenode/172.22.0.2 to namenode:9000 failed on connection exception: java.net.ConnectException: Connection refused",
    "Cannot create directory /incident_b. Name node is in safe mode.",
    "No live nodes contain current block",
    "Application is added to the scheduler and is not yet activated. Queue's AM resource limit exceeded.",
    "Illegal partition for chai (4)",
]
for m in messages:
    layer, cause, first = diagnose(m)
    print(f"{layer:8} | {cause:32} | {first}")

• (r"AM resource limit exceeded", "YARN", "queue's ApplicationMaster limit", "yarn application -status"),Each rule is one playbook row: a clue in the message, the layer, the likely cause and the command that confirms it.
• return "unknown", "not in the playbook yet", "read the full log; add a rule"A message with no rule is a gap in the playbook. After the incident, add a rule.

Run it on your PC: four incidents on purpose

Run this on your PC in the mr-lab folder with the Module 8 lab running. It creates each incident, prints the exact message, and puts things back: it leaves safe mode, restarts the DataNode it stopped and kills the two sleep jobs. It takes about 3 minutes.

# trouble_lab.sh: four common incidents reproduced on the Module 8 lab, with the exact message each one gives.
# Run on your PC.
NN="docker exec lab8-namenode-1"
clean() { grep -vE "^\s+at |INFO|^$|WARN.*retry|Retrying" | cut -c1-170; }
echo "== A. 'Connection refused': wrong port in the URI (8020 is right, 9000 was the Hadoop 1/2 habit)"
$NN hdfs dfs -ls hdfs://namenode:9000/ 2>&1 | grep -m1 -oE "Call From .* failed on connection exception: java.net.ConnectException: Connection refused"
echo "== B. 'Name node is in safe mode': writes fail, reads still work"
$NN hdfs dfsadmin -safemode enter 2>/dev/null
$NN hdfs dfs -mkdir /incident_b 2>&1 | grep -m1 -oE "Cannot create directory /incident_b. Name node is in safe mode."
echo "   read still works: $($NN hdfs dfs -cat /user/hadoop/data/menu.csv 2>/dev/null | head -1)"
$NN hdfs dfsadmin -safemode leave 2>/dev/null
echo "== C. a file with ONE replica, and its DataNode goes down"
$NN bash -c 'hdfs dfs -D dfs.replication=1 -put -f /tmp/mr/menu.csv /incident_c.csv 2>/dev/null; hdfs fsck /incident_c.csv -files -blocks -locations 2>/dev/null | grep -oE "DatanodeInfoWithStorage\[[0-9.]+"'
ip=$($NN hdfs fsck /incident_c.csv -files -blocks -locations 2>/dev/null | grep -oE "DatanodeInfoWithStorage\[[0-9.]+" | grep -oE "[0-9.]+$" | head -1)
for w in worker1 worker2 worker3; do
  if docker exec lab8-$w-1 hostname -i | grep -q "$ip"; then victim=$w; fi
done
echo "   the only replica is on $victim; stopping its DataNode"
docker exec lab8-$victim-1 hdfs --daemon stop datanode
$NN bash -c 'start=$(date +%s); hdfs dfs -cat /incident_c.csv 2>&1 | grep -m2 -oE "Could not obtain block: [^ ]+|BlockMissingException|No live nodes contain current block"; echo "   the read gave up after $(( $(date +%s) - start )) s"'
docker exec lab8-$victim-1 hdfs --daemon start datanode; sleep 15
echo "   DataNode back: $($NN hdfs dfs -cat /incident_c.csv 2>/dev/null | head -1)"
$NN hdfs dfs -rm -skipTrash /incident_c.csv > /dev/null 2>&1
echo "== D. a job stuck in ACCEPTED"
T=/opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.5.0-tests.jar
for n in 1 2; do $NN bash -c "export HADOOP_CLASSPATH=$T; yarn org.apache.hadoop.mapreduce.SleepJob -libjars $T -m 30 -r 1 -mt 30000 -rt 1000 > /tmp/acc$n.log 2>&1" & sleep 3; done
sleep 25
docker exec lab8-resourcemanager-1 yarn application -list 2>/dev/null | grep -E "RUNNING|ACCEPTED" | awk -F'\t' '{print "  ", $1, $6}' | tr -s ' '
app=$(docker exec lab8-resourcemanager-1 yarn application -list -appStates ACCEPTED 2>/dev/null | grep -o "application_[0-9_]*" | head -1)
docker exec lab8-resourcemanager-1 yarn application -status $app 2>/dev/null | grep -A2 "Diagnostics" | sed 's/^\s*/   /' | cut -c1-200
echo "-- the queue's limit on ApplicationMasters:"
docker exec lab8-namenode-1 curl -s http://resourcemanager:8088/ws/v1/cluster/scheduler | python3 -c '
import sys, json
q = json.load(sys.stdin)["scheduler"]["schedulerInfo"]["queues"]["queue"][0]
r = q["resources"]["resourceUsagesByPartition"][0]
print("   queue", q["queueName"], "| AM limit", r["amLimit"]["memory"], "MB | AM used", r["amUsed"]["memory"], "MB | maxAMPercent", q.get("maxAMPercent", q.get("maximumAMResourcePercent")))' 2>&1 | head -2
for a in $(docker exec lab8-resourcemanager-1 yarn application -list 2>/dev/null | grep -o "application_[0-9_]*"); do docker exec lab8-resourcemanager-1 yarn application -kill $a > /dev/null 2>&1; done
wait

# Run on your PC, in the mr-lab folder:
#   bash trouble_lab.sh

Walking the layers in order

A checklist in dependency order: stop at the first layer that fails. Watch layer and ok.

Worked examples: real problems, solved and traced in memory

Four incidents from different industries, each diagnosed from its message.

message + scope: one user

same? False | fix the client config: hdfs://nn-records:8020 -> hdfs://nn-patients:8020

38/40 live · 99.7% reported · threshold 99.9%

150,000 blocks not reported yet; threshold needs 49,950,000

200 GB · AM 10% · 4 GB per AM

AM budget 20 GB -> 5 AMs; with 0.3: 15 AMs

one reducer · month end · 25× one key

normal reducer 0.8 GB, hot reducer 10.8 GB

Check yourself

Quick revision

Interview and viva questions

Practice problems

Diagnosis exercises using the real messages and numbers from our lab.

4 real messages

network  <- Connection refused
HDFS     <- Name node is in safe mode.
YARN     <- Queue's AM resource limit exceeded.
job      <- Error: Java heap space

msgs = {"Connection refused": "network", "Name node is in safe mode.": "HDFS",
        "Queue's AM resource limit exceeded.": "YARN", "Error: Java heap space": "job"}
for m, layer in msgs.items():
    print(f"{layer:8} <- {m}")

two times

296 s

from datetime import datetime
a, b = datetime.strptime("10:21:12", "%H:%M:%S"), datetime.strptime("10:26:08", "%H:%M:%S")
print((b - a).seconds, "s")

6,144 MB · 0.1 · 256

768 MB

import math
print(math.ceil(6144 * 0.1 / 256) * 256, "MB")

3 blocks · worker2 stopped

['blk_1']

blocks = {"blk_1": ["worker2"], "blk_2": ["worker1", "worker2", "worker3"], "blk_3": ["worker2", "worker3"]}
down = {"worker2"}
print([b for b, nodes in blocks.items() if set(nodes) <= down])

incident notes

+  0 min  report missing
+  4 min  AM limit found
+ 11 min  stuck job killed
+ 43 min  report delivered

notes = ["06:05 report missing", "checked dashboards", "06:09 AM limit found", "06:16 stuck job killed", "06:48 report delivered"]
t0 = None
for n in notes:
    first = n.split()[0]
    if ":" not in first:
        continue
    h, m = map(int, first.split(":"))
    t0 = t0 if t0 is not None else h * 60 + m
    print(f"+{h * 60 + m - t0:3d} min  {n[6:]}")

4 attempts, 4 nodes, 1 exception

job or data

attempts = [("worker1", "NumberFormatException"), ("worker2", "NumberFormatException"),
            ("worker3", "NumberFormatException"), ("worker1", "NumberFormatException")]
nodes = {n for n, _ in attempts}
errors = {e for _, e in attempts}
print("job or data" if len(errors) == 1 and len(nodes) > 1 else "look at the nodes")

12 min × 300 vs 5 min × 1

restart first: 3600 analyst-minutes lost; diagnose first: 5 engineer-minutes

print("restart first:", 12 * 300, "analyst-minutes lost; diagnose first:", 5, "engineer-minutes")

rules + message

subprocess failed -> streaming script crashed

import re
rules = {"Java heap space": "task heap too small", "subprocess failed": "streaming script crashed", "failed": "something failed"}
msg = "PipeMapRed.waitOutputThreads(): subprocess failed with code 1"
hits = [p for p in rules if re.search(p, msg)]
best = max(hits, key=len)
print(best, "->", rules[best])

Real-world scenario questions (asked in company interviews)

Troubleshooting questions test method more than memory. Interviewers want to hear how you think under pressure.