Monitoring and alerts

The cluster can't call you. Monitoring can.

Monitoring and alerting for Hadoop

Words you'll meet in this lesson

From our lab (two clusters, one monitor)

Monitoring at three levels

Signals, sources and severities

Architecture at a glance

Daemons publish, a collector scrapes, rules decide, people get told. The dashboard and the canary job sit beside the main path.

Where metrics come from

Animated example: Diwali sale night

Build a monitor, then break the cluster

Our monitor is about 60 lines of Python (monitor.py, in “Run it on your PC”). It runs on a lab8 worker that can reach both clusters: prod (the Module 8 lab) and dr (the federation lab from Lesson 7.2).

1. Read the metrics

   curl -s 'http://namenode:9870/jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState'
   curl -s http://resourcemanager:8088/ws/v1/cluster/metrics

   10:21:11 prod live=3 dead=0 stale=0 used=0.6 under=0 missing=0 files=1658 safemode=False nodes=3 unhealthy=0 lost=0 pending=0 running=0 mem=0 10:21:11 dr live=3 dead=0 stale=0 used=0.1 under=0 missing=0 files=15 safemode=False
   The baseline. Everything a rule needs is in two JSON documents per cluster: no agents, no extra software.

2. Stop a DataNode and wait for the alert

   docker exec lab8-worker2-1 hdfs --daemon stop datanode

   (the alert appeared 296 s after the DataNode stopped) 10:26:08 prod live=3 dead=0 stale=1 used=0.6 under=0 ... 10:26:08 ALERT WARN prod: 1 stale DataNode(s) (no heartbeat for 30 s)
   Stale after 30 s, but the stale count is refreshed on the NameNode's heartbeat check (every 5 minutes by default), so our alert came almost 5 minutes late. Note that live still says 3: the node is declared dead only after 10 min 30 s.

3. Ask for more replicas than the cluster can give

   hdfs dfs -setrep 4 /user/hadoop/data/menu.csv

   Replication 4 set: /user/hadoop/data/menu.csv 10:26:16 ALERT WARN prod: 1 under-replicated blocks
   This one shows up at the next check: the block's target is 4 copies and only 3 DataNodes exist (2 alive).

4. Safe mode on the DR cluster

   docker exec lab72-nn-canteen-1 hdfs dfsadmin -safemode enter

   Safe mode is ON 10:26:17 dr live=3 ... safemode=True 10:26:17 ALERT CRIT dr: NameNode in safe mode (read-only)
   CRIT, because every write to that cluster now fails (Lesson 5.6). The same rule covers both clusters.

5. Fill YARN with two big jobs

   10:26:52 prod live=3 dead=0 stale=1 used=0.6 under=8 missing=0 files=1675 ... pending=1 running=1 mem=50 10:26:52 ALERT WARN prod: 8 under-replicated blocks 10:26:52 ALERT WARN prod: 1 app(s) waiting to start (cluster memory 50% used)
   Two surprises. The second job waited while half the memory was free: the queue's ApplicationMaster limit stopped it (Lesson 12.6). And under-replicated blocks jumped from 1 to 8: each job writes its job.jar and job.split with replication 10.

6. Fix everything and look again

   (stale cleared 246 s after the restart) 10:31:08 prod live=3 dead=0 stale=0 used=0.6 under=7 missing=0 files=1680 ... 10:31:08 ALERT WARN prod: 7 under-replicated blocks
   The jobs were killed, safe mode left, replication reset and the DataNode restarted. Seven blocks still alert: staging files left by the killed jobs, wanting 10 copies forever. Before we cleaned up at the start, the same cause showed 38. Fix the cause (mapreduce.client.submit.file.replication=3, clean old staging folders), don't mute the alert.

Six checks of our monitor

The real monitor output on the left, what happened inside the cluster on the right.

Monitoring at work

Simplified; organisations differ in tools and policies.

An alert rule engine over real snapshots

These snapshots are the prod numbers our monitor printed during the lab, one per check. The engine applies the same rules as monitor.py, plus one improvement: a rule can require its condition to hold for several checks in a row before it fires.

snapshots = [   # prod, from tx124 (time, stale DataNodes, under-replicated blocks, pending apps)
    ("10:21:11", 0, 0, 0), ("10:26:08", 1, 0, 0), ("10:26:16", 1, 1, 0),
    ("10:26:17", 1, 1, 0), ("10:26:52", 1, 8, 1), ("10:31:08", 0, 7, 0),
]
rules = [   # (name, test, checks in a row needed, severity)
    ("stale DataNode", lambda s: s[1] > 0, 1, "WARN"),
    ("under-replicated", lambda s: s[2] > 0, 1, "WARN"),
    ("app waiting", lambda s: s[3] > 0, 1, "WARN"),
]
streak = {name: 0 for name, _, _, _ in rules}
for snap in snapshots:
    fired = []
    for name, test, need, sev in rules:
        streak[name] = streak[name] + 1 if test(snap) else 0
        if streak[name] >= need:
            fired.append(f"{sev} {name}")
    print(snap[0], fired or "ok")

• ("stale DataNode", lambda s: s[1] > 0, 1, "WARN"),Each rule is a test on one snapshot plus how many checks in a row it must hold. 1 = fire at once.
• streak[name] = streak[name] + 1 if test(snap) else 0A streak counter per rule: the basis of “for 5 minutes” rules in Prometheus and other tools.
• ("10:21:11", 0, 0, 0), ("10:26:08", 1, 0, 0), ("10:26:16", 1, 1, 0),Real values from our run. The stale DataNode appeared at 10:26:08, 296 s after the stop.

Run it on your PC: a monitor for two clusters, and four incidents

You need the Module 8 lab (prod) and the Lesson 7.2 federation lab (dr) running, with the lab8 workers joined to the dr network as in Lesson 12.2 (join_clusters.sh). monitor.py runs on lab8-worker1-1 because that machine can reach both clusters. The lab script stops a DataNode, changes replication, enters safe mode, starts two big jobs, then undoes everything. It takes about 15 minutes, mostly waiting for the stale count to refresh.

# monitor.py: a tiny fleet monitor. Reads the JMX/REST endpoints of every NameNode and ResourceManager,
# applies alert rules, prints one line per check. Usage: python3 monitor.py [rounds] [seconds_between]
import json, sys, time, urllib.request

CLUSTERS = {                      # name: (NameNode web, ResourceManager web or None)
    "prod": ("http://namenode:9870", "http://resourcemanager:8088"),
    "dr":   ("http://nn-canteen:9870", None),
}

def get(url):
    with urllib.request.urlopen(url, timeout=5) as r:
        return json.load(r)

def nn_metrics(base):
    beans = get(base + "/jmx?qry=Hadoop:service=NameNode,name=*")["beans"]
    m = {}
    for b in beans:
        m.update(b)
    return m

def check(cluster, nn, rm):
    alerts, info = [], {}
    try:
        m = nn_metrics(nn)
    except Exception as e:
        return [f"CRIT {cluster}: NameNode not answering ({type(e).__name__})"], info
    used = 100.0 * m["CapacityUsed"] / max(m["CapacityTotal"], 1)
    info.update(live=m["NumLiveDataNodes"], dead=m["NumDeadDataNodes"], stale=m["NumStaleDataNodes"],
                used=round(used, 1), under=m["UnderReplicatedBlocks"], missing=m["MissingBlocks"],
                files=m["FilesTotal"], safemode=bool(m.get("Safemode")))
    if info["missing"] > 0:  alerts.append(f"CRIT {cluster}: {info['missing']} missing blocks")
    if info["dead"] > 0:     alerts.append(f"CRIT {cluster}: {info['dead']} dead DataNode(s)")
    if info["stale"] > 0:    alerts.append(f"WARN {cluster}: {info['stale']} stale DataNode(s) (no heartbeat for 30 s)")
    if info["under"] > 0:    alerts.append(f"WARN {cluster}: {info['under']} under-replicated blocks")
    if info["safemode"]:     alerts.append(f"CRIT {cluster}: NameNode in safe mode (read-only)")
    if used > 80:            alerts.append(f"WARN {cluster}: HDFS {used:.0f}% full")
    if rm:
        c = get(rm + "/ws/v1/cluster/metrics")["clusterMetrics"]
        mem = 100.0 * c["allocatedMB"] / max(c["totalMB"], 1)
        info.update(nodes=c["activeNodes"], unhealthy=c["unhealthyNodes"], lost=c["lostNodes"],
                    pending=c["appsPending"], running=c["appsRunning"], mem=round(mem))
        if c["unhealthyNodes"] or c["lostNodes"]:
            alerts.append(f"CRIT {cluster}: {c['unhealthyNodes']} unhealthy / {c['lostNodes']} lost NodeManagers")
        if c["appsPending"] > 0:
            alerts.append(f"WARN {cluster}: {c['appsPending']} app(s) waiting to start (cluster memory {mem:.0f}% used)")
    return alerts, info

rounds = int(sys.argv[1]) if len(sys.argv) > 1 else 1
pause = int(sys.argv[2]) if len(sys.argv) > 2 else 10
for r in range(rounds):
    stamp = time.strftime("%H:%M:%S")
    for name, (nn, rm) in CLUSTERS.items():
        alerts, info = check(name, nn, rm)
        print(f"{stamp} {name:4} " + " ".join(f"{k}={v}" for k, v in info.items()))
        for a in alerts:
            print(f"{stamp}   ALERT {a}")
    if r < rounds - 1:
        time.sleep(pause)

# Copy it to the worker and run one check:
#   docker cp monitor.py lab8-worker1-1:/tmp/monitor.py
#   docker exec lab8-worker1-1 python3 /tmp/monitor.py 1

# monitor_lab.sh: break things on purpose and watch monitor.py notice. Run on your PC.
M="docker exec lab8-worker1-1 python3 /tmp/monitor.py"
docker cp monitor.py lab8-worker1-1:/tmp/monitor.py
echo "== 0. healthy fleet"; $M 1
echo "== 1. a DataNode process dies on prod's worker2"
docker exec lab8-worker2-1 hdfs --daemon stop datanode; s=$(date +%s)
# a DataNode is "stale" after 30 s without a heartbeat, but the NameNode recounts stale nodes only
# on its heartbeat check (every 5 min by default), so the metric can lag: wait until it shows
until $M 1 | grep -q "prod.*stale=[1-9]"; do sleep 10; done
echo "   (the alert appeared $(( $(date +%s) - s )) s after the DataNode stopped)"; $M 1
echo "== 2. someone sets replication 4 on a file (the cluster has only 3 DataNodes, 2 alive)"
docker exec lab8-namenode-1 hdfs dfs -setrep 4 /user/hadoop/data/menu.csv 2>/dev/null | grep -v INFO
sleep 5; $M 1
echo "== 3. an admin puts the DR NameNode into safe mode"
docker exec lab72-nn-canteen-1 hdfs dfsadmin -safemode enter 2>/dev/null
$M 1
echo "== 4. a huge job fills prod's YARN and a second job has to wait"
T=/opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.5.0-tests.jar
for n in 1 2; do docker exec -d lab8-namenode-1 bash -c "export HADOOP_CLASSPATH=$T; yarn org.apache.hadoop.mapreduce.SleepJob -libjars $T -m 30 -r 1 -mt 30000 -rt 1000 > /tmp/mon$n.log 2>&1"; sleep 2; done
sleep 30; $M 1
echo "== 5. fix everything and check again"
for n in 1 2; do app=$(docker exec lab8-namenode-1 grep -o "application_[0-9_]*" /tmp/mon$n.log | head -1); docker exec lab8-resourcemanager-1 yarn application -kill $app > /dev/null 2>&1; done
docker exec lab72-nn-canteen-1 hdfs dfsadmin -safemode leave 2>/dev/null
docker exec lab8-namenode-1 hdfs dfs -setrep 3 /user/hadoop/data/menu.csv > /dev/null 2>&1
docker exec lab8-worker2-1 hdfs --daemon start datanode; s=$(date +%s)
until $M 1 | grep -q "prod.*stale=0"; do sleep 10; done
echo "   (stale cleared $(( $(date +%s) - s )) s after the restart)"; $M 1

# Run on your PC, in the folder that has monitor.py:
#   bash monitor_lab.sh

A “for 3 checks” rule step by step

Watch the streak counter grow and reset. The rule fires only when the condition held for 3 checks in a row, so one bad reading is ignored.

Worked examples: real problems, solved and traced in memory

Four monitoring problems from different domains: designing thresholds, measuring alert noise, forecasting capacity and checking an SLA with a canary.

600 TB · +2 TB/day · WARN 14 days · CRIT 3 days

WARN at 95.3% used (14 days of growth left)
CRIT at 99.0% used (3 days of growth left)

30 pages/week → 1 · 15 min each

about 31 hours of interrupted sleep saved per month

4 weekly readings · 700 TB · target 85%

growing 19.3 TB/week; 85% reached in about 6 weeks

6 canary durations · normal max 75 s

run 3: median 70 s ok
run 4: median 140 s ok
run 5: median 150 s CRIT
run 6: median 150 s CRIT

Check yourself

Quick revision

Interview and viva questions

Practice problems

Monitoring exercises using the real numbers from our lab.

used 4,769 MB of 800,000 MB

0.6% used ok

used, total = 4769, 800000
pct = 100 * used / total
print(f"{pct:.1f}% used", "WARN" if pct > 80 else "ok")

target 3; live 3, 2, 0, 1

3 ok
2 under-replicated
0 missing
1 under-replicated

target = 3
for live in [3, 2, 0, 1]:
    print(live, "missing" if live == 0 else "under-replicated" if live < target else "ok")

4 jobs × 2 blocks

8 under-replicated blocks

print(4 * 2, "under-replicated blocks")

a JSON string

live 3 stale 1 missing 0

import json
reply = '{"beans": [{"NumLiveDataNodes": 3, "NumStaleDataNodes": 1, "MissingBlocks": 0}]}'
b = json.loads(reply)["beans"][0]
print("live", b["NumLiveDataNodes"], "stale", b["NumStaleDataNodes"], "missing", b["MissingBlocks"])

296 s, 30 s

266 s of lag beyond the 30 s definition

print(296 - 30, "s of lag beyond the 30 s definition")

a list of states

check 1: FIRING
check 4: RESOLVED
check 5: FIRING

states = [False, True, True, True, False, True]
prev = False
for i, s in enumerate(states):
    if s != prev:
        print(f"check {i}:", "FIRING" if s else "RESOLVED")
    prev = s

₹6 lakh · 3 × 3 hours saved · ₹2 lakh/hour

saves Rs 18 lakh vs costs Rs 6 lakh

saved = 3 * (4 - 1) * 2
print(f"saves Rs {saved} lakh vs costs Rs 6 lakh")

prod and dr snapshots

WARN prod 1 stale DataNode(s)
CRIT dr safe mode

fleet = {"prod": {"missing": 0, "safemode": False, "stale": 1},
         "dr": {"missing": 0, "safemode": True, "stale": 0}}
for name, m in fleet.items():
    if m["missing"]: print("CRIT", name, "missing blocks")
    if m["safemode"]: print("CRIT", name, "safe mode")
    if m["stale"]: print("WARN", name, m["stale"], "stale DataNode(s)")

Real-world scenario questions (asked in company interviews)

Monitoring questions are about judgement: what deserves a page, and what you do when one arrives.