Heartbeats and block reports

1 The story

“Present, sir!” every three seconds

Kabir freezes datanode-4 with docker pause: the process is still there, but it can't do anything. He keeps asking the NameNode about it. “Live, last contact 10 seconds ago… 27 seconds… 43… 51…” Still live. Then at 57 seconds: dead. He unpauses it, and six seconds later the NameNode says “live, last contact 0 seconds”. Riya: “How does the NameNode know any of this? It never looks at the DataNodes.”

Arjun Sir

Like a class register. Every DataNode calls out “present” to the NameNode every 3 seconds: that's a heartbeat, with a few numbers about its disks and load. The NameNode just notes the time. If a DataNode goes quiet, the NameNode doesn't phone it; silence itself is the information.

Kabir

And how did it know what blocks datanode-4 has, after it came back?

Arjun Sir

That's the second message: the block report. A DataNode lists every block it holds when it registers, then again every 6 hours, plus small updates whenever it gets or deletes a block. Heartbeats say “I'm alive”; block reports say “here's what I have”. And the NameNode's replies to heartbeats are how it gives orders: “copy this block”, “delete that one”.

2 Definition

Heartbeats and block reports

Formal definition (write this in exams)

A heartbeat is a periodic RPC from each DataNode to the NameNode (every dfs.heartbeat.interval = 3 s) carrying the DataNode's capacity, used and remaining space, number of active transfers and other load figures. It tells the NameNode the DataNode is alive, and the NameNode's reply carries commands for the DataNode, such as replicating a block to another node, deleting (invalidating) blocks, or re-registering. A DataNode not heard from for dfs.namenode.stale.datanode.interval (30 s) is stale; after 2 × dfs.namenode.heartbeat.recheck-interval + 10 × heartbeat interval (10 min 30 s by default) it is declared dead and its replicas are re-replicated. A block report is a list of all block replicas a DataNode stores (ID, length, generation stamp) sent at registration and then every dfs.blockreport.intervalMsec (6 hours); between full reports, incremental block reports announce replicas received or deleted. The NameNode builds and corrects its block map from these reports.

In simple words

Heartbeat: “I'm alive, and here's how full I am” every 3 seconds. Block report: “Here is every block I hold”, when starting and every 6 hours, with small updates in between. The NameNode answers heartbeats with jobs to do.

Words you'll meet in this lesson

HeartbeatSmall RPC every 3 s: alive + capacity and load numbers.

Heartbeat replyThe NameNode's answer, carrying commands: replicate, invalidate (delete), etc.

Last contactSeconds since the last heartbeat, shown in dfsadmin -report and the UI.

Stale nodeNo heartbeat for 30 s. Still “live”, but can be avoided for reads/writes.

Dead nodeNo heartbeat for 10 min 30 s (default). Its replicas are re-created elsewhere.

Full block reportEvery replica on the DataNode. At start-up and every 6 hours.

Incremental block reportOnly changes: replicas received or deleted since the last report.

RegistrationA DataNode's first contact: it identifies itself, then sends a full block report.

Timers checked on Hadoop 3.5.0

dfs.heartbeat.interval	3 s
dfs.namenode.stale.datanode.interval	30,000 ms (30 s); in our lab the node was counted stale at about 38 s
dfs.namenode.heartbeat.recheck-interval	300,000 ms (5 min) → dead after 10 min 30 s. Lab: 15,000 ms → dead after about 60 s (seen at 57 s)
dfs.blockreport.intervalMsec	21,600,000 ms (6 hours)
dfs.blockreport.initialDelay	0 s (first report sent right after registration)
Avoid stale nodes	`dfs.namenode.avoid.read.stale.datanode` / `…avoid.write…`: false by default

Heartbeats at three levels

Pick your level. New to this? Start at Beginner and climb.

Beginner Attendance and inventory

Heartbeat = roll call: every 3 seconds each shelf-keeper says “present”, and how much space their shelf has left.
Block report = stock-taking: the shelf-keeper lists every book on the shelf. Done when they join, then every 6 hours.
The head librarian answers the roll call with tasks: “photocopy book 25 for shelf 3”, “throw away the extra copy of book 7”.

Intermediate What the NameNode does with them

Updates each DataNode's “last contact” and capacity, which dfsadmin -report shows.
Chooses targets for new blocks using reported free space and load.
Marks nodes stale, then dead, purely from missing heartbeats.
Rebuilds the block map after a restart from full block reports (Lesson 5.6).
Finds lost, extra or corrupt replicas by comparing reports with what it expects.

Advanced Large-cluster details

The NameNode never initiates calls to DataNodes; all commands travel in heartbeat replies. This keeps one NameNode able to manage thousands of nodes.
Full block reports from big DataNodes (millions of blocks) are expensive; they are spread out over time and can be split per disk.
A busy NameNode may process heartbeats late; the optional lifeline RPC (dfs.namenode.lifeline.rpc-address) keeps busy-but-healthy nodes from being declared dead.
hdfs dfsadmin -triggerBlockReport host:9867 forces a full report, useful after disk repairs.

Exam tip: “Explain heartbeat and block report”: heartbeat = DataNode → NameNode every 3 s, liveness + storage stats; reply carries commands; no heartbeat for 10 min 30 s → dead → re-replication. Block report = list of all blocks on a DataNode, at start-up and every 6 h (plus incremental reports); used to build the block map.

3 Types and ways to use it

The messages between DataNodes and the NameNode

Message	When	Carries	NameNode uses it to
Registration	DataNode start, or when asked to re-register	Identity, storage IDs, cluster ID	Add the node to the topology
Heartbeat	Every 3 s	Capacity, used, remaining, transfers in progress	Track liveness and load; send commands back
Full block report	After registration; every 6 h	Every replica: ID, length, generation stamp	Build/correct the block map
Incremental block report	When replicas are received or deleted	Only the changes	Keep the block map current between full reports
Cache report	Periodically, if caching is used	Blocks cached in memory	Centralised cache management

Commands the NameNode sends back

Transfer

“Copy block X to DataNode Y”: how re-replication happens.

Replicate

Invalidate

“Delete these replicas”: for deleted files or over-replicated blocks.

Delete

Recover

“Recover this block”: finish or repair a block whose writer died.

Recovery

Register / shut down

Ask the node to re-register, or tell it to stop.

Control

When a DataNode looks “dead” but its process is running, check the network to the NameNode and the DataNode's log for long GC pauses or disk hangs. A frozen process, like our paused container, is exactly as dead as a stopped one from the NameNode's point of view.

4 Step by step

Watch heartbeats and block reports on the real cluster

Module 5 lab cluster: dfs.namenode.heartbeat.recheck-interval is 15 s, so dead detection takes about 60 s instead of 10 min 30 s. Everything else uses defaults.

Read the timers
```
for k in dfs.heartbeat.interval dfs.namenode.heartbeat.recheck-interval \
         dfs.namenode.stale.datanode.interval dfs.blockreport.intervalMsec; do
  echo "$k = $(docker compose exec namenode hdfs getconf -confKey $k)"
done
```
dfs.heartbeat.interval = 3 dfs.namenode.heartbeat.recheck-interval = 15000 dfs.namenode.stale.datanode.interval = 30000 dfs.blockreport.intervalMsec = 21600000
Seconds for the heartbeat; milliseconds for the others. 21,600,000 ms = 6 hours.
See each DataNode's last contact
```
docker compose exec namenode bash -c 'curl -s "http://namenode:9870/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo"' \
  | python3 -c "import sys,json; b=json.load(sys.stdin)['beans'][0]; [print(n, v['lastContact'], v['numBlocks']) for n,v in json.loads(b['LiveNodes']).items()]"
```
ab3b1bedf98a:9866 2 8 46201bb62e68:9866 1 8 729c51a07ee2:9866 0 8 264eb9c0f8e2:9866 0 0
Columns: node, seconds since last heartbeat, blocks held. Every DataNode was heard from within the last 3 seconds. The names are the containers' host names (container IDs); 729c51a07ee2 is datanode-4. One node holds 0 blocks after our earlier experiments: being alive and holding data are separate facts.
Freeze a DataNode and watch
```
docker pause lab5-datanode-4
# every few seconds, ask the NameNode about 729c51a07ee2
# (or run heartbeat_watch.py from Part 4b)
```
t= 8s NameNode says: live lastContact=10s t= 16s NameNode says: live lastContact=19s t= 25s NameNode says: live lastContact=27s t= 33s NameNode says: live lastContact=35s t= 41s NameNode says: live lastContact=43s t= 49s NameNode says: live lastContact=51s t= 57s NameNode says: dead lastContact=60s t= 65s NameNode says: dead lastContact=68s t= 82s NameNode says: dead lastContact=84s
“Last contact” simply grows. At 60 s (2 × 15 s + 10 × 3 s) the node moves to the dead list. In another run we also read the NameNode's counters: stale 0 up to 32 s, stale 1 from 38 s, then dead 1 at 64 s.
Unfreeze it: re-registration and a block report
```
docker unpause lab5-datanode-4
docker compose logs namenode | grep -E "registerDatanode|processReport" | tail -3
```
BLOCK* registerDatanode: from DatanodeRegistration(172.19.0.7:9866, datanodeUuid=1a7e1105-…, infoPort=9864, infoSecurePort=0, ipcPort=9867, … BLOCK* processReport 0x283bca9221d1422b with lease ID 0x7921c25a0ab90066: Processing first storage report for DS-e40699e9-… from datanode DatanodeRegistration(172.19.0.7:9866, … BLOCK* processReport 0x283bca9221d1422b with lease ID 0x7921c25a0ab90066: from storage DS-e40699e9-… node DatanodeRegistration(172.19.0.7:9866, …
The NameNode had forgotten this node, so on its next heartbeat it was told to re-register; then it sent a full block report, and its replicas were back in the block map within seconds (“lastContact 0”).
Check and trigger a full block report
```
docker compose exec namenode bash -c 'hdfs dfsadmin -report | grep -E "^Name|Last Block Report" | head -2'
docker compose exec namenode hdfs dfsadmin -triggerBlockReport lab5-datanode-2:9867
docker compose exec namenode bash -c 'hdfs dfsadmin -report | grep -E "^Name|Last Block Report" | head -2'
```
Name: 172.19.0.4:9866 (lab5-datanode-2.lab5_default) Last Block Report: Wed Oct 21 06:49:03 UTC 2026 Triggering a full block report on lab5-datanode-2:9867. Name: 172.19.0.4:9866 (lab5-datanode-2.lab5_default) Last Block Report: Wed Oct 21 06:59:19 UTC 2026
datanode-2's last full report was from its start at 06:49 (the next one was due 6 hours later). The trigger made it report at once. The command uses the DataNode's IPC port, 9867.

Don't lower heartbeat and dead-node intervals on real clusters just to “detect failures faster”. Short timers turn GC pauses and network blips into false deaths, which trigger mass re-replication and can overload the cluster. The lab uses short values only so you can watch.

5 See it happen

From live to dead and back

The real readings from step 3 and step 4, alongside what the NameNode is thinking.

datanode-4 goes silent 6 steps

What you see

Behind the screen

Try it yourself: heartbeats in the playground

Pink dots are heartbeats. Stop a node and use Skip to timeout to jump to the 10 min 30 s dead-node limit.

6 Real-world use

Heartbeats and reports at work

1. Alerts on dead and stale nodes

Monitoring reads NumDeadDataNodes and NumStaleDataNodes from the NameNode's /jmx. One dead node is routine; several at once usually means a rack or network problem worth waking someone up for.

/jmx→Dead/stale counts→Alert

2. Diagnosing flapping DataNodes

A DataNode that keeps going stale and coming back often has long Java garbage-collection pauses, a hung disk or network drops. Engineers compare its “last contact” history with GC and kernel logs.

Stale again and again→GC / disk / network→Fix the node

3. Block report storms

After a full cluster restart, thousands of DataNodes send full block reports at once. Big clusters tune report leases and splitting so the NameNode isn't overwhelmed, which shortens safe mode (Lesson 5.6).

Restart→Reports in waves→NameNode stays responsive

4. After disk repairs

When a failed disk is replaced or a data folder restored, admins run dfsadmin -triggerBlockReport so the NameNode learns the node's real contents now instead of in up to 6 hours.

Repair→Trigger report→Block map correct

Simplified; organisations differ in tools and policies.

7 Run the code

The NameNode's view of a DataNode

This function classifies a DataNode from the seconds since its last heartbeat, using the real default thresholds. Run it for the readings from the lab, then try the challenges.

HEARTBEAT = 3            # dfs.heartbeat.interval (s)
STALE = 30               # dfs.namenode.stale.datanode.interval (s)
RECHECK = 300            # dfs.namenode.heartbeat.recheck-interval (s)
DEAD = 2 * RECHECK + 10 * HEARTBEAT

def state(last_contact):
    if last_contact >= DEAD:
        return "DEAD"
    if last_contact >= STALE:
        return "STALE"
    return "LIVE"

print("dead after", DEAD, "seconds")
for seconds in [2, 27, 43, 300, 629, 630, 900]:
    print(f"last contact {seconds:>3}s -> {state(seconds)}")

dead after 630 seconds last contact 2s -> LIVE last contact 27s -> LIVE last contact 43s -> STALE last contact 300s -> STALE last contact 629s -> STALE last contact 630s -> DEAD last contact 900s -> DEAD

DEAD = 2 * RECHECK + 10 * HEARTBEATThe famous 10 min 30 s: 2 × 300 s + 10 × 3 s = 630 s.
if last_contact >= STALE:Stale nodes are still live; they can be avoided for reads and writes if the avoid-stale options are on.

Run it on your PC: watch a DataNode die and come back

This script freezes datanode-4 with docker pause (no heartbeats, but the process stays), asks the NameNode about it every 8 seconds for 80 seconds, then unfreezes it. Save it as heartbeat_watch.py in the hdfs-lab folder and run it while the lab cluster is up.

Live → dead → live, from the NameNode's side

# heartbeat_watch.py: freeze one DataNode and watch the NameNode's view of it
# change from live to stale to dead. Run in the hdfs-lab folder.
import json
import subprocess
import time

DN = "lab5-datanode-4"
URL = "http://namenode:9870/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo"

def sh(*cmd):
    return subprocess.run(cmd, capture_output=True, text=True).stdout

def nn_view(host):
    info = json.loads(sh("docker", "compose", "exec", "-T", "namenode", "curl", "-s", URL))["beans"][0]
    for state in ("LiveNodes", "DeadNodes"):
        for name, node in json.loads(info[state]).items():
            if name.startswith(host):
                return state[:-5].lower(), node["lastContact"]
    return "unknown", None

host = sh("docker", "inspect", "-f", "{{.Config.Hostname}}", DN).strip()
sh("docker", "pause", DN)                      # the process freezes: no more heartbeats
start = time.time()
try:
    while time.time() - start < 80:
        time.sleep(8)
        state, last = nn_view(host)
        print(f"t={time.time() - start:3.0f}s  NameNode says: {state:5} lastContact={last}s")
finally:
    sh("docker", "unpause", DN)
time.sleep(6)
print("after unpause:", nn_view(host))

# In the hdfs-lab folder:
#   python3 heartbeat_watch.py

t= 8s NameNode says: live lastContact=10s t= 16s NameNode says: live lastContact=19s t= 25s NameNode says: live lastContact=27s t= 33s NameNode says: live lastContact=35s t= 41s NameNode says: live lastContact=43s t= 49s NameNode says: live lastContact=51s t= 57s NameNode says: dead lastContact=60s t= 65s NameNode says: dead lastContact=68s t= 74s NameNode says: dead lastContact=76s t= 82s NameNode says: dead lastContact=84s after unpause: ('live', 0)

Real output with the lab's 15 s recheck interval. With default settings the same script would need to run for more than 10 minutes to see “dead”. The block re-replication this triggers is covered in Lesson 5.3.

8 Line by line

Rebuilding a block map from reports

Follow the NameNode as two full block reports and one incremental report arrive. Watch the block map grow and change.

Worked examples: real problems, solved and traced in memory

Three calculations about heartbeats and reports that platform teams make: NameNode load, timer trade-offs, and the cost of a false death. Each runs live and is traced in memory.

Example 1BeginnerLoad · NameNode

How many messages does the NameNode handle?

Problem statement

A cluster has 600 DataNodes. Each sends a heartbeat every 3 s and a full block report every 6 hours, and on average 2 incremental reports per second while jobs run. How many messages per second reach the NameNode from DataNodes?

Input

600 nodes · heartbeat 3 s · full report 6 h · 2 incremental/s per node

Expected output

heartbeats 200/s, full reports 0.028/s, incremental 1200/s
total ~1400 messages per second

Approach

Heartbeats/s = nodes / 3.
Full reports/s = nodes / (6 × 3600).
Incremental/s = nodes × 2. Add them up.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 2IntermediateTuning · trade-off

Faster detection vs false alarms

Problem statement

A team wants faster dead-node detection. With recheck intervals of 300, 120 and 30 s, compute the dead time. Then, if the cluster sees GC pauses up to 90 s about once a week per node across 500 nodes, which settings would declare healthy nodes dead?

Input

recheck 300 / 120 / 30 s · heartbeat 3 s · GC pauses ≤ 90 s

Expected output

recheck 300s: dead after 630s, false deaths/week ~0
recheck 120s: dead after 270s, false deaths/week ~0
recheck  30s: dead after  90s, false deaths/week ~500

Approach

Dead time = 2 × recheck + 10 × 3.
A pause longer than the dead time causes a false death.
Count false deaths per week.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 3AdvancedOperations · re-registration

How long until a returning node's data counts again?

Problem statement

A DataNode with 12 lakh replicas comes back after a network outage. It re-registers on its next heartbeat, then sends a full block report that the NameNode processes at 4 lakh replicas per second. How long until all its replicas are back in the block map?

Input

12,00,000 replicas · heartbeat 3 s · 4,00,000 replicas/s

Expected output

about 6 seconds

Approach

Worst-case wait for the next heartbeat: 3 s.
Processing time = replicas / rate.
Add them.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

9 Quiz and revision

Check yourself

Quick revision

Lesson 5.5 in one screen

Heartbeat: DataNode → NameNode every 3 s: alive + capacity/load. Reply carries commands.
The NameNode never calls DataNodes; it only measures silence.
Stale after 30 s; dead after 2 × recheck (5 min) + 10 × 3 s = 10 min 30 s.
Full block report: all replicas, at registration and every 6 h. Incremental: changes in between.
Block reports build the block map (not saved in the fsimage).
Commands: transfer (replicate), invalidate (delete), recover, re-register.
Check: “Last contact”, “Last Block Report” in dfsadmin -report; /jmx LiveNodes/DeadNodes.
hdfs dfsadmin -triggerBlockReport host:9867 forces a report.

Interview and viva questions

What is a heartbeat in Hadoop?

A small periodic message (every 3 seconds by default) from each DataNode to the NameNode saying it is alive, with its storage capacity, usage and load. The NameNode uses the reply to send the DataNode commands.

What happens if the NameNode stops receiving heartbeats from a DataNode?

After 30 seconds the node is marked stale; after 10 minutes 30 seconds (default) it is marked dead. Its replicas are then removed from the block map and the affected blocks are re-replicated from other copies.

What is a block report?

A list of all the block replicas stored on a DataNode, sent when it registers and then every 6 hours by default. Incremental block reports announce new and deleted replicas in between. The NameNode uses them to know where each block is.

Why does a DataNode need to send block reports if the NameNode chose where blocks go?

Because reality can differ: disks fail, files are corrupted or deleted, nodes return with old replicas, and block locations are not stored in the fsimage. Reports are the source of truth for which replicas exist.

How is the dead-node timeout calculated?

2 × dfs.namenode.heartbeat.recheck-interval + 10 × dfs.heartbeat.interval. With defaults: 2 × 300 s + 10 × 3 s = 630 s.

Can the NameNode contact a DataNode directly?

No. In the DataNode protocol the DataNode always initiates the RPC (heartbeats, reports); the NameNode responds, putting any commands in its replies.

Practice problems

Timer and report exercises using the real readings from the lab cluster.

P1Easy

Lab dead time

Compute the dead-node time for the lab (recheck 15 s, heartbeat 3 s).

Input

15 s · 3 s

Expected output

60 s

Show solution and explanation

print(2 * 15 + 10 * 3, "s")

60 s

How it works: 60 s, matching the real timeline (live at 51 s, dead at 57–60 s).

P2Easy

Milliseconds to hours

Convert dfs.blockreport.intervalMsec = 21600000 to hours.

Input

21,600,000 ms

Expected output

6.0 hours

Show solution and explanation

print(21600000 / 1000 / 3600, "hours")

6.0 hours

How it works: 6.0 hours.

P3Easy

Stale or not

With a 30 s stale interval, classify last-contact readings 27, 35 and 51 s.

Input

27 · 35 · 51

Expected output

27 fresh
35 stale
51 stale

Show solution and explanation

for s in [27, 35, 51]:
    print(s, "stale" if s >= 30 else "fresh")

27 fresh 35 stale 51 stale

How it works: 27 s fresh; 35 and 51 s stale (but still live until the dead limit).

P4Medium

Parse the JMX reading

From the real JMX line "729c51a07ee2:9866 lastContact=51s blocks=8", extract the host and the seconds.

Input

729c51a07ee2:9866 lastContact=51s blocks=8

Expected output

729c51a07ee2 51

Show solution and explanation

line = "729c51a07ee2:9866 lastContact=51s blocks=8"
host = line.split(":")[0]
seconds = int(line.split("lastContact=")[1].split("s")[0])
print(host, seconds)

729c51a07ee2 51

How it works: Host 729c51a07ee2 (datanode-4), 51 seconds since its last heartbeat.

P5Medium

Apply an incremental report

Apply the incremental report +b4, -b1 from dn3 to the block map.

Input

b1:[dn1,dn3] · b4:[dn2]

Expected output

{'b1': ['dn1'], 'b4': ['dn2', 'dn3']}

Show solution and explanation

bm = {"b1": ["dn1", "dn3"], "b4": ["dn2"]}
for item in ["+b4", "-b1"]:
    blk = item[1:]
    if item[0] == "+":
        bm.setdefault(blk, []).append("dn3")
    else:
        bm[blk].remove("dn3")
print(bm)

{'b1': ['dn1'], 'b4': ['dn2', 'dn3']}

How it works: b1 keeps only dn1, b4 gains dn3. Incremental reports keep the map current between full reports.

P6Medium

Heartbeat load

How many heartbeats per second does a NameNode receive from 3,000 DataNodes?

Input

3000 nodes · 3 s

Expected output

1000.0 per second

Show solution and explanation

print(3000 / 3, "per second")

1000.0 per second

How it works: 1,000 heartbeats per second, each answered, which is why heartbeat handling must be cheap.

P7Business problem

False death cost

A false dead-node event re-replicates 20 TB at 4 GB/s cluster-wide and slows jobs by 20% during copying. How long are jobs slowed?

Input

20 TB · 4 GB/s

Expected output

85 minutes of slower jobs

Show solution and explanation

seconds = 20 * 1024 / 4
print(f"{seconds / 60:.0f} minutes of slower jobs")

85 minutes of slower jobs

How it works: About 85 minutes of degraded performance for nothing, if the node comes back right after. Hence conservative timeouts.

P8Advanced

Timeline classifier

Given (t, lastContact) samples, print the first time the node becomes stale and dead, for stale 30 s and dead 60 s.

Input

real lab samples

Expected output

stale first seen at t = 33 s; dead first seen at t = 57 s

Show solution and explanation

samples = [(8, 10), (16, 19), (25, 27), (33, 35), (41, 43), (49, 51), (57, 60), (65, 68)]
stale = next(t for t, lc in samples if lc >= 30)
dead = next(t for t, lc in samples if lc >= 60)
print("stale first seen at t =", stale, "s; dead first seen at t =", dead, "s")

stale first seen at t = 33 s; dead first seen at t = 57 s

How it works: Stale at t = 33 s (last contact 35 s) and dead at t = 57 s (60 s), consistent with the thresholds and the real readings.

Real-world scenario questions (asked in company interviews)

Heartbeat questions check whether you understand how the NameNode can manage thousands of nodes cheaply.

HDFS · conceptsFresher

“How does the NameNode know a DataNode has failed?”

Asked in almost every HDFS interview.

The interviewer is checking: heartbeats, timeouts and consequences.

Show a strong answer

DataNodes send heartbeats every 3 s. The NameNode tracks time since the last one.
After 30 s the node is stale; after 10 min 30 s (2 × recheck + 10 heartbeats) it is declared dead.
Then its replicas are removed from the block map, and under-replicated blocks are copied from surviving replicas, with commands sent in other DataNodes' heartbeat replies.

Operations · flappingExperienced

A DataNode keeps going dead and coming back

The NameNode UI shows one node dead several times a day, each time followed by re-replication and then excess deletions.

The interviewer is checking: systematic debugging.

Show a strong answer

Correlate dead times with the DataNode's GC log (long pauses), kernel/disk logs (hung I/O), and network errors.
Common fixes: right-size the DataNode heap and GC settings, replace the failing disk, fix the NIC/switch port.
Meanwhile, consider decommissioning the node so it stops causing re-replication storms.

Design · protocolExperienced

Why do DataNodes call the NameNode, not the other way round?

A system design follow-up question.

The interviewer is checking: scalability reasoning.

Show a strong answer

With thousands of DataNodes, the NameNode would need many outgoing connections and timeouts to poll them; instead each node pushes a tiny message on a schedule.
Failure detection becomes trivial: silence. No connection management for dead nodes.
Commands piggyback on replies, so there is no extra round trip.
The same push model is used by YARN NodeManagers to the ResourceManager.

Operations · repairFresher

After replacing a disk, the NameNode still shows old block counts

A disk on a DataNode was replaced and its data restored from another copy, but the NameNode's numbers look out of date.

The interviewer is checking: block report timing.

Show a strong answer

The next full block report may be hours away (every 6 h).
Run hdfs dfsadmin -triggerBlockReport host:9867, then check “Last Block Report” and the block counts in dfsadmin -report.
Verify with hdfs fsck / that no blocks are missing or corrupt.

Heartbeats and block reports

“Present, sir!” every three seconds

Heartbeats and block reports

Words you'll meet in this lesson

Timers checked on Hadoop 3.5.0

Heartbeats at three levels

Beginner Attendance and inventory

Intermediate What the NameNode does with them

Advanced Large-cluster details

The messages between DataNodes and the NameNode

Commands the NameNode sends back

Transfer

Invalidate

Recover

Register / shut down

Watch heartbeats and block reports on the real cluster

Read the timers

See each DataNode's last contact

Freeze a DataNode and watch

Unfreeze it: re-registration and a block report

Check and trigger a full block report

From live to dead and back

Try it yourself: heartbeats in the playground

Heartbeats and reports at work

1. Alerts on dead and stale nodes

2. Diagnosing flapping DataNodes

3. Block report storms

4. After disk repairs

The NameNode's view of a DataNode

Run it on your PC: watch a DataNode die and come back

Live → dead → live, from the NameNode's side

Rebuilding a block map from reports

Worked examples: real problems, solved and traced in memory

How many messages does the NameNode handle?

Faster detection vs false alarms

How long until a returning node's data counts again?

Check yourself

Quick revision

Lesson 5.5 in one screen

Interview and viva questions

Practice problems

Lab dead time

Milliseconds to hours

Stale or not

Parse the JMX reading

Apply an incremental report

Heartbeat load

False death cost

Timeline classifier

Real-world scenario questions (asked in company interviews)

“How does the NameNode know a DataNode has failed?”

A DataNode keeps going dead and coming back

Why do DataNodes call the NameNode, not the other way round?

After replacing a disk, the NameNode still shows old block counts