Failures are normal

1 The story

The night Riya's laptop died

The night before the project submission, Riya's laptop shows a black screen and a clicking sound. The hard disk is dead. Her code, her report, three weeks of work: all on that one disk. Kabir's copy is on his laptop and on Google Drive, so the team survives. The next morning Arjun Sir hears the story and smiles. “Now imagine you are responsible for ten thousand disks,” he says.

Riya

Ten thousand disks, Sir? Then something must be breaking every day!

Arjun Sir

Several things, every day. Disks, power supplies, memory chips, network cables, whole switches. If one disk fails 1 or 2 times in a hundred per year, ten thousand disks give you a failure almost every day.

Kabir

At my internship nobody seemed worried when a node died yesterday. They just opened a ticket to replace the disk.

Arjun Sir

Because the system was designed for failure. That was the big idea behind Google's file system and then Hadoop: buy ordinary hardware, expect it to fail, and let the software keep extra copies, notice failures and repair itself.

Riya

So we don't try to stop failures. We make them not matter.

Arjun Sir

Perfect, Riya. Today you'll see every kind of failure a cluster faces, do the arithmetic of how often they happen, and break the playground cluster on purpose to watch HDFS heal.

2 Definition

Failures are normal: fault tolerance

Formal definition (write this in exams)

In a large cluster of commodity hardware, component failures (disks, nodes, network links, racks) are expected events rather than exceptions, because the probability of at least one failure grows with the number of components. Fault tolerance is the ability of a system to continue operating correctly when some of its components fail. Hadoop achieves it through data replication across nodes and racks, failure detection using heartbeats and timeouts, automatic re-replication and task re-execution, checksums to detect corrupted data, and high-availability masters.

In simple words

With thousands of parts, something is always breaking. So instead of buying perfect machines, Hadoop keeps several copies of everything, notices quickly when something dies, and automatically re-copies data and re-runs work somewhere else. Users usually never notice.

Words you'll meet in this lesson

Fault toleranceKeep working correctly even when some parts fail.

AFRAnnualised failure rate: the share of devices that fail in a year, for example 1.5% of disks.

MTBFMean time between failures: the average time a device runs before it fails.

ReplicaOne copy of a block. HDFS keeps 3 by default.

Re-replicationMaking new copies when some are lost, until every block is back to its replication factor.

ChecksumA small number computed from data. If the data changes, the checksum no longer matches, so corruption is detected.

Failure domainA group of parts that can fail together: a disk, a node, a rack, a data centre.

StragglerA node that is alive but very slow. Can hold up a whole job.

Retry / re-executionRunning a failed task again, usually on another node.

Typical numbers (approximate)

Hard disk annual failure rate	around 1–2% per year in large fleets (varies by model and age)
HDFS default replication	3 copies, on at least 2 racks (`dfs.replication`)
HDFS checksum	CRC32C for every 512 bytes of data, checked on every read
DataNode block scanner	re-reads and verifies every block in the background, by default about once every 3 weeks
Map/reduce task attempts	a failing task is tried up to 4 times before the job fails (default)
Dead DataNode timeout	about 10 minutes 30 seconds without heartbeats

Failure rates are ballpark figures from public reports on large disk fleets. Your hardware may be better or worse.

Failures at three levels

Pick your level. New to this? Start at Beginner and climb.

Beginner The tiffin delivery team

A delivery boy's bike gets a puncture every few weeks. With 500 delivery boys, several punctures happen every day.
The company doesn't buy puncture-proof bikes (too expensive). It keeps spare boys nearby, so a tiffin is handed over and still arrives.
The office knows where every boy is (heartbeat from the app). If one goes silent, his tiffins are reassigned.
That is Hadoop: cheap parts, spares, tracking and automatic reassignment.

Intermediate The arithmetic of “normal”

Expected failures per year = number of devices × AFR. 3,000 disks × 1.5% ≈ 45 a year, almost one a week.
Chance of at least one failure among n devices = 1 − (1 − p)ⁿ. It climbs quickly as n grows.
A long job is more likely to see a failure than a short one: a 6-hour job on 500 nodes will often see a task fail.
With 3 replicas, losing data needs all three copies to die before re-replication finishes. That is very unlikely if copies are on independent nodes and racks.

Advanced Failures that are harder than “a node dies”

Correlated failures: a rack's power strip or switch takes 20–40 nodes down at once. That's why HDFS puts replicas on at least two racks.
Silent data corruption: a disk returns wrong bytes without an error. HDFS stores a checksum with every 512 bytes; a mismatch marks that replica corrupt, and a good copy is used and re-replicated.
Gray failures / stragglers: a node is “alive” but very slow (failing disk, overheating). Speculative execution runs a backup copy of its slow task.
Network partitions: nodes are alive but can't reach the master. The master must decide whether to treat them as dead, which is why timeouts are not too short.
Master failure and split-brain: with two NameNodes, they must never both act as active. HDFS HA uses ZooKeeper and fencing to guarantee only one is active.
Human error: a wrong rm -r on HDFS is copied to all replicas instantly. Replication is not backup: use HDFS trash, snapshots and copies in another cluster.

Exam tip: Explain “failure is the norm, not the exception” with the formula expected failures = devices × failure rate and a worked number. Then list Hadoop's fault-tolerance mechanisms: replication (3, rack-aware), heartbeats, re-replication, task re-execution, checksums, NameNode HA.

3 Types and ways to use it

What fails, and how Hadoop survives it

Failure	How often (big cluster)	Hadoop's answer
One disk	Weekly or daily	Other replicas are read; the DataNode reports the lost blocks; NameNode re-replicates. The node can keep running on its other disks if configured to tolerate failed volumes.
Whole node (power supply, motherboard, OS crash)	Weekly to monthly	Heartbeats stop → dead after timeout → re-replicate all its blocks; YARN re-runs its tasks.
Whole rack (ToR switch, power strip)	A few times a year	Rack-aware placement: every block has a copy on another rack, so data stays available.
Corrupt data (bad sectors, bit flips)	Occasional	Checksums on every read and a background block scanner; corrupt replica replaced from a good one.
Slow node (straggler)	Common	Speculative execution runs a backup task; the faster copy wins.
Task bug or out of memory	Common	Task retried (up to 4 attempts by default); then the job fails with a clear error.
Master (NameNode) failure	Rare, but serious	High Availability: standby NameNode takes over automatically.
Whole data centre (flood, fire, power)	Very rare	Not handled by one cluster. A second cluster in another city, kept in sync with DistCp (Module 12).
Human mistake (deleting data)	Happens!	HDFS trash, snapshots, backups. Replication alone copies the mistake.

Four ways to tolerate faults

Redundancy

Keep extra copies of data (replicas or erasure-coded parity) and extra machines (standby masters, spare nodes).

Data survives

Detection

Heartbeats, timeouts, checksums and block reports find failures quickly and automatically.

Notice fast

Recovery

Re-replicate lost blocks, re-run lost tasks, fail over to a standby master, without a human.

Self-healing

Isolation

Spread copies across failure domains (nodes, racks, data centres) so one event can't hit them all.

Limit blast radius

Replication is not a backup. If someone deletes a file or a buggy job overwrites good data, HDFS faithfully applies that change to all three copies. Protect important data with trash, snapshots and a copy in another cluster.

4 Step by step

A disk dies at 3 am: what happens, minute by minute

03:00 · The disk stops responding

One of the 12 disks in dn17 fails. The DataNode's reads and writes to that disk return errors.
If the DataNode is set to tolerate a failed volume (dfs.datanode.failed.volumes.tolerated), it keeps running on its other 11 disks. With the default of 0, the whole DataNode shuts down, which is handled like a node failure.
03:00 · A job was reading from it

A map task was reading block 42 from that disk. The read fails.
The HDFS client simply switches to another replica of block 42 on another node and continues. The task may not even fail.
03:01 · The NameNode learns what was lost

dn17 reports the failed volume; its blocks on that disk no longer count.
Every block that had a replica on that disk is now under-replicated: 2 copies instead of 3. The NameNode puts them in its re-replication queue, most endangered first (blocks with only 1 copy left go to the front).
03:02 → 04:30 · Re-replication
```
lost_tb = 12                       # one 12 TB disk, full of replicas
rate_mb = 40 * 50                  # 40 nodes copying at 50 MB/s each
print(round(lost_tb * 1024**2 / rate_mb / 3600, 1), "hours to restore 3 copies")
```
1.7 hours to restore 3 copies
Many nodes copy in parallel, each block from a surviving replica to a new node chosen with the rack rules. The NameNode limits how fast this goes so that normal jobs aren't starved.
Morning · A technician swaps the disk

The ops team gets an alert, replaces the disk, and the DataNode starts using it again (empty).
No data was lost, no job failed, and the replacement is not urgent: the cluster was healthy again within two hours on its own. That is “failures are normal” in practice.

5 See it happen

A node dies: what Riya sees, and how HDFS heals

A file with 4 blocks is stored on 6 nodes in 2 racks. Node dn2 dies. Left: Riya's commands. Right: the replicas.

Losing a node, keeping the data 6 steps

What you see

Behind the screen

Try it yourself: break the cluster on purpose

Every block here has 3 copies on 2 racks. Find out how much you can break before data is lost.

6 Real-world use

Living with failure every day

1. Disk replacement as routine work

Large companies replace failed disks every day as a normal ticket. The cluster has already re-copied the data, so the swap can wait for the next working day.

Disk fails→HDFS re-replicates→Swap tomorrow

2. A rack goes dark during maintenance

An electrician switches off the wrong power strip and a whole rack goes down. With rack-aware placement every block still has a copy on another rack: jobs slow a little, nothing is lost.

Rack offline→Copies on other racks→Jobs continue

3. Silent corruption caught by checksums

A worn disk returns a few wrong bytes without any error. The checksum check on read catches it; the client reads a good replica, reports the bad one, and HDFS replaces it. Without checksums, wrong numbers could reach a bank report.

Bad bytes→Checksum mismatch→Good replica used

4. The straggler that delays the report

A node with a dying disk runs tasks five times slower. Speculative execution starts a copy of its slow task elsewhere, and the report finishes on time. The monitoring team then removes the node for repair.

Slow task→Backup attempt→First one wins

5. A whole site down: disaster recovery

Regulated businesses such as banks and stock exchanges keep a disaster-recovery site in another city. Important HDFS data is copied there regularly, so if the main data centre is lost, work moves to the DR cluster.

Main DC→Scheduled copies→DR cluster ready

Examples are simplified. Exact procedures differ between companies.

7 Run the code

The arithmetic of failure

How many failures should a cluster expect? Multiply the number of devices by the yearly failure rate. Run it, then try the challenges, including the chance that every copy of a block is lost.

afr = {"disk": 0.015, "node": 0.04, "rack switch": 0.02}
count = {"disk": 3000, "node": 250, "rack switch": 10}

for part in afr:
    per_year = count[part] * afr[part]
    print(f"{part:12}: {per_year:5.1f} failures a year "
          f"(one every {365 / per_year:.0f} days)")

disk : 45.0 failures a year (one every 8 days) node : 10.0 failures a year (one every 36 days) rack switch : 0.2 failures a year (one every 1825 days)

afr = {"disk": 0.015, ...}Annualised failure rates (illustrative): 1.5% of disks, 4% of whole nodes, 2% of rack switches fail in a year.
per_year = count[part] * afr[part]Expected failures = devices × rate. 3,000 × 0.015 = 45.
365 / per_yearThe average gap between failures in days.

Run it on your PC: five years of failures, simulated

This Monte Carlo simulation places 20,000 blocks on 200 nodes with 1, 2 or 3 copies, then lets nodes fail at random for five simulated years. A failed node is repaired within a day. Each “block-loss event” is a day on which every copy of some block was on a dead node. Save it as loss_sim.py and run it.

Why HDFS keeps three copies

# loss_sim.py: how likely is data loss with 1, 2 or 3 copies? (a Monte Carlo simulation)
import random

random.seed(42)
NODES, BLOCKS = 200, 20_000
AFR = 0.04            # chance that a node fails in one year (disk, power supply, board …)
REPAIR_DAYS = 1       # time to re-copy a dead node's blocks
YEARS = 5

def lost_blocks(copies):
    placement = [random.sample(range(NODES), copies) for _ in range(BLOCKS)]
    lost = 0
    for day in range(365 * YEARS):
        # which nodes are down today (failed within the last REPAIR_DAYS)?
        down = {n for n in range(NODES) if random.random() < AFR / 365 * REPAIR_DAYS}
        if len(down) < copies:
            continue
        for p in placement:
            if all(n in down for n in p):
                lost += 1
    return lost

for c in [1, 2, 3]:
    print(f"{c} cop{'y' if c == 1 else 'ies'}: {lost_blocks(c):>6} block-loss events in {YEARS} years")

# In a terminal:  python loss_sim.py

1 copy: 4850 block-loss events in 5 years 2 copies: 4 block-loss events in 5 years 3 copies: 0 block-loss events in 5 years

Recorded with random.seed(42), so you will get the same numbers. Change the seed or the failure rate and run again: one copy always loses data, two copies sometimes do, three almost never. This model is simplified (it ignores racks and whole-rack failures), which is exactly why real HDFS also spreads copies over racks.

8 Line by line

Re-replication planning, one line at a time

A node has died. Watch the NameNode-style loop count the surviving copies of each block and decide how many new copies to make.

Worked examples: real problems, solved and traced in memory

Three questions a platform team answers about failures: how often, how long to recover, and whether the placement is safe. Each solution runs live and is traced in memory.

Example 1BeginnerOperations · planning

How many disk swaps per month?

Problem statement

A cluster has 80 nodes with 12 disks each. About 1.5% of disks fail per year. How many disk replacements should the ops team plan for each month, and how many spare disks should they keep for a quarter?

Input

80 × 12 disks · AFR 1.5%

Expected output

Disks: 960
Failures per month: 1.2
Spare disks for a quarter: 5

Approach

Total disks = nodes × disks per node.
Failures per year = disks × 0.015. Divide by 12 for a month.
Spares for a quarter = 3 months, rounded up, plus one for luck.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 2IntermediateReliability · recovery time

How long is the cluster “at risk” after a node dies?

Problem statement

A node with 100 TB of replicas dies. Re-replication can use 30 nodes copying at 60 MB/s each. Until it finishes, some blocks have only 2 copies. How many hours is that window, with and without waiting for the 10.5-minute timeout?

Input

100 TB · 30 nodes × 60 MB/s · timeout 630 s

Expected output

Copying       : 16.2 hours
With detection: 16.4 hours

Approach

Copy time = data ÷ total copy speed.
Add the detection timeout before copying starts.
Print both in hours.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 3AdvancedData safety · audit

Which blocks would a rack failure destroy?

Problem statement

An auditor wants to know: if any one rack fails, which blocks lose every copy? Check each rack in turn against the block placement and print the blocks at risk.

Input

racks r1 = {dn1, dn2, dn3}, r2 = {dn4, dn5} · B1 dn1 dn4 dn5 · B2 dn2 dn3 dn1 · B3 dn4 dn5 dn2

Expected output

If r1 fails, lost: ['B2']
If r2 fails, lost: []

Approach

For each rack, take the set of its nodes.
A block is lost with that rack if all of its nodes are in the rack (set subset).
Print the blocks at risk for each rack.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

9 Quiz and revision

Check yourself

Quick revision

Lesson 1.4 in one screen

At scale, failure is the norm: expected failures = devices × annual failure rate.
Chance of ≥1 failure among n devices: 1 − (1 − p)ⁿ. Grows fast with n.
HDFS: 3 replicas, on at least 2 racks. Lose a node → read another copy.
Dead node (≈10.5 min) → re-replication from surviving copies, most endangered blocks first.
Returned node → surplus replicas deleted.
Checksums (CRC32C per 512 bytes) + background block scanner catch corruption.
Tasks retried up to 4 times; speculative execution for stragglers.
NameNode HA for master failure; a DR cluster for a whole site.
Replication is not backup: use trash, snapshots and copies elsewhere.

Interview and viva questions

Why is “failure is the norm” in big data systems?

Because a cluster has thousands of disks, nodes, cables and switches. Even if each part rarely fails, the chance that some part fails on a given day is high: 3,000 disks at 1.5% a year is about 45 disk failures a year. Systems like HDFS are therefore designed to expect failures and recover automatically.

How does HDFS achieve fault tolerance?

It splits files into blocks and stores each block on several DataNodes (3 by default) across at least two racks. DataNodes send heartbeats and block reports; when a node or disk is lost, the NameNode re-replicates under-replicated blocks from surviving copies. Checksums detect corrupt replicas. Clients reading data simply switch to another replica if one fails. The NameNode itself is protected by High Availability.

What is the difference between replication and backup?

Replication keeps live copies of the current data to survive hardware failure. Every change, including a mistaken delete, is applied to all replicas. A backup is a separate, point-in-time copy (or a snapshot) that lets you go back to an earlier state after user errors, bugs or ransomware.

How does MapReduce handle a failed task?

The ApplicationMaster notices the failure (error, lost container, or no progress within the timeout) and reschedules the task attempt, preferably on another node. By default a task is tried up to 4 times; if all attempts fail the job fails. Completed map outputs on a lost node are re-run because they were stored on that node's local disk.

What is speculative execution?

When a task runs much slower than other tasks of the same job, the framework may launch a duplicate attempt on another node. Whichever attempt finishes first is used and the other is killed. It helps with stragglers caused by slow hardware, not with tasks that are slow because they have more data (skew).

Practice problems

Reliability calculations and checks, from simple counts to the audits real platform teams run.

P1Easy

Failures per week

A cluster has 5,000 disks with an annual failure rate of 2%. How many disk failures per week?

Input

5000 disks · 2%

Expected output

1.9 per week

Show solution and explanation

print(round(5000 * 0.02 / 52, 1), "per week")

1.9 per week

How it works: 100 failures a year ÷ 52 weeks ≈ 1.9 a week.

P2Easy

MTBF across the cluster

Each node fails on average once every 25 years (MTBF). With 500 nodes, how many days between node failures, on average?

Input

MTBF 25 years · 500 nodes

Expected output

18.2 days

Show solution and explanation

print(round(25 * 365 / 500, 1), "days")

18.2 days

How it works: 500 nodes × (1/25) = 20 failures a year, one every 18.25 days.

P3Easy

Copies left

A block has replicas on dn1, dn4, dn7. dn4 and dn7 die. How many copies are left, and is the block under-replicated?

Input

[dn1, dn4, dn7] · dead {dn4, dn7}

Expected output

Left: ['dn1'] | under-replicated: True

Show solution and explanation

replicas = ["dn1", "dn4", "dn7"]
dead = {"dn4", "dn7"}
left = [n for n in replicas if n not in dead]
print("Left:", left, "| under-replicated:", len(left) < 3)

Left: ['dn1'] | under-replicated: True

How it works: One copy left: the block is very endangered and goes to the front of the re-replication queue.

P4Medium

Find missing and under-replicated blocks

Given block placements and the dead nodes, print each block's status: OK, UNDER (1–2 live copies) or MISSING (0).

Input

B1 [1,2,3] · B2 [2,4,5] · B3 [4,5,6] · dead {4, 5, 6}

Expected output

B1 3 OK
B2 1 UNDER
B3 0 MISSING

Show solution and explanation

blocks = {"B1": [1, 2, 3], "B2": [2, 4, 5], "B3": [4, 5, 6]}
dead = {4, 5, 6}
for b, nodes in blocks.items():
    live = len([n for n in nodes if n not in dead])
    status = "OK" if live == 3 else "MISSING" if live == 0 else "UNDER"
    print(b, live, status)

B1 3 OK B2 1 UNDER B3 0 MISSING

How it works: This is what hdfs fsck reports. B3 is missing: all its copies were on the three dead nodes, probably the same rack.

P5Medium

Retry success rate

A flaky network makes 20% of task attempts fail. With up to 4 attempts, what share of tasks finally succeed?

Input

20% fail per attempt · 4 attempts

Expected output

Succeed: 99.84%

Show solution and explanation

p = 0.20
print(f"Succeed: {1 - p ** 4:.2%}")

Succeed: 99.84%

How it works: Fails only if all 4 attempts fail: 0.2⁴ = 0.16%. Retries turn an unreliable network into a reliable job.

P6Medium

Checksum check

Use Python's zlib.crc32 to compute a checksum of a data chunk, flip one character, and show that the checksum changes.

Input

b"order,city,amount"

Expected output

3993753734 581031960 False

Show solution and explanation

import zlib
good = b"order,city,amount"
bad = b"order,citx,amount"
print(zlib.crc32(good), zlib.crc32(bad), zlib.crc32(good) == zlib.crc32(bad))

3993753734 581031960 False

How it works: HDFS uses CRC32C (a close relative) for every 512 bytes. One changed byte gives a different checksum, so corruption is caught.

P7Business problem

Cost of not having rack awareness

Without rack awareness, about 2% of blocks end up with all copies in one rack. A rack fails twice a year and holds 1/8 of the cluster. The cluster has 50 lakh blocks. How many blocks are expected to become unavailable each year?

Input

50,00,000 blocks · 2% single-rack · 1/8 per rack · 2 rack failures/year

Expected output

Blocks unavailable per year: 25000

Show solution and explanation

blocks = 50_00_000
single_rack = blocks * 0.02
per_rack = single_rack / 8
print("Blocks unavailable per year:", int(per_rack * 2))

Blocks unavailable per year: 25000

How it works: About 25,000 blocks a year become unreadable until the rack is fixed, each one possibly breaking a job or a dashboard. A topology script fixes this for free.

P8Business problem

How much extra disk does safety cost?

The business stores 400 TB. Compare the raw disk needed with replication 2 and 3, and the extra cost at Rs 2,000 per TB of raw disk.

Input

400 TB · repl 2 vs 3 · Rs 2,000/TB

Expected output

replication 2: 800 TB raw, Rs 16 lakh
replication 3: 1200 TB raw, Rs 24 lakh
Extra for the 3rd copy: Rs 8 lakh

Show solution and explanation

data, price = 400, 2000
for r in [2, 3]:
    print(f"replication {r}: {data * r} TB raw, Rs {data * r * price / 1e5:.0f} lakh")
print(f"Extra for the 3rd copy: Rs {data * price / 1e5:.0f} lakh")

replication 2: 800 TB raw, Rs 16 lakh replication 3: 1200 TB raw, Rs 24 lakh Extra for the 3rd copy: Rs 8 lakh

How it works: The third copy costs Rs 8 lakh here (illustrative price), but it is what makes data loss practically impossible. Erasure coding (Module 7) cuts this for cold data.

Real-world scenario questions (asked in company interviews)

Failure situations interviewers love, because they reveal whether you understand how Hadoop actually behaves.

Platform team · incidentFresher

A DataNode died. Should we restart everything?

At 11 pm the monitoring shows one DataNode down out of 60. A junior admin wants to restart the whole cluster “to be safe”. You are on call. What do you do?

The interviewer is checking: do you trust and understand HDFS's self-healing, and avoid making things worse?

Show a strong answer

Don't restart the cluster. That would stop all jobs and put the NameNode in safe mode while it waits for block reports.
Check hdfs dfsadmin -report and hdfs fsck /: expect under-replicated blocks but 0 missing.
After the timeout, HDFS re-replicates automatically. Watch the under-replicated count fall.
Find out why the node died (logs, hardware alerts), and fix or replace it during working hours.

Avoid: panicking, or manually copying blocks around.

Banking · data safety1–3 years

“We have 3 replicas, so we don't need backups”

In a design review for a bank's reporting cluster, a manager says backups are unnecessary because HDFS already stores everything three times.

The interviewer is checking: do you know what replication protects against, and what it doesn't?

Show a strong answer

Replication protects against hardware failures: disks, nodes, a rack.
It does not protect against a user deleting data, a buggy job overwriting it, ransomware, or losing the whole data centre: every change reaches all replicas.
Use HDFS trash (deleted files kept for a while), snapshots of important directories, and regular copies to a second cluster or cloud storage (DistCp). Regulated industries usually require a DR site anyway.

Avoid: agreeing that replication equals backup.

E-commerce · incident1–3 years

fsck says “CORRUPT”: 12 missing blocks

After a power problem in one row of racks, hdfs fsck / reports 12 missing blocks in yesterday's sales folder. Five nodes are still down.

The interviewer is checking: can you investigate missing blocks calmly, and do you know they may come back?

Show a strong answer

Missing usually means “no live replica right now”, not “destroyed”. Bring the five nodes back: when they send block reports, the blocks often reappear.
Use hdfs fsck / -list-corruptfileblocks and -files -blocks -locations to see which files and nodes are involved.
If nodes are truly lost, restore the affected files from the source system, a snapshot or the DR copy, then delete the broken files.
Afterwards, ask why all copies were in the failed racks: is rack awareness configured correctly?

Avoid: running fsck -delete straight away. It removes the files before you know whether the blocks can come back.

Any company · conceptsFresher

“Why not just buy servers that never fail?”

The interviewer asks: “Wouldn't it be simpler to buy top-quality servers with RAID and dual everything, so failures don't happen?”

The interviewer is checking: do you understand the economics behind Hadoop's design?

Show a strong answer

No hardware never fails. With thousands of machines, even very reliable ones fail regularly.
Highly reliable servers cost much more per TB and per core, and you still need a plan for when they fail.
Hadoop's idea: use affordable commodity servers and make failures harmless in software (replicas, retries, re-replication). Adding capacity stays cheap.
Spend extra on the few components that matter most: master nodes, network core, power.

Avoid: saying commodity hardware is “bad hardware”. It is ordinary, well-supported hardware without a premium for extra reliability.

The night Riya's laptop died

Failures are normal: fault tolerance

Words you'll meet in this lesson

Typical numbers (approximate)

Failures at three levels

Beginner The tiffin delivery team

Intermediate The arithmetic of “normal”

Advanced Failures that are harder than “a node dies”

What fails, and how Hadoop survives it

Four ways to tolerate faults

Redundancy

Detection

Recovery

Isolation

A disk dies at 3 am: what happens, minute by minute

03:00 · The disk stops responding

03:00 · A job was reading from it

03:01 · The NameNode learns what was lost

03:02 → 04:30 · Re-replication

Morning · A technician swaps the disk

A node dies: what Riya sees, and how HDFS heals

Try it yourself: break the cluster on purpose

Living with failure every day

1. Disk replacement as routine work

2. A rack goes dark during maintenance

3. Silent corruption caught by checksums

4. The straggler that delays the report

5. A whole site down: disaster recovery

The arithmetic of failure

Run it on your PC: five years of failures, simulated

Why HDFS keeps three copies

Re-replication planning, one line at a time

Worked examples: real problems, solved and traced in memory

How many disk swaps per month?

How long is the cluster “at risk” after a node dies?

Which blocks would a rack failure destroy?

Check yourself

Quick revision

Lesson 1.4 in one screen

Interview and viva questions

Practice problems

Failures per week

MTBF across the cluster

Copies left

Find missing and under-replicated blocks

Retry success rate

Checksum check

Cost of not having rack awareness

How much extra disk does safety cost?

Real-world scenario questions (asked in company interviews)

A DataNode died. Should we restart everything?

“We have 3 replicas, so we don't need backups”

fsck says “CORRUPT”: 12 missing blocks

“Why not just buy servers that never fail?”