Replication and rack awareness

1 The story

Three copies, but not three copies anywhere

The lab cluster has two racks: datanode-1 and datanode-2 in /rack1, datanode-3 and datanode-4 in /rack2. Riya checks the five blocks of the canteen file. Every block has three copies, and every block has them in both racks: two in one rack, one in the other. Never all three together. Then Kabir pulls the plug on datanode-1. A minute later, the NameNode quietly makes new copies elsewhere. Nobody lost a single order.

Riya

Why not put the three copies on three different racks? Wouldn't that be safest?

Arjun Sir

Safest, but slowest. Data crossing between racks goes through the core switch, which is shared and limited. HDFS's default rule is a compromise: one copy on one rack, two copies on another rack. Lose a whole rack and you still have a copy, but a write crosses racks only once.

Kabir

And when I killed datanode-1?

Arjun Sir

The NameNode stopped hearing its heartbeats, waited, declared it dead, noticed some blocks had only two copies, and asked other DataNodes to copy them. When you switched datanode-1 back on, its old copies became extra, and the NameNode deleted them. All of it automatic. That's replication: the reason Hadoop could run on cheap machines that fail.

2 Definition

Replication and rack awareness

Formal definition (write this in exams)

HDFS stores each block as several replicas on different DataNodes; the replication factor (dfs.replication, default 3) is set per file and can be changed with hdfs dfs -setrep. Rack awareness means the NameNode knows each DataNode's rack, from a topology script or mapping (net.topology.script.file.name); without one, every node is in /default-rack. The default block placement policy (BlockPlacementPolicyDefault) puts the first replica on the writer's node if the writer is a DataNode (otherwise on a random node), the second on a node in a different rack, and the third on a different node in the same rack as the second; further replicas go to random nodes, with at most two replicas per rack where possible. The NameNode continuously compares live replicas with the target: under-replicated blocks are copied, over-replicated blocks have extra replicas deleted.

In simple words

Every block is kept three times on three machines, spread over two racks, so a broken disk, machine or even a whole rack doesn't lose data. If a copy disappears, HDFS makes a new one by itself.

Words you'll meet in this lesson

ReplicaOne stored copy of a block on one DataNode.

Replication factorTarget number of replicas for a file's blocks (the “3” in hdfs dfs -ls).

RackA group of nodes sharing a switch; a common failure and bandwidth boundary.

Topology scriptA program that tells Hadoop the rack of each DataNode IP or host name.

Under-replicatedFewer live replicas than the target. The NameNode schedules copies.

Over-replicatedMore replicas than the target, e.g. after a node returns. Extras are deleted.

Mis-replicatedEnough replicas, but placed against the rack rule (e.g. all in one rack).

setrephdfs dfs -setrep [-w] N path: change a file's replication factor.

Replication facts (Hadoop 3.5.0)

Default factor	`dfs.replication` = 3; minimum for a write to succeed: `dfs.namenode.replication.min` = 1
Placement policy	`BlockPlacementPolicyDefault`: writer's node, other rack, same rack as 2nd
Rack mapping	`net.topology.script.file.name`; none set → `/default-rack` for all
Dead node detected	After 2 × `recheck-interval` (5 min) + 10 × heartbeat (3 s) = 10 min 30 s by default
Copy speed limits	`dfs.namenode.replication.max-streams` = 2 copies per DataNode at a time (more for high-priority blocks)
Storage cost	Replication 3 = 3× raw disk (200% overhead); erasure coding is the alternative (Lesson 7.3)

Replication at three levels

Pick your level. New to this? Start at Beginner and climb.

Beginner Important documents in three lockers

You keep three photocopies of your marksheet: one at home, two at your uncle's house in another city.
If your house floods (a rack fails), the uncle's copies survive. If one copy at the uncle's tears, you still have two.
Whenever a copy is lost, you make a new photocopy straight away. That's re-replication.

Intermediate Why this exact placement

Write cost: the pipeline crosses racks only once (between replica 1 and replica 2).
Rack failure: two racks hold copies, so losing one rack never loses a block.
Read speed: readers on two racks find a nearby copy; with locality, tasks run on a node holding a replica.
With no topology script, HDFS thinks everything is in one rack and spreads copies only across nodes.

Advanced What the NameNode does continuously

Keeps queues of low-redundancy blocks by priority: a block with 1 live replica left is repaired before one with 2.
Chooses a source DataNode and a target (following the rack rules) and sends the copy command in a heartbeat reply.
When a dead node returns, its replicas make blocks over-replicated; extras are deleted (“Pending deletion blocks”).
Decommissioning a node first copies all its blocks elsewhere, then lets it leave: the safe way to remove hardware.
Replication factor 1 is allowed (scratch data), but a single disk failure then loses blocks.

Exam tip: For “Explain rack awareness / replica placement”: draw two racks; replica 1 on the writer's node (or random), replica 2 on another rack, replica 3 on another node of replica 2's rack; state the benefits (rack-failure tolerance, one cross-rack transfer per write, read locality) and how racks are configured (net.topology.script.file.name).

3 Types and ways to use it

Replication factors and placement choices

Factor	Raw disk	Survives	Typical use
1	1×	Nothing: one disk failure loses blocks	Temporary or easily regenerated data; pseudo-distributed mode
2	2×	One failure	Less important data on small clusters
3 (default)	3×	Two node failures, or a whole rack	Almost everything
5–10	5–10×	Many failures; more nodes can read locally	Small, very popular files (job jars, lookup tables)

How racks are told to Hadoop

Topology script

A small script returns a rack per IP or host. The method used in this lesson.

Most common

Table mapping

TableMapping reads a two-column file: host, rack.

Simple

Cluster manager

Cloudera Manager / Ambari let admins assign racks per host in the UI.

Managed

Cloud

Racks are often mapped to availability zones or fault domains.

Cloud

Rack awareness only helps if it matches reality. A topology script that puts every node in one rack, or maps nodes to the wrong racks, gives a false sense of safety. Check it with hdfs dfsadmin -printTopology.

4 Step by step

Watch placement, setrep and re-replication on the real cluster

The Module 5 lab cluster from Lesson 5.1: 4 DataNodes, 2 racks, dead-node check every 15 s (so a node is declared dead after about 60 s instead of 10 min 30 s).

Check the racks
```
docker compose exec namenode hdfs dfsadmin -printTopology
```
Rack: /rack1 172.19.0.4:9866 (lab5-datanode-2.lab5_default) In Service 172.19.0.5:9866 (lab5-datanode-1.lab5_default) In Service Rack: /rack2 172.19.0.7:9866 (lab5-datanode-4.lab5_default) In Service 172.19.0.6:9866 (lab5-datanode-3.lab5_default) In Service
The racks come from topology.sh (Lesson 5.1, step 1), which the NameNode runs for each DataNode address.
See where each replica went
```
docker compose exec namenode hdfs fsck /user/hadoop/data/canteen_1mb.csv -files -blocks -racks
```
0. …:blk_1073741826_1002 len=1048576 Live_repl=3 [/rack1/172.19.0.5:9866, /rack2/172.19.0.6:9866, /rack2/172.19.0.7:9866] 1. …:blk_1073741827_1003 len=1048576 Live_repl=3 [/rack1/172.19.0.5:9866, /rack1/172.19.0.4:9866, /rack2/172.19.0.7:9866] 2. …:blk_1073741828_1004 len=1048576 Live_repl=3 [/rack2/172.19.0.6:9866, /rack2/172.19.0.7:9866, /rack1/172.19.0.5:9866] 3. …:blk_1073741829_1005 len=1048576 Live_repl=3 [/rack2/172.19.0.6:9866, /rack2/172.19.0.7:9866, /rack1/172.19.0.5:9866] 4. …:blk_1073741830_1006 len=497705 Live_repl=3 [/rack1/172.19.0.5:9866, /rack1/172.19.0.4:9866, /rack2/172.19.0.6:9866] Number of racks: 2
Every block: replicas in both racks, never more than two in one rack. (Our client ran in the namenode container, which is not a DataNode, so the first replica went to a random node.)
Lower and raise the replication factor
```
docker compose exec namenode hdfs dfs -setrep -w 2 data/canteen_orders.csv
docker compose exec namenode hdfs fsck /user/hadoop/data/canteen_orders.csv -files -blocks -racks | grep blk_
docker compose exec namenode hdfs dfs -setrep -w 3 data/canteen_orders.csv
docker compose exec namenode hdfs fsck /user/hadoop/data/canteen_orders.csv -files -blocks -racks | grep blk_
```
Replication 2 set: data/canteen_orders.csv Waiting for data/canteen_orders.csv ... WARNING: the waiting time may be long for DECREASING the number of replications. . done 0. …:blk_1073741825_1001 len=4692009 Live_repl=2 [/rack2/172.19.0.6:9866, /rack1/172.19.0.5:9866] Replication 3 set: data/canteen_orders.csv Waiting for data/canteen_orders.csv .... done 0. …:blk_1073741825_1001 len=4692009 Live_repl=3 [/rack2/172.19.0.6:9866, /rack1/172.19.0.5:9866, /rack2/172.19.0.7:9866]
Down to 2: one replica deleted, still one per rack. Back to 3: a new copy made on a second node in rack 2. -w waits until the NameNode reports the target reached.
Stop a DataNode and watch
```
docker stop lab5-datanode-3
# every few seconds:
docker compose exec namenode hdfs dfsadmin -report | grep "^Live datanodes"
docker compose exec namenode hdfs fsck / | grep "Under-replicated blocks"
```
t=0s Live datanodes (4) Under-replicated blocks: 0 (0.0 %) t=27s Live datanodes (4) Under-replicated blocks: 0 (0.0 %) t=49s Live datanodes (4) Under-replicated blocks: 0 (0.0 %) t=56s Live datanodes (3) Under-replicated blocks: 0 (0.0 %) t=63s Live datanodes (3) Under-replicated blocks: 4 (50.0 %) t=70s Live datanodes (3) Under-replicated blocks: 0 (0.0 %) t=111s Live datanodes (3) Under-replicated blocks: 0 (0.0 %)
For about 50 s nothing changes: the NameNode is waiting, because a node that missed a few heartbeats may just be slow. Once the node is declared dead, 4 of the 8 blocks are under-replicated, and within seconds they have three copies again. (Real timings; with default settings the wait is 10 min 30 s.)
Read the NameNode's own log
```
docker compose logs namenode | grep -E "removeDeadDatanode|Removing a node"
```
INFO StateChange:927 - BLOCK* removeDeadDatanode: lost heartbeat from 172.19.0.6:9866, removeBlocksFromBlockMap true INFO NetworkTopology:227 - Removing a node: /rack2/172.19.0.6:9866
“lost heartbeat”: the NameNode removes the node from its topology and from the block map, which is what makes those blocks under-replicated.
Bring a node back: extra copies are deleted
```
docker start lab5-datanode-1      # stopped earlier, its blocks were already re-replicated
docker compose exec namenode hdfs dfsadmin -report | grep "Pending deletion blocks"
```
Pending deletion blocks: 7 … Pending deletion blocks: 0
datanode-1 returned with old replicas of blocks that already had three copies. They were over-replicated, so the NameNode scheduled 7 deletions, and a few seconds later there were none pending.

Stopping more nodes than the replication factor at once (for example, two nodes holding the only two replicas of a replication-2 file) loses data until they return. For maintenance, use decommissioning or maintenance mode, and take nodes down one rack at a time at most.

5 See it happen

A node dies, the cluster heals

The real timeline from step 4, frame by frame.

Re-replication after datanode-3 stops 6 steps

What you see

Behind the screen

Try it yourself: break nodes and racks

The playground follows the same default placement policy and the same dead-node timing (sped up). Watch the Event log while you break things.

6 Real-world use

Replication in real clusters

1. Surviving a rack outage

A top-of-rack switch or power strip failure takes 20–40 servers offline at once. With correct rack mapping, every block still has a copy in another rack, and jobs keep running while the rack is repaired.

Rack down→Copies in other racks→No data loss

2. Extra copies for hot files

Files read by every task (job jars, small lookup tables) get a higher replication factor so many nodes can read them locally. MapReduce does this automatically for job files (mapreduce.client.submit.file.replication, default 10).

Hot file→More replicas→Local reads everywhere

3. Retiring servers safely

Before old servers are removed, admins decommission them: HDFS copies every replica they hold to other nodes, and only then marks them safe to switch off.

Exclude node→Re-replicate→Remove hardware

4. Cheaper protection for cold data

Replication triples storage. Large platforms move rarely-read data to erasure coding, which protects it with about 1.5× storage instead of 3× (Lesson 7.3).

Hot: replication→Cold: erasure coding→Half the disk

Simplified; organisations differ in tools and policies.

7 Run the code

Simulate the default placement policy

This program places three replicas of each block with HDFS's default rule on a cluster of two racks, then checks the result the way fsck would: at least two racks, at most two replicas per rack.

import random
random.seed(7)
racks = {"/rack1": ["dn1", "dn2"], "/rack2": ["dn3", "dn4"]}
rack_of = {dn: r for r, nodes in racks.items() for dn in nodes}

def place(writer=None):
    first = writer if writer in rack_of else random.choice(list(rack_of))
    other_racks = [r for r in racks if r != rack_of[first]]
    second = random.choice(racks[random.choice(other_racks)])
    third = random.choice([d for d in racks[rack_of[second]] if d != second])
    return [first, second, third]

for blk in ["blk_A", "blk_B", "blk_C"]:
    replicas = place()
    used = [rack_of[d] for d in replicas]
    ok = len(set(used)) >= 2 and max(used.count(r) for r in used) <= 2
    print(blk, replicas, used, "OK" if ok else "BAD")

blk_A ['dn3', 'dn2', 'dn1'] ['/rack2', '/rack1', '/rack1'] OK blk_B ['dn1', 'dn4', 'dn3'] ['/rack1', '/rack2', '/rack2'] OK blk_C ['dn2', 'dn3', 'dn4'] ['/rack1', '/rack2', '/rack2'] OK

first = writer if writer in rack_of else random.choice(list(rack_of))Replica 1: on the writer's own node if the writer is a DataNode (fast, local write); otherwise a random node.
second = random.choice(racks[random.choice(other_racks)])Replica 2: any node in a different rack, so a rack failure can't take everything.
third = random.choice([d for d in racks[rack_of[second]] if d != second])Replica 3: a different node in the same rack as replica 2, so the data crosses racks only once.

Run it on your PC: check every block's rack spread

This script runs fsck -racks on the lab cluster and checks each block against the placement rules. Save it as rack_check.py in the hdfs-lab folder and run it while the cluster is up.

Rack spread of every block under /user/hadoop/data

# rack_check.py: does every block follow the rack-aware placement rules?
# Run in the hdfs-lab folder while the cluster is up.
import re
import subprocess
from collections import Counter

cmd = ["docker", "compose", "exec", "-T", "namenode",
       "hdfs", "fsck", "/user/hadoop/data", "-files", "-blocks", "-racks"]
out = subprocess.run(cmd, capture_output=True, text=True).stdout

for line in out.splitlines():
    m = re.search(r"(blk_\d+)_\d+ len=(\d+).*\[(.*)\]", line)
    if not m:
        continue
    blk, racks = m.group(1), re.findall(r"(/rack\d+)/", m.group(3))
    per_rack = Counter(racks)
    ok = len(per_rack) >= 2 and max(per_rack.values()) <= 2
    print(f"{blk}: {dict(per_rack)}  {'OK' if ok else 'CHECK'}")

# In the hdfs-lab folder:
#   python3 rack_check.py

blk_1073741826: {'/rack1': 2, '/rack2': 1} OK blk_1073741827: {'/rack2': 2, '/rack1': 1} OK blk_1073741828: {'/rack2': 2, '/rack1': 1} OK blk_1073741829: {'/rack1': 2, '/rack2': 1} OK blk_1073741830: {'/rack1': 2, '/rack2': 1} OK blk_1073741825: {'/rack1': 2, '/rack2': 1} OK

Real output, taken after the DataNode stop/start experiments in Part 4: re-replication kept the rack rule for every block. Run it after your own experiments; a “CHECK” line means a block is in only one rack (mis-replicated).

8 Line by line

How the NameNode reacts to a dead DataNode

Follow the block map when dn3 is declared dead: which blocks become under-replicated, and where the new copies go.

Worked examples: real problems, solved and traced in memory

Three replication questions engineers answer with numbers: what replication costs in disk, how long re-replication takes after a failure, and whether a planned maintenance window is safe. Each runs live and is traced in memory.

Example 1BeginnerCapacity · disk

Raw disk for different replication factors

Problem statement

A bank keeps 200 TB of transaction history. How much raw disk is needed with replication 1, 2 and 3, keeping 20% of disk free?

Input

200 TB · replication 1, 2, 3 · 20% free

Expected output

replication 1: 200 TB raw, buy 250 TB
replication 2: 400 TB raw, buy 500 TB
replication 3: 600 TB raw, buy 750 TB

Approach

Raw = data × replication.
Disk to buy = raw / 0.8.
Compare the three.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 2IntermediateOperations · recovery

How long does re-replication take?

Problem statement

A DataNode with 24 TB of block replicas dies. 40 other DataNodes help, each copying at about 100 MB/s in total for re-replication. Roughly how long until every block has 3 replicas again?

Input

24 TB lost · 40 nodes · 100 MB/s each

Expected output

about 105 minutes (1.7 hours)

Approach

Total bandwidth = nodes × rate.
Time = data / bandwidth.
Convert to minutes.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 3AdvancedMaintenance · risk

Is it safe to take these nodes down?

Problem statement

You want to reboot dn2 and dn5 together. Given each block's replica locations, list the blocks that would have no live replica during the reboot, and those left with only one.

Input

5 blocks with replica lists · nodes down: dn2, dn5

Expected output

unavailable: ['b3']
only one copy left: ['b2']

Approach

For each block, remove the nodes going down.
Zero left → data unavailable; one left → at risk.
Report both lists before approving the change.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

9 Quiz and revision

Check yourself

Quick revision

Lesson 5.3 in one screen

Replication factor: dfs.replication = 3, per file; change with hdfs dfs -setrep [-w] N.
Default placement: 1st on writer's node (or random), 2nd on another rack, 3rd on another node of the 2nd's rack.
Benefits: survives a rack failure; only one cross-rack transfer per write; nearby replicas for reads.
Racks come from net.topology.script.file.name; otherwise /default-rack. Check: hdfs dfsadmin -printTopology.
Dead node: 2 × recheck (5 min) + 10 × heartbeat (3 s) = 10 min 30 s by default.
Under-replicated → copied; over-replicated → extras deleted; mis-replicated → wrong rack spread.
Real lab: dead at ~56 s (15 s recheck), 4 under-replicated blocks repaired within 7 s.
Replication 3 = 3× raw disk; erasure coding ≈ 1.5× for cold data.

Interview and viva questions

What is rack awareness in Hadoop?

The NameNode knows which rack each DataNode is in (from a topology script or mapping) and uses it to place replicas: by default one replica on one rack and two on another. This protects data against a whole rack failing while limiting cross-rack network traffic, and lets schedulers prefer rack-local data.

How does HDFS place the three replicas of a block?

The first replica goes on the writer's node if the writer is a DataNode, otherwise on a random node; the second on a node in a different rack; the third on a different node in the same rack as the second. More replicas go to random nodes, keeping at most two per rack where possible.

What happens when a DataNode fails?

The NameNode stops receiving its heartbeats. After the dead-node interval (10 min 30 s by default) it marks the node dead, removes its replicas from the block map, and schedules copies of the now under-replicated blocks from surviving replicas to other DataNodes.

How do I change the replication of a file?

Use hdfs dfs -setrep N path (add -w to wait until the change is complete; giving a directory applies it to every file below it). New files follow dfs.replication of the writing client.

What is the difference between under-replicated and missing blocks?

An under-replicated block still has at least one live replica, so it is readable and HDFS will copy it. A missing block has no live replica at all: its data cannot be read until a node holding it returns.

Is replication a backup?

No. Replication protects against hardware failure, but deletes and bad writes are applied to every replica. Backups, snapshots and DistCp copies to another cluster protect against those.

How is the rack of a DataNode configured?

Set net.topology.script.file.name in core-site.xml to a script that prints one rack path (e.g. /dc1/rack07) for each IP or host name it is given, or use TableMapping with a host-to-rack file.

Practice problems

Replication and placement exercises, including checks on the real fsck output from the lab.

P1Easy

Replicas on disk

A 640 MB file with 128 MB blocks and replication 3: how many block replicas are stored in total?

Input

640 MB · 128 MB · 3

Expected output

15 replicas

Show solution and explanation

print(640 // 128 * 3, "replicas")

15 replicas

How it works: 5 blocks × 3 = 15 replicas.

P2Easy

Dead-node timer

Compute the dead-node time for recheck-interval 300 s and heartbeat 3 s.

Input

300 s · 3 s

Expected output

630 seconds

Show solution and explanation

print(2 * 300 + 10 * 3, "seconds")

630 seconds

How it works: 630 seconds = 10 minutes 30 seconds.

P3Easy

Racks per block

Count the racks used by each replica list from the real fsck output.

Input

[rack1, rack2, rack2] · [rack1, rack1, rack2]

Expected output

['/rack1', '/rack2', '/rack2'] -> 2 racks
['/rack1', '/rack1', '/rack2'] -> 2 racks

Show solution and explanation

for reps in [["/rack1", "/rack2", "/rack2"], ["/rack1", "/rack1", "/rack2"]]:
    print(reps, "->", len(set(reps)), "racks")

['/rack1', '/rack2', '/rack2'] -> 2 racks ['/rack1', '/rack1', '/rack2'] -> 2 racks

How it works: Both use 2 racks with 2 replicas in one of them: exactly the default policy.

P4Medium

Mis-replicated check

Flag blocks whose replicas are all in one rack or have more than 2 replicas in one rack.

Input

three replica-rack lists

Expected output

b1 OK
b2 MIS-REPLICATED
b3 OK

Show solution and explanation

from collections import Counter
blocks = {"b1": ["r1", "r2", "r2"], "b2": ["r1", "r1", "r1"], "b3": ["r2", "r1", "r1"]}
for b, rs in blocks.items():
    c = Counter(rs)
    print(b, "OK" if len(c) >= 2 and max(c.values()) <= 2 else "MIS-REPLICATED")

b1 OK b2 MIS-REPLICATED b3 OK

How it works: b2 has all three replicas in rack r1: a rack failure would make it unavailable.

P5Medium

Lost rack

For each block, how many replicas survive if rack r1 is lost?

Input

b1: r1,r2,r2 · b2: r1,r1,r2 · b3: r1,r1,r1

Expected output

b1 2 left
b2 1 left
b3 0 left

Show solution and explanation

blocks = {"b1": ["r1", "r2", "r2"], "b2": ["r1", "r1", "r2"], "b3": ["r1", "r1", "r1"]}
for b, rs in blocks.items():
    print(b, sum(r != "r1" for r in rs), "left")

b1 2 left b2 1 left b3 0 left

How it works: b1 keeps 2, b2 keeps 1, b3 keeps 0: the mis-replicated block is the only one lost.

P6Medium

Cross-rack bytes

A 1 GB file is written with the default policy by a client on a DataNode. How many GB cross racks? How many if replication is 2?

Input

1 GB · replication 3 and 2

Expected output

replication 3: ['dn1', 'dn3', 'dn4'] -> 1 GB crosses racks
replication 2: ['dn1', 'dn3'] -> 1 GB crosses racks

Show solution and explanation

rack = {"dn1": "r1", "dn3": "r2", "dn4": "r2"}
for pipeline in [["dn1", "dn3", "dn4"], ["dn1", "dn3"]]:
    hops = sum(rack[a] != rack[b] for a, b in zip(pipeline, pipeline[1:]))
    print(f"replication {len(pipeline)}: {pipeline} -> {hops} GB crosses racks")

replication 3: ['dn1', 'dn3', 'dn4'] -> 1 GB crosses racks replication 2: ['dn1', 'dn3'] -> 1 GB crosses racks

How it works: The write pipeline dn1 → dn3 → dn4 has one cross-rack hop (dn1 → dn3), so 1 GB crosses racks with replication 3 or 2; the dn3 → dn4 hop stays inside rack r2.

P7Business problem

Cost of replication

Disk costs Rs 3,000 per TB. 500 TB of data: compare the disk cost of replication 3 with erasure coding at 1.5×.

Input

500 TB · Rs 3,000/TB · 3× vs 1.5×

Expected output

replication 3  : Rs 45.0 lakh
erasure coding : Rs 22.5 lakh

Show solution and explanation

for name, f in [("replication 3", 3), ("erasure coding", 1.5)]:
    print(f"{name:15}: Rs {500 * f * 3000 / 1e5:.1f} lakh")

replication 3 : Rs 45.0 lakh erasure coding : Rs 22.5 lakh

How it works: Rs 45 lakh against Rs 22.5 lakh of raw disk: why cold data moves to erasure coding.

P8Advanced

Placement for a writer on a DataNode

Write a function placing 3 replicas for a writer on dn4 with racks r1=[dn1,dn2] and r2=[dn3,dn4], and check the rule.

Input

writer dn4

Expected output

['dn4', 'dn1', 'dn2'] ['r2', 'r1', 'r1']

Show solution and explanation

racks = {"r1": ["dn1", "dn2"], "r2": ["dn3", "dn4"]}
rack_of = {d: r for r, ds in racks.items() for d in ds}
def place(writer):
    first = writer
    second = racks[[r for r in racks if r != rack_of[first]][0]][0]
    third = [d for d in racks[rack_of[second]] if d != second][0]
    return [first, second, third]
p = place("dn4")
print(p, [rack_of[d] for d in p])

['dn4', 'dn1', 'dn2'] ['r2', 'r1', 'r1']

How it works: [dn4, dn1, dn2] in racks [r2, r1, r1]: local first replica, then two in the other rack.

Real-world scenario questions (asked in company interviews)

Replication questions test whether you can reason about failures and their cost.

HDFS · placementFresher

“Explain where the three replicas of a block go, and why”

Probably the most common HDFS interview question.

The interviewer is checking: the rule and its reasons.

Show a strong answer

Replica 1 on the writer's node (or a random node if the client is outside the cluster), replica 2 on another rack, replica 3 on a different node in replica 2's rack.
Reasons: survives a whole rack failure; the write crosses racks only once; reads can find a nearby copy.
Requires a correct topology script; otherwise everything is in /default-rack.

Operations · failureExperienced

A DataNode died 2 minutes ago. Why is nothing being re-replicated?

A junior engineer sees a DataNode process crash and expects immediate copying.

The interviewer is checking: the dead-node timeout and why it exists.

Show a strong answer

The NameNode waits 10 min 30 s by default (2 × recheck interval + 10 heartbeats) before declaring a node dead.
The delay avoids copying terabytes every time a node is briefly slow or restarting.
Meanwhile the node is marked stale after 30 s, and reads can avoid it if dfs.namenode.avoid.read.stale.datanode is enabled.
If the node won't return, re-replication starts automatically after the timeout.

Design · racksExperienced

After a rack power failure, some files were unreadable

The cluster has replication 3, yet a single rack outage made several files unavailable.

The interviewer is checking: that you suspect the rack mapping.

Show a strong answer

With correct rack awareness that can't happen, so check hdfs dfsadmin -printTopology: probably no topology script, or nodes mapped to the wrong racks.
Also check files with replication 1 or 2, and mis-replicated blocks in fsck.
Fix the mapping; then fsck shows mis-replicated blocks, which the NameNode corrects over time (or run the balancer).

Cost · storageFresher

“Can we save money with replication 2?”

A manager wants to reduce disk costs by a third.

The interviewer is checking: balanced judgement about risk.

Show a strong answer

Replication 2 survives only one failure; a second disk failure during re-replication loses blocks, and rack failure protection becomes weaker.
Better options: erasure coding for cold data (≈1.5×), deleting or archiving unused data, compressing with efficient formats.
Replication 2 can be acceptable for data that can be regenerated quickly (intermediate results).

Replication and rack awareness

Three copies, but not three copies anywhere

Replication and rack awareness

Words you'll meet in this lesson

Replication facts (Hadoop 3.5.0)

Replication at three levels

Beginner Important documents in three lockers

Intermediate Why this exact placement

Advanced What the NameNode does continuously

Replication factors and placement choices

How racks are told to Hadoop

Topology script

Table mapping

Cluster manager

Cloud

Watch placement, setrep and re-replication on the real cluster

Check the racks

See where each replica went

Lower and raise the replication factor

Stop a DataNode and watch

Read the NameNode's own log

Bring a node back: extra copies are deleted

A node dies, the cluster heals

Try it yourself: break nodes and racks

Replication in real clusters

1. Surviving a rack outage

2. Extra copies for hot files

3. Retiring servers safely

4. Cheaper protection for cold data

Simulate the default placement policy

Run it on your PC: check every block's rack spread

Rack spread of every block under /user/hadoop/data

How the NameNode reacts to a dead DataNode

Worked examples: real problems, solved and traced in memory

Raw disk for different replication factors

How long does re-replication take?

Is it safe to take these nodes down?

Check yourself

Quick revision

Lesson 5.3 in one screen

Interview and viva questions

Practice problems

Replicas on disk

Dead-node timer

Racks per block

Mis-replicated check

Lost rack

Cross-rack bytes

Cost of replication

Placement for a writer on a DataNode

Real-world scenario questions (asked in company interviews)

“Explain where the three replicas of a block go, and why”

A DataNode died 2 minutes ago. Why is nothing being re-replicated?

After a rack power failure, some files were unreadable

“Can we save money with replication 2?”