Paying three times for data nobody reads
The university's finance office looks at the storage bill. Five years of old exam scans, canteen logs and CCTV exports sit in HDFS, read maybe once a year for audits. Every byte is stored three times. 400 TB of data uses 1.2 PB of disk. “Can't we keep fewer copies of the old stuff?” Riya says replication 2 would be risky. Arjun Sir has a better idea, borrowed from the same mathematics that lets a scratched DVD still play.
Since Hadoop 3, HDFS supports erasure coding. Instead of copying a block three times, it cuts the data into pieces and computes extra parity pieces with Reed-Solomon mathematics. With the policy RS-3-2, 3 data pieces get 2 parity pieces: any 2 of the 5 can be lost, and the data can still be rebuilt.
Same safety as three copies, but 167% instead of 300%?
For large, cold files, yes. But nothing is free: reading after a failure means calculating the missing data, a small file gains nothing, and you need enough DataNodes and racks. Today we store the same file both ways on our 5-node cluster, kill DataNodes one by one, and see exactly where erasure coding wins and where it doesn't.
Erasure coding in HDFS
Erasure coding (EC) is a data protection method that encodes k data units into n = k + m units by adding m parity units, such that the original data can be reconstructed from any k of the n units. HDFS (since Hadoop 3.0) applies EC per directory through an erasure coding policy (e.g. RS-6-3-1024k: Reed-Solomon codec, 6 data and 3 parity units, 1 MB cells). Files under the directory are written in striped layout: data is split into cells distributed round-robin over the k data internal blocks of a block group, and parity cells are computed for each stripe and stored in m parity internal blocks, each internal block on a different DataNode (and rack where possible). Storage overhead is (k + m) / k and up to m simultaneous failures are tolerated; missing units are reconstructed by decoding on read or in the background.
Split the data into 3 parts, compute 2 “recovery” parts, put all 5 on different machines. Any 2 machines can die and the data can be recomputed from the other 3. It costs 1.67× space instead of 3×.
Words you'll meet in this lesson
Built-in policies (Hadoop 3.5.0, from hdfs ec -listPolicies)
| RS-6-3-1024k | ENABLED by default · needs 9 DataNodes · overhead 1.5× · tolerates 3 failures |
|---|---|
| RS-3-2-1024k | Disabled by default · needs 5 DataNodes (3 racks for rack tolerance) · 1.67× · tolerates 2 |
| RS-10-4-1024k | Disabled · 14 DataNodes · 1.4× · tolerates 4 |
| XOR-2-1-1024k | Disabled · 3 DataNodes · 1.5× · tolerates 1 |
| RS-LEGACY-6-3-1024k | Disabled · older Reed-Solomon implementation, kept for compatibility |
Erasure coding at three levels
Beginner A puzzle with spare clues
- Three friends each keep one page of a secret recipe (data). Two more friends keep clue cards made by mixing the pages (parity).
- If any two friends lose their paper, the remaining three can work out the full recipe.
- Photocopying the recipe three times also works, but uses far more paper.
Intermediate How HDFS stores and reads an EC file
- Set a policy on a directory:
hdfs ec -setPolicy -path /ec -policy RS-3-2-1024k. New files below it are erasure-coded; existing files are not converted. - The client cuts the file into 1 MB cells, writes them round-robin to 3 data internal blocks, computes 2 parity cells per stripe, and writes all 5 in parallel to 5 DataNodes.
hdfs dfs -dushows the real cost: 140,760,270 bytes of data use 235,132,110 bytes of disk (1.67×), vs 422,280,810 (3×) with replication.- Reading normally fetches the 3 data blocks; if some are missing, the client fetches parity and decodes on the fly.
Advanced Trade-offs experienced engineers weigh
- Small files: a 15-byte file under RS-3-2 used 45 bytes, the same 3× as replication: one data cell plus two full parity cells. EC only saves space for files spanning several cells.
- Data locality: a striped file's bytes are spread over many nodes, so map tasks can't read a whole split locally. EC suits cold, rarely processed data better than hot data.
- Recovery cost: rebuilding a lost replica copies one block; rebuilding an EC unit reads k units over the network and uses CPU (ISA-L native libraries speed this up).
- Cluster shape: RS-6-3 needs 9 DataNodes, and rack fault tolerance needs enough racks;
hdfs ec -verifyClusterSetupchecks this. Our 2-rack lab got a warning for RS-3-2. - Operations: EC files can't be appended to in older releases and some features (e.g. hflush/hsync semantics) differ; check your version's documentation before using EC for active data.
Exam tip: For “Explain erasure coding in Hadoop 3”, give: the idea (k data + m parity, survive any m losses), the RS-6-3 example (9 units, 1.5× overhead vs 3×), striped layout with cells and block groups, the command hdfs ec -setPolicy, and two disadvantages (CPU/network cost of reconstruction, poor fit for small files and hot data).
Replication vs erasure coding
| Replication 3 | RS-3-2 | RS-6-3 | RS-10-4 | |
|---|---|---|---|---|
| Raw space | 300% | 167% | 150% | 140% |
| Failures tolerated | 2 | 2 | 3 | 4 |
| Min DataNodes | 3 | 5 | 9 | 14 |
| Normal read | One local replica | Striped from 3 nodes | Striped from 6 | Striped from 10 |
| Repair | Copy one block | Read k units, decode, write the missing one | ||
| Best for | Hot data, small files | Large, cold or archival files | ||
Commands you'll use
See policies
hdfs ec -listPolicies, -enablePolicy -policy …
Check the cluster
hdfs ec -verifyClusterSetup -policy RS-6-3-1024k: enough nodes and racks?
Apply to a folder
hdfs ec -setPolicy -path /archive -policy RS-6-3-1024k; -getPolicy; -unsetPolicy.
Convert old data
Copy existing files into the EC folder (e.g. distcp), verify, then delete the replicated originals.
Setting a policy changes only new files. Existing files keep their layout until rewritten. That is why our lab copied orders_x30.csv into /ec rather than setting the policy on the original folder.
The same file, two ways, five DataNodes
Our 5-DataNode, 2-rack cluster from Module 5. All output is from ec_lab.sh (in “Run it on your PC”).
Enable a policy and check the cluster
RS-6-3 is impossible with 5 DataNodes. RS-3-2 works, but with only 2 racks one rack holds 3 of the 5 units, so losing that whole rack would lose data. We continue for the lab; production needs 3+ racks.hdfs ec -enablePolicy -policy RS-3-2-1024k hdfs ec -verifyClusterSetup -policy RS-6-3-1024k hdfs ec -verifyClusterSetup -policy RS-3-2-1024kErasure coding policy RS-3-2-1024k is enabled Warning: The cluster setup does not support EC policy RS-3-2-1024k. Reason: 3 racks are required for the erasure coding policies: RS-3-2-1024k. The number of racks is only 2. 9 DataNodes are required for the erasure coding policies: RS-6-3-1024k. The number of DataNodes is only 5. 3 racks are required for the erasure coding policies: RS-3-2-1024k. The number of racks is only 2.Write the same file with and without EC
Big file: 422 MB vs 235 MB on disk (3.00× vs 1.67×), a 44% saving. Tiny file: 45 bytes both ways. One cell of data still needs two full parity cells, so EC saves nothing on small files.hdfs ec -setPolicy -path /ec -policy RS-3-2-1024k hdfs dfs -cp /user/hadoop/data/orders_x30.csv /rep/ hdfs dfs -cp /user/hadoop/data/orders_x30.csv /ec/ echo 'one tiny order' | hdfs dfs -put - /ec/tiny.txt # and the same into /rep hdfs dfs -du /rep /ec140760270 422280810 /rep/orders_x30.csv 15 45 /rep/tiny.txt 140760270 235132110 /ec/orders_x30.csv 15 45 /ec/tiny.txtOne block group, five internal blocks
The whole 134 MB file fits in one block group (3 × 128 MB of data capacity). Its five internal blocks have consecutive negative IDs: indexes 0–2 hold data, 3–4 hold parity, each on a different DataNode. (Our listing is the fsck locations, shortened.)hdfs fsck /ec/orders_x30.csv -files -blocks -locations/ec/orders_x30.csv 140760270 bytes, erasure-coded: policy=RS-3-2-1024k, 1 block(s): OK blk_-9223372036854775728 172.19.0.7 blk_-9223372036854775727 172.19.0.4 blk_-9223372036854775726 172.19.0.5 blk_-9223372036854775725 172.19.0.6 blk_-9223372036854775724 172.19.0.8Lose two DataNodes
Identical bytes, decoded from the surviving units. The 63 s were mostly waiting: right after the stop, the NameNode still listed the two DataNodes as live, so the client spent a 60 s connect timeout before using parity. In another run done a little later, the same read took 2 s.docker stop lab5-datanode-4 lab5-datanode-5 hdfs dfs -get -f /ec/orders_x30.csv /tmp/ec2.csv ; md5sum /tmp/ec2.csvmd5 before: f07bba61afc730a4e29f03702b25e2fb read took 63 s f07bba61afc730a4e29f03702b25e2fb /tmp/ec2.csvLose a third DataNode
Three of five units missing, more than the 2 parity units: the EC file is unreadable. The replicated copy still read fine, because with only 5 DataNodes some of its replicas survived on the 2 remaining nodes. Equal fault tolerance on paper (2 failures each); in this case replication happened to survive a third.docker stop lab5-datanode-3 hdfs dfs -get -f /ec/orders_x30.csv /tmp/ec3.csv hdfs dfs -get -f /rep/orders_x30.csv /tmp/rep3.csv ; md5sum /tmp/rep3.csvexit=255 3 missing blocks, the stripe is: AlignedStripe(Offset=0, length=1048576, fetchedChunksNum=0, missingChunksNum=3) get: Fatal internal error read took 2 s f07bba61afc730a4e29f03702b25e2fb /tmp/rep3.csv
Never set an EC policy on folders with many small files, or on hot data that jobs read all day. Check verifyClusterSetup first, and keep enough racks: an EC policy without rack diversity can lose data to a single rack failure.
Writing stripes, losing nodes, decoding
Six steps for one stripe of RS-3-2 (three 1 MB data cells, two parity cells).
- The admin sets RS-3-2 on the
/ecfolder. - The file is copied in; new files in the folder are erasure-coded.
-dushows 1.67× instead of 3× raw space.- Two DataNodes stop. The file still reads correctly.
- A third stops. The EC file can no longer be read.
- Small files gain nothing: 15 bytes still cost 45.
- RS-3-2 needs five internal blocks on five different DataNodes.
- The client cuts data into 1 MB cells, writes them round-robin to three data blocks, and computes two parity cells for each stripe.
- Every 3 MB of data costs 5 MB of disk: 1.67×. Three replicas would cost 9 MB.
- Two DataNodes are lost. Any 3 of the 5 units are enough to rebuild each stripe.
- With a third loss only two units remain, fewer than the three needed: the data is lost until nodes return.
- A tiny file fills only part of one cell, but both parity cells are as big as that cell: no saving.
Erasure coding at work
1. Cold and archival data
Clusters apply RS-6-3 to folders of old partitions, logs and backups that are rarely read, cutting raw storage for that data roughly in half compared with 3× replication.
2. Hot and cold in one cluster
Today's and this month's partitions stay replicated for fast local reads; a scheduled job moves older partitions into an EC directory and updates the table's location.
3. Planning racks for EC
Architects size racks so each block group spreads widely: with RS-6-3 across 9 or more racks, a whole rack failure costs at most one unit per group.
4. The same idea everywhere
Cloud object stores and many storage systems also use erasure coding internally to achieve high durability without full copies; HDFS brought the technique to Hadoop clusters.
Simplified; organisations differ in tools and policies.
Parity you can compute: XOR erasure coding
HDFS's simplest policy, XOR-2-1, computes one parity cell as the XOR of two data cells. Lose any one of the three and XOR the other two to get it back. Reed-Solomon does the same idea with more parity. Run it, then lose a different cell.
d1 = b"order_id,item,amount\n1,chai,20\n"
d2 = b"2,poha,30\n3,dosa,40\n4,idli,30\n"
size = max(len(d1), len(d2))
d1, d2 = d1.ljust(size, b"\0"), d2.ljust(size, b"\0") # cells are padded to equal size
parity = bytes(a ^ b for a, b in zip(d1, d2)) # XOR-2-1: one parity cell
print("stored cells:", len(d1), "+", len(d2), "+", len(parity), "bytes")
lost = "d2" # a DataNode dies
rebuilt = bytes(a ^ p for a, p in zip(d1, parity)) # d1 XOR parity = d2
print("rebuilt", lost, "correct:", rebuilt == d2)
print(rebuilt.rstrip(b"\0").decode())
print(f"overhead: {(len(d1) + len(d2) + len(parity)) / (2 * size):.2f}x")parity = bytes(a ^ b for a, b in zip(d1, d2)) # XOR-2-1: one parity cellXOR has a handy property: if p = a ⊕ b, then a = p ⊕ b and b = p ⊕ a. One parity cell protects against one loss.rebuilt = bytes(a ^ p for a, p in zip(d1, parity)) # d1 XOR parity = d2Reconstruction is computation, not copying: the reader needs the surviving cells and CPU time.print(f"overhead: {(len(d1) + len(d2) + len(parity)) / (2 * size):.2f}x")3 cells stored for 2 cells of data: 1.5×, versus 3× with replication. RS-6-3 also reaches 1.5× but survives 3 losses.
Run it on your PC: replication vs RS-3-2 on five DataNodes
Use the 5-DataNode cluster from Module 5 (lab5). The script enables RS-3-2, checks the cluster, copies the same file into a replicated folder and an erasure-coded folder, compares disk usage, then stops DataNodes one by one and reads the files. It restarts the DataNodes at the end. Save as ec_lab.sh in the lab folder and run bash ec_lab.sh.
Erasure coding lab
#!/usr/bin/env bash
# ec_lab.sh: store the same file with 3x replication and with RS-3-2 erasure coding, then lose DataNodes.
P=lab5
NN="docker exec $P-namenode-1"
run() { echo "\$ $1"; $NN bash -c "$1" 2>&1 | grep -v -E "^\s*$|WARN|INFO"; echo; }
run "hdfs ec -enablePolicy -policy RS-3-2-1024k"
run "hdfs ec -verifyClusterSetup -policy RS-6-3-1024k"
run "hdfs ec -verifyClusterSetup -policy RS-3-2-1024k"
run "hdfs dfs -rm -r -f -skipTrash /ec /rep > /dev/null; hdfs dfs -mkdir -p /ec /rep && hdfs ec -setPolicy -path /ec -policy RS-3-2-1024k"
run "hdfs ec -getPolicy -path /ec ; hdfs ec -getPolicy -path /rep"
run "hdfs dfs -cp /user/hadoop/data/orders_x30.csv /rep/ && hdfs dfs -cp /user/hadoop/data/orders_x30.csv /ec/"
run "echo 'one tiny order' | hdfs dfs -put - /ec/tiny.txt; echo 'one tiny order' | hdfs dfs -put - /rep/tiny.txt"
run "hdfs dfs -du /rep /ec"
run "hdfs fsck /ec/orders_x30.csv -files -blocks -locations | grep -E 'bytes|blk_' | cut -c1-210"
run "hdfs dfs -get -f /ec/orders_x30.csv /tmp/ec.csv > /dev/null 2>&1; hdfs dfs -get -f /rep/orders_x30.csv /tmp/rep.csv > /dev/null 2>&1; md5sum /tmp/ec.csv /tmp/rep.csv"
echo ">>> docker stop $P-datanode-4 $P-datanode-5 (2 of 5 DataNodes gone)"; docker stop $P-datanode-4 $P-datanode-5 > /dev/null; echo
run "s=\$(date +%s); hdfs dfs -get -f /ec/orders_x30.csv /tmp/ec2.csv > /dev/null 2>&1; echo read took \$(( \$(date +%s) - s )) s; md5sum /tmp/ec2.csv"
echo ">>> docker stop $P-datanode-3 (3 of 5 gone: more than the 2 parity blocks)"; docker stop $P-datanode-3 > /dev/null; echo
run "hdfs dfs -get -f /ec/orders_x30.csv /tmp/ec3.csv > /tmp/err 2>&1; echo exit=\$?; grep -m1 -oE '[0-9]+ missing blocks, the stripe is: AlignedStripe[^;]*' /tmp/err; grep -m1 -oE 'get: .*' /tmp/err"
run "s=\$(date +%s); hdfs dfs -get -f /rep/orders_x30.csv /tmp/rep3.csv > /dev/null 2>&1; echo read took \$(( \$(date +%s) - s )) s; md5sum /tmp/rep3.csv"
docker start $P-datanode-3 $P-datanode-4 $P-datanode-5 > /dev/null
# Run: bash ec_lab.shReal output. The read with 2 DataNodes down took 63 s here because the NameNode still listed them as live and the client waited for a connect timeout; a repeat run took 2 s. Block IDs and IPs will differ on your run.
Striping cells round-robin
Follow how a writer spreads 1 MB cells over three data internal blocks, stripe by stripe, and adds two parity cells per stripe.
Worked examples: real problems, solved and traced in memory
Three erasure-coding decisions with numbers: storage saved, how many DataNodes a policy needs, and whether a folder should be erasure-coded at all.
How much disk does EC save on the archive?
Problem statement
The university's archive holds 400 TB of data, all with replication 3. If it moves to RS-6-3, how much raw disk is freed?
400 TB · 3× vs 1.5×
replication 1200 TB, RS-6-3 600 TB, freed 600 TB
Approach
- Raw with replication = 400 × 3.
- Raw with RS-6-3 = 400 × 9/6.
- Difference.
Solution: run it live
What happens in memory, step by step
Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.
Can our cluster use the policy?
Problem statement
The cluster has 8 DataNodes in 3 racks. Which of RS-3-2, RS-6-3 and RS-10-4 can it run, using the rule “at least k + m DataNodes”?
8 DataNodes
RS-3-2 needs 5 DataNodes -> OK RS-6-3 needs 9 DataNodes -> not enough RS-10-4 needs 14 DataNodes -> not enough
Approach
- Units per group = k + m.
- Compare with the DataNode count.
- Report usable policies.
Solution: run it live
What happens in memory, step by step
Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.
Should this folder be erasure-coded?
Problem statement
Three folders: /logs/2021 (400 TB, average file 900 MB, read once a year), /tmp (2 TB, average file 40 KB, read constantly) and /warehouse/today (50 TB, 256 MB files, read by every morning job). Score each for EC.
3 folders
/logs/2021 EC /tmp keep replication /warehouse/today keep replication
Approach
- EC needs files much larger than one stripe (k × 1 MB).
- EC suits cold data.
- Recommend.
Solution: run it live
What happens in memory, step by step
Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.
Check yourself
Quick revision
Lesson 7.3 in one screen
- EC: k data + m parity units; survive any m losses; overhead (k+m)/k.
- Policies: RS-6-3 (enabled by default, 9 DNs, 1.5×), RS-3-2 (5 DNs, 1.67×), RS-10-4, XOR-2-1.
- Striped layout: 1 MB cells round-robin; a block group of k+m internal blocks on different DataNodes.
- Lab: 422,280,810 B (rep 3) vs 235,132,110 B (RS-3-2) for the same file; same MD5 after 2 DNs died; “3 missing blocks” after the third.
- Small files: no saving (15 B → 45 B). Hot data: loses locality, costs decode CPU.
- Commands:
hdfs ec -listPolicies / -enablePolicy / -verifyClusterSetup / -setPolicy / -getPolicy. - Policy affects only new files; move existing data with a copy.
Interview and viva questions
What is erasure coding in Hadoop?
A storage method introduced in Hadoop 3 in which file data is split into data units and parity units are computed with an algorithm such as Reed-Solomon. The units are stored on different DataNodes, and the original data can be reconstructed from any k of the k + m units. It gives similar or better fault tolerance than 3× replication at around 1.4–1.67× storage.
What are the disadvantages of erasure coding?
Reconstruction needs network reads from several nodes and CPU for decoding; reads lose data locality because data is striped across nodes; small files gain no space; it needs at least k + m DataNodes and enough racks; and some write features behave differently from replicated files. It is best for large, cold data.
How do you enable erasure coding on a directory?
Enable the policy if needed (hdfs ec -enablePolicy -policy RS-6-3-1024k), check the cluster with hdfs ec -verifyClusterSetup, then run hdfs ec -setPolicy -path /dir -policy RS-6-3-1024k. Files created afterwards in that directory are erasure-coded; existing files must be copied to be converted.
What is a block group?
In striped erasure coding, the set of k data and m parity internal blocks that together store a range of a file. Each internal block is stored on a different DataNode and has an ID derived from the group ID.
Replication or erasure coding: how do you choose?
Use replication for hot data, small files and data read with locality by many jobs. Use erasure coding for large files that are rarely read, such as archives, old partitions and backups, where storage savings matter more than read performance.
Practice problems
Erasure-coding exercises using the real sizes, IDs and messages from the lab.
Overhead formula
Print the overhead (k+m)/k for RS-3-2, RS-6-3 and RS-10-4.
3 policies
RS-3-2: 1.67x RS-6-3: 1.50x RS-10-4: 1.40x
Show solution and explanation
for k, m in [(3, 2), (6, 3), (10, 4)]:
print(f"RS-{k}-{m}: {(k + m) / k:.2f}x")How it works: 1.67×, 1.50×, 1.40×.
Saving in the lab
How many bytes did RS-3-2 save compared with replication for the lab file?
422,280,810 vs 235,132,110
187,148,700 bytes saved
Show solution and explanation
print(f"{422_280_810 - 235_132_110:,} bytes saved")How it works: 187,148,700 bytes, about 44% less raw space.
Internal block indexes
Internal block IDs are the group ID plus the index. Given the five IDs from fsck, print each index and whether it holds data or parity (k = 3).
-9223372036854775728 … -9223372036854775724
-9223372036854775728 index 0 data -9223372036854775727 index 1 data -9223372036854775726 index 2 data -9223372036854775725 index 3 parity -9223372036854775724 index 4 parity
Show solution and explanation
ids = [-9223372036854775728 + i for i in range(5)]
group = ids[0]
for b in ids:
i = b - group
print(b, "index", i, "data" if i < 3 else "parity")How it works: Indexes 0–2 are data, 3–4 parity: consecutive IDs in one block group.
XOR recovery
Data cells a = b"HDFS", b = b"EC!!"; parity p = a XOR b. Rebuild a from b and p.
two 4-byte cells
b'HDFS'
Show solution and explanation
a, b = b"HDFS", b"EC!!"
p = bytes(x ^ y for x, y in zip(a, b))
print(bytes(x ^ y for x, y in zip(b, p)))How it works: b"HDFS": XOR parity is the simplest erasure code.
Readable or not?
For RS-6-3, given the number of lost units in a block group, print whether it can be read.
0 to 4 lost
0 lost -> readable 1 lost -> readable 2 lost -> readable 3 lost -> readable 4 lost -> LOST
Show solution and explanation
k, m = 6, 3
for lost in range(5):
print(lost, "lost ->", "readable" if lost <= m else "LOST")How it works: Up to 3 losses are fine; the fourth makes the group unreadable.
Stripes in a file
How many full stripes does a 140,760,270-byte file have under RS-3-2 with 1 MB cells?
140,760,270 bytes
44 full stripes
Show solution and explanation
cell, k = 1024 * 1024, 3
print(140_760_270 // (k * cell), "full stripes")How it works: 44 full stripes of 3 MB, plus a partial one; each stripe has its own 2 parity cells.
Archive disk budget
Disk costs ₹4,000 per TB (with servers and power, illustrative). An archive of 300 TB moves from replication 3 to RS-6-3. What is the saving?
300 TB
450 TB freed -> Rs 1,800,000
Show solution and explanation
freed = 300 * 3 - 300 * 1.5
print(f"{freed:.0f} TB freed -> Rs {freed * 4000:,.0f}")How it works: 450 TB freed, about ₹18 lakh of disk (illustrative), or room for years of new data.
Placement check
Given rack assignments of the 5 internal blocks, check whether losing any single rack loses more than m = 2 units.
5 units over 2 racks
worst rack holds 3 units -> a rack failure loses data
Show solution and explanation
from collections import Counter
racks = ["r1", "r1", "r2", "r2", "r2"] # rack of each internal block
worst = max(Counter(racks).values())
print("worst rack holds", worst, "units ->", "safe" if worst <= 2 else "a rack failure loses data")How it works: With 2 racks, one holds 3 units: exactly why verifyClusterSetup asked for 3 racks for RS-3-2.
Real-world scenario questions (asked in company interviews)
Erasure coding questions are common in Hadoop 3 interviews, especially for platform and administrator roles.
“What's new in Hadoop 3 for storage?”
A common Hadoop 2 vs 3 question.
The interviewer is checking: erasure coding and its trade-offs.
Show a strong answer
- Erasure coding: e.g. RS-6-3 stores 1.5× instead of 3× and tolerates 3 failures.
- Also: more than two NameNodes for HA, intra-DataNode disk balancer, and other improvements.
- Mention when not to use EC: small files, hot data needing locality.
Storage is 85% full; buying disks takes 4 months
A 2 PB cluster, mostly replicated. 60% of data is older than a year and rarely read.
The interviewer is checking: practical EC adoption.
Show a strong answer
- Identify cold, large-file folders; check
verifyClusterSetupfor RS-6-3 (nodes and racks). - Create EC folders, copy cold partitions with distcp, verify checksums/row counts, repoint tables, delete originals.
- Moving 1.2 PB of data from 3× to 1.5× frees roughly 1.8 PB of raw space; do it in batches off-peak.
Jobs on EC data are slower and network traffic jumped
After converting a frequently queried table to EC, morning jobs take 40% longer.
The interviewer is checking: locality and decode costs.
Show a strong answer
- Striped files have no node-local splits: every read pulls cells from several DataNodes over the network.
- Any failed node forces decoding, which costs CPU (check that native ISA-L acceleration is enabled).
- Keep hot tables replicated; apply EC only after data cools down.
“Is EC as safe as replication?”
A manager's question before approving EC.
The interviewer is checking: fault tolerance with conditions.
Show a strong answer
- RS-6-3 tolerates 3 simultaneous losses per block group vs 2 for replication 3, at half the space.
- Conditions: enough DataNodes and racks so units are spread out, and capacity for reconstruction traffic.
- Real demo: data survived 2 lost DataNodes with an identical checksum under RS-3-2, and failed only at the third.