How a file is read

1 The story

Nearest copy first, and never trust a damaged one

Kabir reads the 5-block canteen file twice: once from the namenode container, once from datanode-2. From the namenode container the blocks come from datanode-2, then datanode-3 three times, then datanode-1. From datanode-2, every single block comes from datanode-2 itself. Then Riya does something wicked: she overwrites one byte of a block file on datanode-2's disk with an X, and reads again from datanode-2. A warning appears: Found Checksum error … at 499712. Yet the file that comes out is byte-for-byte identical to the original.

Arjun Sir

A read has two parts. First the client asks the NameNode “where are the blocks of this file?”. The NameNode answers with every block's replica locations, sorted by distance from the client. Then the client reads each block directly from the first DataNode in that list.

Kabir

And from the namenode container there was no “nearest”?

Arjun Sir

Right: it isn't a DataNode, and all three replicas were equally far, so the order was shuffled to spread the load. From datanode-2, its own disk is always closest. And the corrupted byte? Every 512 bytes of a block has a checksum. The client verified it, saw a mismatch, quietly fetched that part from another replica, and told the NameNode the copy on datanode-2 was bad. A few seconds later the NameNode had made a fresh copy elsewhere and deleted the bad one.

2 Definition

The HDFS read path

Formal definition (write this in exams)

To read a file, the client calls open; the DFSInputStream asks the NameNode (getBlockLocations) for the locations of the file's blocks, in batches. For each block the NameNode returns the DataNodes holding a replica, sorted by network distance from the client (local node, then same rack, then other racks; equal distances are shuffled). The client connects to the closest DataNode and streams the block directly; at a block boundary it moves to the best DataNode for the next block. Data is verified with checksums (one per 512 bytes) as it is read. On a checksum error or a DataNode failure, the client reads from the next replica, remembers the bad DataNode for that stream, and reports corrupt replicas to the NameNode (reportBadBlocks), which schedules re-replication from a good replica and deletes the corrupt one. The NameNode serves only metadata; data never passes through it.

In simple words

Ask the NameNode where the pieces are, then fetch each piece from the nearest machine that has it, checking every 512 bytes. If a piece is broken or a machine doesn't answer, use another copy and tell the NameNode.

Words you'll meet in this lesson

getBlockLocationsNameNode call returning blocks and their replica locations for a byte range of a file.

LocatedBlocksThe reply: file length and a list of blocks, each with sorted locations.

Network distance0 same node, 2 same rack, 4 different rack (in the default topology).

DFSInputStreamThe client-side stream that picks DataNodes, reads, verifies and retries.

ChecksumExceptionRaised when data doesn't match its stored checksum.

reportBadBlocksClient or DataNode tells the NameNode a replica is corrupt.

Short-circuit readOptional: a client on the same machine reads the block file directly, bypassing the DataNode process.

Positional read (pread)Read bytes at an offset without moving the stream position; used by Parquet/ORC readers.

Read facts (Hadoop 3.5.0, seen in the lab)

Checksum unit	512 bytes (`dfs.bytes-per-checksum`); error reported at offset 499,712 = 976 × 512, the chunk containing the changed byte 500,000
Client on a DataNode	All 5 blocks read from its own DataNode
Client off the DataNodes	Replicas at equal distance are shuffled: we saw dn-2, dn-3, dn-3, dn-3, dn-1
Short-circuit reads	`dfs.client.read.shortcircuit` = false by default (needs a domain socket path)
Corrupt replica handling	reportBadBlocks at 09:37:30, new replica on another DataNode 6 s later, bad file removed

The read path at three levels

Pick your level. New to this? Start at Beginner and climb.

Beginner A library with photocopies

You ask the index desk: “Where is chapter 1?” The librarian lists shelves holding copies, the nearest first.
You walk to the nearest shelf and read it there. The librarian never fetches pages.
If a page is smudged, you take another copy and tell the librarian, who replaces the smudged one.

Intermediate Step by step

open(): client asks the NameNode for block locations of the first part of the file.
read(): connect to the first (closest) DataNode for block 0 and stream it, verifying checksums.
At the end of block 0, pick the best DataNode for block 1 (maybe a different one), and so on.
For long files, ask the NameNode for more locations as needed.
close(): release connections.

Advanced Robustness and performance

Failures: a DataNode that doesn't respond is added to the stream's dead-node list and the next replica is used.
Corruption: detected end-to-end by the client (checksums were created by the writer), so disk, network and memory errors are all caught.
DataNodes also run a background block scanner that finds corrupt replicas nobody has read for weeks.
Data locality in MapReduce and Spark is the same idea: put the task where the closest replica is distance 0.
Hedged reads (optional) start a second read from another replica if the first is slow.

Exam tip: “Explain the anatomy of a file read in HDFS”: client open() → NameNode returns block locations sorted by proximity → client reads each block directly from the nearest DataNode, verifying checksums → moves to the next block → on error, tries another replica and reports the bad block → close(). Stress that the NameNode never sends data.

3 Types and ways to use it

Kinds of reads

Read	Where the client is	How data travels	Typical user
Local	On a DataNode holding the block	From the local DataNode (over a local socket)	MapReduce/Spark task with data locality
Short-circuit local	Same, with short-circuit enabled	Reads the block file directly from disk	Impala, HBase, tuned Spark clusters
Rack-local	Same rack as a replica	Through the rack switch	Tasks that missed node locality
Remote	Other rack or outside the cluster	Through the core network	Edge-node users, BI tools, `hdfs dfs -get`

Read-side protections

Checksums

CRC per 512 bytes, checked by the reading client. Catches disk and network corruption.

Integrity

Replica failover

Dead or slow DataNode → next replica in the sorted list.

Availability

Block scanner

Each DataNode re-verifies its blocks in the background (default every 3 weeks).

Background

reportBadBlocks

Corrupt replicas are replaced from good ones automatically.

Self-healing

hdfs dfs -checksum path prints a file-level checksum built from block checksums. Compare it between two clusters (with the same block size) to check that a copy is identical without reading the data yourself.

4 Step by step

Watch reads on the real cluster

IP addresses at the time: datanode-1 = 172.19.0.4, datanode-2 = .3, datanode-3 = .6, datanode-4 = .7. Debug logging is switched on for the client only (configuration in read_trace.py, Part 4b).

What the NameNode tells the client
```
HADOOP_CLIENT_OPTS="-Dlog4j.configuration=file:/tmp/dbg2.properties" \
  hdfs dfs -cat /user/hadoop/data/canteen_1mb.csv 2>/tmp/rd.log >/dev/null
grep -o "LocatedBlocks{.*" /tmp/rd.log | cut -c1-420
```
LocatedBlocks{; fileLength=4692009; underConstruction=false; blocks=[LocatedBlock{BP-1507681208-172.19.0.3-1792565333483:blk_1073741826_1002; getBlockSize()=1048576; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[172.19.0.3:9866,…], DatanodeInfoWithStorage[172.19.0.4:9866,…], DatanodeInfoWithStorage[172.19.0.6:9866,…]]; cachedLocs=[]}, LocatedBlock{…:blk_1073741827_1003; getBlockSize()=1048576; corrupt=false; offset=1048576…
One answer describes the whole file: its length, and for each block the ID, size, offset in the file and the replica locations. That's all the client needs; it never asks about data again.
Read from outside the DataNodes
```
grep "Connecting to datanode" /tmp/rd.log
```
09:36:54,867 DEBUG DFSClient - Connecting to datanode 172.19.0.6:9866 09:36:55,119 DEBUG DFSClient - Connecting to datanode 172.19.0.6:9866 09:36:55,135 DEBUG DFSClient - Connecting to datanode 172.19.0.4:9866 09:36:55,149 DEBUG DFSClient - Connecting to datanode 172.19.0.6:9866 09:36:55,161 DEBUG DFSClient - Connecting to datanode 172.19.0.3:9866
The namenode container holds no replicas and isn't in either rack, so every replica is equally far. The order is shuffled per block, spreading readers across DataNodes.
Read from a DataNode that holds the blocks
```
HADOOP_CLIENT_OPTS="-Dlog4j.configuration=file:/tmp/dbg2.properties" \
  hdfs dfs -cat /user/hadoop/data/canteen_1mb.csv 2>&1 >/dev/null | grep "Connecting to datanode"
```
09:36:57,823 DEBUG DFSClient - Connecting to datanode 172.19.0.3:9866 09:36:57,924 DEBUG DFSClient - Connecting to datanode 172.19.0.3:9866 09:36:57,942 DEBUG DFSClient - Connecting to datanode 172.19.0.3:9866 09:36:57,961 DEBUG DFSClient - Connecting to datanode 172.19.0.3:9866 09:36:57,970 DEBUG DFSClient - Connecting to datanode 172.19.0.3:9866
Every block from 172.19.0.3, datanode-2 itself: distance 0. This is exactly what a map task with data locality gets.
Damage one replica
```
f=$(find /tmp/hadoop-hadoop/dfs/data -name blk_1073741827)
printf 'X' | dd of=$f bs=1 seek=500000 conv=notrunc
```
One byte in the middle of a 1 MB block replica on datanode-2 is overwritten. The file size doesn't change and HDFS isn't told. (Never do this on a real cluster: it's a demonstration of corruption.)
Read again from datanode-2
```
hdfs dfs -cat /user/hadoop/data/canteen_1mb.csv > /tmp/out.csv
md5sum /tmp/out.csv
md5sum canteen_orders.csv          # the original, on your PC
```
09:37:30 WARN DFSClient - Found Checksum error for BP-1507681208-…:blk_1073741827_1003 from DatanodeInfoWithStorage[172.19.0.3:9866,…] at 499712 6cba714b557a8b2d1e01eb091ff42ca2 /tmp/out.csv 6cba714b557a8b2d1e01eb091ff42ca2 canteen_orders.csv
The client noticed that the 512-byte chunk starting at 499,712 didn't match its checksum, read that block from another replica, and produced a file identical to the original (same MD5).
The NameNode heals the block
```
docker compose logs namenode | grep blk_1073741827 | tail -2
docker compose exec namenode hdfs fsck /user/hadoop/data/canteen_1mb.csv -files -blocks -locations | grep blk_1073741827
docker exec lab5-datanode-2 bash -c 'find /tmp/hadoop-hadoop/dfs/data -name "blk_1073741827*" | wc -l'
```
09:37:30 INFO StateChange:6023 - *DIR* reportBadBlocks for block: …:blk_1073741827_1003 on datanode: 172.19.0.3:9866 09:37:36 INFO BlockStateChange:3806 - BLOCK* addStoredBlock: 172.19.0.7:9866 is added to blk_1073741827_1003 (size=1048576) 1. …:blk_1073741827_1003 len=1048576 Live_repl=3 [172.19.0.4, 172.19.0.7, 172.19.0.6] 0
The client reported the bad replica; six seconds later a good copy existed on datanode-4 (.7), and the corrupt files on datanode-2 were deleted (0 left). Three healthy replicas again, without anyone running a command.

If every replica of a block is corrupt or unavailable, the read fails with a BlockMissingException and fsck marks the file CORRUPT. Replication protects against independent failures, not against writing bad data in the first place.

5 See it happen

A read that survives a corrupt replica

Steps 3 to 6 from the lab, frame by frame.

Reading canteen_1mb.csv from datanode-2 6 steps

What you see

Behind the screen

6 Real-world use

The read path at work

1. Data-local tasks

Schedulers place map and Spark tasks on nodes holding a replica, so most reads are distance 0, like step 3. Job counters such as “Data-local map tasks” show how well this worked.

Block on dn-2→Task on dn-2→Local read

2. Short-circuit reads for fast engines

Interactive SQL engines and HBase enable short-circuit local reads so they read block files straight from disk, skipping the DataNode's network stack and cutting latency.

Local block→Read file directly→Lower latency

3. Silent disk errors caught

On thousands of disks, bit rot and bad sectors happen. Checksum errors show up as WARN lines and DataNode metrics; the data is healed automatically, and repeated errors on one disk tell admins to replace it.

Bad sector→Checksum error→Replace replica, then disk

4. Verifying copies between clusters

After copying data to a DR cluster, teams compare hdfs dfs -checksum (or let DistCp do it) instead of reading terabytes back to compare.

Copy→Compare checksums→Proven identical

Simplified; organisations differ in tools and policies.

7 Run the code

Sort replicas by distance, like the NameNode

This program computes the network distance from a client to each replica (0 same node, 2 same rack, 4 other rack) and sorts the replica list, shuffling ties as HDFS does. Try it for a client on a DataNode and for one outside the racks.

import random
random.seed(3)
rack = {"dn-1": "/rack1", "dn-2": "/rack1", "dn-3": "/rack2", "dn-4": "/rack2"}

def distance(client, dn):
    if client == dn:
        return 0
    if rack.get(client) == rack[dn]:
        return 2
    return 4

def sorted_replicas(client, replicas):
    shuffled = random.sample(replicas, len(replicas))     # ties in random order
    return sorted(shuffled, key=lambda dn: distance(client, dn))

replicas = ["dn-1", "dn-2", "dn-3"]
for client in ["dn-2", "dn-4", "edge-node"]:
    order = sorted_replicas(client, replicas)
    print(f"{client:9} reads from {order[0]}   (order {order})")

dn-2 reads from dn-2 (order ['dn-2', 'dn-1', 'dn-3']) dn-4 reads from dn-3 (order ['dn-3', 'dn-2', 'dn-1']) edge-node reads from dn-3 (order ['dn-3', 'dn-1', 'dn-2'])

if rack.get(client) == rack[dn]:A client that is not in any rack (like an edge node or our namenode container) never matches, so every replica is distance 4.
shuffled = random.sample(replicas, len(replicas)) # ties in random orderShuffling before a stable sort keeps the distance order but randomises equal distances, spreading load. That is why the off-cluster read used several DataNodes.

Run it on your PC: who serves your reads?

This script prints where each block of canteen_1mb.csv has replicas, then reads the file from two clients (the namenode container and datanode-2) with client debug logging and shows which DataNode served each block. Save it as read_trace.py in the hdfs-lab folder.

Same file, two clients, different DataNodes

# read_trace.py: read the same file from two different clients and show
# which DataNode served each block. Run in the hdfs-lab folder.
import re
import subprocess

LOG4J = """log4j.rootLogger=WARN, err
log4j.appender.err=org.apache.log4j.ConsoleAppender
log4j.appender.err.Target=System.err
log4j.appender.err.layout=org.apache.log4j.PatternLayout
log4j.appender.err.layout.ConversionPattern=%m%n
log4j.logger.org.apache.hadoop.hdfs.DFSClient=DEBUG
"""
FILE = "/user/hadoop/data/canteen_1mb.csv"

def sh(cmd):
    return subprocess.run(cmd, shell=True, capture_output=True, text=True)

names = {}
for n in sh("docker ps --format '{{.Names}}'").stdout.split():
    ip = sh("docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' " + n).stdout.strip()
    names[ip] = n.replace("lab5-", "")

fsck = sh(f"docker compose exec -T namenode hdfs fsck {FILE} -files -blocks -locations").stdout
for i, line in enumerate(l for l in fsck.splitlines() if "blk_" in l):
    hosts = re.findall(r"(\d+\.\d+\.\d+\.\d+):9866", line)
    print(f"block {i} replicas on: {', '.join(names[h] for h in hosts)}")

for client in ["lab5-namenode-1", "lab5-datanode-2"]:
    sh(f"docker exec -i {client} bash -c 'cat > /tmp/r.properties' <<'X'\n{LOG4J}X")
    err = sh(f"docker exec {client} bash -c 'HADOOP_CLIENT_OPTS=-Dlog4j.configuration=file:/tmp/r.properties "
             f"hdfs dfs -cat {FILE} > /dev/null'").stderr
    served = re.findall(r"Connecting to datanode (\d+\.\d+\.\d+\.\d+):9866", err)
    print(f"client {client.replace('lab5-', ''):11} read from: {', '.join(names[s] for s in served)}")

# In the hdfs-lab folder:
#   python3 read_trace.py

block 0 replicas on: datanode-1, datanode-2, datanode-3 block 1 replicas on: datanode-1, datanode-2, datanode-3 block 2 replicas on: datanode-1, datanode-2, datanode-3 block 3 replicas on: datanode-1, datanode-2, datanode-3 block 4 replicas on: datanode-1, datanode-2, datanode-3 client namenode-1 read from: datanode-2, datanode-3, datanode-3, datanode-3, datanode-1 client datanode-2 read from: datanode-2, datanode-2, datanode-2, datanode-2, datanode-2

Real output. The off-cluster client's choices are random among equally distant replicas, so your order will differ; the client on datanode-2 always reads locally.

8 Line by line

Reading a file block by block

Follow a client reading a 3-block file from the sorted location lists, skipping a dead DataNode for one block.

Worked examples: real problems, solved and traced in memory

Three read-path questions with numbers: how much locality saves on the network, how rare silent corruption really is at scale, and how hedged reads cut tail latency. Each runs live and is traced in memory.

Example 1BeginnerNetwork · locality

How much traffic does data locality save?

Problem statement

A daily job reads 40 TB. With good scheduling, 85% of reads are node-local, 12% rack-local and 3% off-rack. Without locality, assume 0% local, 20% rack-local and 80% off-rack. How many TB cross the core network in each case?

Input

40 TB · 85/12/3% vs 0/20/80%

Expected output

with locality : 1.2 TB through the core network
without       : 32.0 TB through the core network

Approach

Only off-rack reads cross the core switch.
Core traffic = total × off-rack share.
Compare.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 2IntermediateIntegrity · scale

How many corrupt chunks per day?

Problem statement

Suppose disks return a silently wrong byte once per 10¹⁴ bits read. A cluster reads 2 PB per day. How many checksum errors per day would you expect, and why does HDFS still return correct data?

Input

1 error per 1e14 bits · 2 PB/day

Expected output

about 180 silent errors per day, each caught by a checksum

Approach

Bits read = 2 PB × 8.
Errors = bits / 1e14.
Each is caught by the 512-byte checksums and served from another replica.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 3AdvancedLatency · hedged reads

Do hedged reads help the slowest requests?

Problem statement

Each block read normally takes 20 ms, but out of 50 reads, 2 hit a slow disk and take 800 ms. With hedged reads, if a read hasn't finished in 50 ms, a second read starts on another replica (which takes the normal 20 ms) and the faster one wins. Compare the slowest read and the average with and without hedging.

Input

50 reads · 2 at 800 ms · hedge after 50 ms

Expected output

slowest read: without 800 ms, with hedging 70 ms
average: without 51.2 ms, with hedging 22.0 ms
extra reads started: 2

Approach

Without hedging, slow reads take 800 ms.
With hedging, a slow read takes 50 ms + 20 ms for the second replica.
Compare the worst case and the average.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

9 Quiz and revision

Check yourself

Quick revision

Lesson 6.2 in one screen

Read = open() → getBlockLocations (NameNode) → read each block directly from a DataNode → close().
Locations sorted by distance: same node (0) < same rack (2) < other rack (4); ties shuffled.
Lab: client on dn-2 read all 5 blocks locally; off-cluster client spread across dn-1/2/3.
Checksums every 512 bytes, verified by the reading client (end-to-end).
Checksum error → next replica + reportBadBlocks → NameNode re-replicates and deletes the bad copy.
Dead DataNode → next replica; the stream remembers it as dead.
Extras: short-circuit local reads, hedged reads, block scanner (every 504 h).
NameNode never sends data.

Interview and viva questions

How does a client read a file from HDFS?

It opens the file, which asks the NameNode for the locations of the file's blocks. For each block, the NameNode lists the DataNodes holding replicas sorted by distance from the client. The client reads each block directly from the closest DataNode, verifying checksums, then moves to the next block.

How does HDFS choose which replica to read?

The NameNode sorts replicas by network distance from the client: same node first, then same rack, then other racks, using the rack topology. Replicas at the same distance are shuffled so load is spread. The client tries them in that order.

How does HDFS detect corrupted data?

Writers compute a CRC checksum for every 512 bytes, stored in the .meta file next to each block replica. Readers recompute and compare. DataNodes also verify blocks in the background with the block scanner.

What happens when a client finds a corrupt block?

It reads that part from another replica, so the application still gets correct data, and reports the corrupt replica to the NameNode. The NameNode marks it corrupt, creates a new replica from a good copy and deletes the bad one.

What are short-circuit local reads?

When the client and the replica are on the same machine and dfs.client.read.shortcircuit is enabled (with a domain socket path), the client reads the block file directly from local disk using a file descriptor passed by the DataNode, avoiding the TCP data path.

Does reading put load on the NameNode?

Only for metadata: open and block location lookups. The data transfer itself is between client and DataNodes, which is why HDFS reads scale with the number of DataNodes.

Practice problems

Read-path exercises using real offsets, IPs and logs from the lab.

P1Easy

Distance table

Compute the distance from a client on dn-3 (rack2) to dn-1 (rack1), dn-4 (rack2) and dn-3.

Input

client dn-3

Expected output

dn-1 4
dn-4 2
dn-3 0

Show solution and explanation

rack = {"dn-1": 1, "dn-3": 2, "dn-4": 2}
for dn in ["dn-1", "dn-4", "dn-3"]:
    d = 0 if dn == "dn-3" else 2 if rack[dn] == 2 else 4
    print(dn, d)

dn-1 4 dn-4 2 dn-3 0

How it works: dn-1: 4 (other rack), dn-4: 2 (same rack), dn-3: 0 (itself).

P2Easy

Block for an offset

With 1 MB blocks, which block holds byte 1,500,000 of the file, and at what offset inside it?

Input

1,500,000 · 1,048,576

Expected output

1 451424

Show solution and explanation

print(1500000 // 1048576, 1500000 % 1048576)

1 451424

How it works: Block 1 (the second), offset 451,424 inside it: blk_1073741827 in our file.

P3Easy

Count servers

Count how many blocks each DataNode served in the real off-cluster read: dn-2, dn-3, dn-3, dn-3, dn-1.

Input

dn-2 · dn-3 · dn-3 · dn-3 · dn-1

Expected output

Counter({'dn-3': 3, 'dn-2': 1, 'dn-1': 1})

Show solution and explanation

from collections import Counter
print(Counter(["dn-2", "dn-3", "dn-3", "dn-3", "dn-1"]))

Counter({'dn-3': 3, 'dn-2': 1, 'dn-1': 1})

How it works: dn-3: 3, dn-2: 1, dn-1: 1. Random among equal distances; over many reads it evens out.

P4Medium

Which replica healed it?

Before: replicas [.3, .4, .6]. After the bad-block report: [.4, .7, .6]. Which replica was removed and which added?

Input

before / after lists

Expected output

removed: {'172.19.0.3'}  added: {'172.19.0.7'}

Show solution and explanation

before, after = {"172.19.0.3", "172.19.0.4", "172.19.0.6"}, {"172.19.0.4", "172.19.0.7", "172.19.0.6"}
print("removed:", before - after, " added:", after - before)

removed: {'172.19.0.3'} added: {'172.19.0.7'}

How it works: Removed 172.19.0.3 (datanode-2, the corrupted copy); added 172.19.0.7 (datanode-4).

P5Medium

Checksum bytes per block

How many checksum bytes does a 128 MB block need, and what fraction of the data is that?

Input

134,217,728 bytes · 4 B per 512 B

Expected output

1048576 bytes (0.78%)

Show solution and explanation

b = 134217728
c = b // 512 * 4
print(c, "bytes", f"({c / b:.2%})")

1048576 bytes (0.78%)

How it works: 1,048,576 bytes (1 MB) of checksums per 128 MB block: 0.78% overhead for end-to-end integrity.

P6Medium

Parse a LocatedBlock

From the real LocatedBlock text, extract block ID, size, offset and location IPs.

Input

LocatedBlock{…blk_1073741826_1002; getBlockSize()=1048576; … offset=0; locs=[…]}

Expected output

blk_1073741826 1048576 0 ['172.19.0.3', '172.19.0.4', '172.19.0.6']

Show solution and explanation

import re
s = ("LocatedBlock{BP-1507681208-172.19.0.3-1792565333483:blk_1073741826_1002; getBlockSize()=1048576; "
     "corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[172.19.0.3:9866,DS-1,DISK], "
     "DatanodeInfoWithStorage[172.19.0.4:9866,DS-2,DISK], DatanodeInfoWithStorage[172.19.0.6:9866,DS-3,DISK]]")
print(re.search(r"(blk_\d+)", s).group(1), re.search(r"getBlockSize\(\)=(\d+)", s).group(1),
      re.search(r"offset=(\d+)", s).group(1), re.findall(r"(\d+\.\d+\.\d+\.\d+):9866", s))

blk_1073741826 1048576 0 ['172.19.0.3', '172.19.0.4', '172.19.0.6']

How it works: blk_1073741826, 1048576 bytes at offset 0, replicas on .3, .4 and .6.

P7Business problem

Saving by locality

Core network capacity costs Rs 8 lakh per 10 Gbit/s per year. Without locality, a cluster would need 40 Gbit/s of core capacity; with locality, 4 Gbit/s. What is the yearly saving?

Input

Rs 8 lakh per 10 Gbit/s · 40 vs 4 Gbit/s

Expected output

Rs 24 lakh per year

Show solution and explanation

import math
cost = lambda g: math.ceil(g / 10) * 8
print(f"Rs {cost(40) - cost(4)} lakh per year")

Rs 24 lakh per year

How it works: Rs 32 lakh − Rs 8 lakh = Rs 24 lakh per year (illustrative prices), just from reading where the data is.

P8Advanced

Read with corruption and failover

Simulate reading 3 blocks with sorted replicas, where some replicas are corrupt and some DataNodes dead. Return the serving DataNode per block and the list of bad-block reports.

Input

3 blocks · corrupt {(1, dn-2)} · dead {dn-3}

Expected output

served: {0: 'dn-2', 1: 'dn-1', 2: 'dn-2'}  reported bad: [(1, 'dn-2')]

Show solution and explanation

locs = {0: ["dn-2", "dn-1", "dn-3"], 1: ["dn-2", "dn-3", "dn-1"], 2: ["dn-3", "dn-2", "dn-1"]}
corrupt, dead = {(1, "dn-2")}, {"dn-3"}
served, reports = {}, []
for b, nodes in locs.items():
    for dn in nodes:
        if dn in dead:
            continue
        if (b, dn) in corrupt:
            reports.append((b, dn)); continue
        served[b] = dn; break
print("served:", served, " reported bad:", reports)

served: {0: 'dn-2', 1: 'dn-1', 2: 'dn-2'} reported bad: [(1, 'dn-2')]

How it works: Block 0 from dn-2, block 1 from dn-1 (after the corrupt dn-2 copy and dead dn-3), block 2 from dn-2. One bad-block report: block 1 on dn-2.

Real-world scenario questions (asked in company interviews)

Read-path questions test whether you understand locality and integrity, two of HDFS's core promises.

HDFS · internalsFresher

“Explain how HDFS reads a file”

The anatomy-of-a-read question.

The interviewer is checking: NameNode vs DataNode roles and replica choice.

Show a strong answer

Client opens the file; the NameNode returns block locations, each list sorted by distance from the client.
The client reads each block directly from the nearest DataNode, verifying 512-byte checksums, then moves to the next block.
On failure or checksum error it uses the next replica and reports bad replicas. The NameNode never transfers data.

Operations · integrityExperienced

Logs show many “Found Checksum error” warnings from one DataNode

Jobs still succeed, but the warnings keep appearing for different blocks on the same host.

The interviewer is checking: that you treat it as a hardware signal.

Show a strong answer

Clients are reading around corrupt replicas and HDFS re-replicates them, so data is safe for now.
Repeated errors on one host point to a failing disk, controller or memory. Check which volume the blocks are on, SMART data, kernel logs.
Mark the volume failed or decommission the node, replace the hardware, and run fsck to confirm no file has lost all good replicas.

Performance · localityExperienced

Reads from BI tools on the edge node are slow; jobs are fast

Analysts pull large files to an edge node and complain about speed.

The interviewer is checking: remote reads vs local reads.

Show a strong answer

The edge node isn't a DataNode, so every read is remote and limited by its network link; jobs read locally on DataNodes.
Move the heavy work into the cluster (Hive/Spark queries returning small results) instead of copying raw data out.
If raw transfers are needed: faster edge-node networking, parallel readers, or columnar formats so less data is read.

HDFS · reliabilityFresher

“If a disk silently flips a bit, will my report be wrong?”

A manager worries about silent data corruption.

The interviewer is checking: end-to-end checksums.

Show a strong answer

No (unless every replica is bad). Each 512 bytes has a checksum created when the data was written; readers verify it.
A mismatch makes the client read another replica and report the bad one, which HDFS replaces.
Real demo: one byte changed on disk → “Found Checksum error … at 499712” → output identical to the original.

Nearest copy first, and never trust a damaged one

The HDFS read path

Words you'll meet in this lesson

Read facts (Hadoop 3.5.0, seen in the lab)

The read path at three levels

Beginner A library with photocopies

Intermediate Step by step

Advanced Robustness and performance

Kinds of reads

Read-side protections

Checksums

Replica failover

Block scanner

reportBadBlocks

Watch reads on the real cluster

What the NameNode tells the client

Read from outside the DataNodes

Read from a DataNode that holds the blocks

Damage one replica

Read again from datanode-2

The NameNode heals the block

A read that survives a corrupt replica

The read path at work

1. Data-local tasks

2. Short-circuit reads for fast engines

3. Silent disk errors caught

4. Verifying copies between clusters

Sort replicas by distance, like the NameNode

Run it on your PC: who serves your reads?

Same file, two clients, different DataNodes

Reading a file block by block

Worked examples: real problems, solved and traced in memory

How much traffic does data locality save?

How many corrupt chunks per day?

Do hedged reads help the slowest requests?

Check yourself

Quick revision

Lesson 6.2 in one screen

Interview and viva questions

Practice problems

Distance table

Block for an offset

Count servers

Which replica healed it?

Checksum bytes per block

Parse a LocatedBlock

Saving by locality

Read with corruption and failover

Real-world scenario questions (asked in company interviews)

“Explain how HDFS reads a file”

Logs show many “Found Checksum error” warnings from one DataNode

Reads from BI tools on the edge node are slow; jobs are fast

“If a disk silently flips a bit, will my report be wrong?”