How a file is written: the write pipeline

1 The story

One upload, three copies, one trip

Riya uploads 134 MB of canteen orders from datanode-1 with hdfs dfs -put. It finishes in under three seconds, and fsck shows three replicas of every block. Kabir does the arithmetic: “Three copies of 134 MB is 402 MB. Did the client really send 402 MB in three seconds?” Then he switches on debug logging and reads the client's diary: pipeline = [172.19.0.4, 172.19.0.6, 172.19.0.7], 2,186 packets, and after each packet: reply: SUCCESS reply: SUCCESS reply: SUCCESS.

Arjun Sir

No. The client sent each byte once. The NameNode only chose three DataNodes; the client then opened a pipeline: it streams 64 KB packets to the first DataNode, which writes each packet to its disk and forwards it to the second, which does the same for the third. Each packet's acknowledgement travels back down the chain.

Riya

So the network load is shared by the DataNodes.

Arjun Sir

Exactly, and the copies are made at the same time, not one after another. Today we'll read the real logs of this pipeline: the NameNode choosing nodes, the packets, the acknowledgements, and what happens when we kill a DataNode in the middle of a block.

2 Definition

The HDFS write pipeline

Formal definition (write this in exams)

To write a file, the client calls create on the NameNode, which checks permissions and that the file doesn't exist, records it in the namespace (with a lease for the writer) and returns an output stream. As data is written, the client's DataStreamer asks the NameNode to allocate a block; the NameNode returns a new block ID and a list of DataNodes chosen by the placement policy, ordered as a pipeline. The client splits data into packets (about 64 KB of data, made of 512-byte chunks each with a 4-byte checksum) and sends them to the first DataNode, which stores and forwards each packet to the next, and so on. Acknowledgements return in the reverse direction; the client keeps each packet in an ack queue until all DataNodes in the pipeline have acknowledged it. When a block is full the pipeline is closed and a new block is allocated. Finally the client calls complete, and the NameNode closes the file once the minimum replication is reached. If a DataNode fails, the client removes it, the NameNode bumps the block's generation stamp, and writing continues on the remaining DataNodes; the block is re-replicated later.

In simple words

The NameNode picks three DataNodes. The client sends the data in small packets to the first one, which passes them to the second, which passes them to the third. Each packet is confirmed by all three before the client forgets it.

Words you'll meet in this lesson

PipelineThe ordered chain of DataNodes that receive one block, e.g. dn-1 → dn-3 → dn-4.

PacketUnit sent down the pipeline: up to 64,512 data bytes (126 chunks).

Chunk512 bytes of data + 4-byte checksum = 516 bytes.

AckAcknowledgement of one packet; returns from the last DataNode to the client.

Data queue / ack queuePackets waiting to be sent / sent but not yet acknowledged by everyone.

LeaseThe NameNode's record that one client is writing a file. Only one writer at a time.

_COPYING_hdfs dfs -put writes to name._COPYING_ and renames it when done.

Generation stampVersion number of a block (_1023). Bumped during recovery so stale replicas are recognised.

Write facts (Hadoop 3.5.0, measured in the lab)

Packet	`dfs.client-write-packet-size` = 65,536; real log: 126 chunks × 516 bytes = 65,016 bytes, 64,512 of them data
Our 134 MB upload	2 blocks, 2,186 packets, every one acknowledged by 3 DataNodes, about 2.6 s
Pipeline order	Local DataNode first (if the writer runs on one), then other nodes, each hop to the nearest remaining node
File visible	As soon as created; length grows block by block; final after `complete`
Close condition	Each block has at least `dfs.namenode.replication.min` (1) replica; the rest catch up
On DataNode failure	Policy `dfs.client.block.write.replace-datanode-on-failure.policy` = DEFAULT (may add a replacement node)

The write pipeline at three levels

Pick your level. New to this? Start at Beginner and climb.

Beginner A bucket chain

Villagers pass buckets of water hand to hand. Each person also fills their own pot from every bucket before passing it on.
The water carrier (client) walks only to the first person. The supervisor (NameNode) just decided who stands in the chain.
When the last person shouts “got it”, the shout travels back, and the carrier knows that bucket is safe.

Intermediate Step by step

create: NameNode checks and records the file; client gets a lease.
addBlock: NameNode returns block ID + 3 DataNodes (placement policy, Lesson 5.3).
Pipeline setup: client connects to DN1, which connects to DN2, which connects to DN3.
Streaming: packets flow down; each DataNode verifies checksums, writes to disk, forwards; acks flow up.
Next block: when 128 MB are written, close the pipeline and repeat addBlock.
complete: NameNode marks the file closed; put renames ._COPYING_ to the final name.

Advanced Failures and guarantees

Setup failure (a DataNode can't be reached): the client abandons the block, excludes that node, and asks for a new block.
Failure while streaming: unacknowledged packets go back to the data queue, the bad DataNode is removed, the NameNode bumps the generation stamp, and streaming resumes on the survivors; under-replication is fixed afterwards.
Visibility: readers see data up to the last completed block, or what hflush() has pushed; hsync() also forces it to disk.
Single writer, append only: HDFS files can't be edited in the middle.

Exam tip: “Explain the anatomy of a file write in HDFS”: client → create() → NameNode checks and adds the file → client's DataStreamer asks for blocks → NameNode returns a list of DataNodes → pipeline DN1 → DN2 → DN3 → data sent as packets; ack queue holds packets until all DataNodes acknowledge → close() → complete() on the NameNode. Mention failure handling and the default rack-aware placement.

3 Types and ways to use it

Who talks to whom during a write

Step	Between	Carries	Seen in our logs as
create	Client → NameNode	Path, permissions, replication, block size	`cmd=create src=…orders_x30.csv._COPYING_`
addBlock	Client → NameNode	Request for a new block	`BLOCK* allocate blk_1073741840_1016, replicas=…`
Pipeline + packets	Client → DN1 → DN2 → DN3	Data + checksums	`Receiving … src: /172.19.0.4 dest: /172.19.0.6`
Acks	DN3 → DN2 → DN1 → Client	Per-packet status	`reply: SUCCESS reply: SUCCESS reply: SUCCESS`
Block received	DataNodes → NameNode	Incremental block report	`addStoredBlock: … is added to blk_…`
complete / rename	Client → NameNode	Close the file	`completeFile`, `cmd=rename`

Ways to write to HDFS

hdfs dfs -put

Copies a local file; writes to ._COPYING_ and renames at the end.

Shell

FileSystem.create()

The Java API used by every tool (Lesson 6.5).

API

Jobs (MapReduce, Spark, Hive)

Each task writes its own file through the same pipeline, in parallel.

Engines

append / hflush

Add to the end of a closed file, or make data visible while still writing (logs, streams).

Special

Writing from a machine that is a DataNode puts the first replica on that machine's own disk. That's fast, but if all your ingestion runs on one DataNode, its disks fill first. Spread ingestion clients, or write from edge nodes.

4 Step by step

Read the pipeline in real logs

On the Module 5 lab cluster (4 DataNodes at the time of these runs; a 5th is added in Lesson 6.4). IP addresses at the time: datanode-1 = 172.19.0.4, datanode-2 = .3, datanode-3 = .6, datanode-4 = .7. The upload runs on datanode-1 with client debug logging switched on (the log configuration is inside write_trace.py, Part 4b).

Upload with debug logging
```
for i in $(seq 30); do cat /tmp/canteen_orders.csv; done > /tmp/orders_x30.csv
HADOOP_CLIENT_OPTS="-Dlog4j.configuration=file:/tmp/dbg.properties" \
  hdfs dfs -put -f /tmp/orders_x30.csv /user/hadoop/data/orders_x30.csv
```
09:32:52,867 DEBUG DataStreamer - stage=PIPELINE_SETUP_CREATE, block==null 09:32:52,956 DEBUG DataStreamer - pipeline = [DatanodeInfoWithStorage[172.19.0.4:9866,…], DatanodeInfoWithStorage[172.19.0.6:9866,…], DatanodeInfoWithStorage[172.19.0.7:9866,…]], blk_1073741840_1016 09:32:52,962 DEBUG DataStreamer - Connecting to datanode 172.19.0.4:9866 09:32:53,055 DEBUG DataStreamer - blk_1073741840_1016 sending packet seqno: 0 offsetInBlock: 0 lastPacketInBlock: false lastByteOffsetInBlock: 64512 09:32:53,058 DEBUG DataStreamer - blk_1073741840_1016 sending packet seqno: 1 offsetInBlock: 64512 lastPacketInBlock: false lastByteOffsetInBlock: 129024 09:32:53,213 DEBUG DataStreamer - DFSClient seqno: 0 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1840487 flag: 0 flag: 0 flag: 0 … 09:32:55,680 DEBUG DataStreamer - blk_1073741840_1016 sending packet seqno: 2082 offsetInBlock: 134217728 lastPacketInBlock: true lastByteOffsetInBlock: 134217728 09:32:55,726 DEBUG DataStreamer - Closing old block BP-1507681208-…:blk_1073741840_1016 09:32:55,838 DEBUG DataStreamer - pipeline = [DatanodeInfoWithStorage[172.19.0.4:9866,…], DatanodeInfoWithStorage[172.19.0.7:9866,…], DatanodeInfoWithStorage[172.19.0.6:9866,…]], blk_1073741841_1017
The client connects only to the first DataNode (its own, .4). Each packet carries 64,512 data bytes; the ack for packet 0 lists three SUCCESS replies, one per DataNode. Packet 2082 is the empty “last packet” that closes the block at exactly 134,217,728 bytes; then a new pipeline is built for block 2.
The NameNode's side
```
docker compose logs namenode | grep -E "orders_x30|blk_1073741840|blk_1073741841"
```
audit - allowed=true ugi=hadoop ip=/172.19.0.4 cmd=create src=/user/hadoop/data/orders_x30.csv._COPYING_ perm=hadoop:supergroup:rw-r--r-- StateChange - BLOCK* allocate blk_1073741840_1016, replicas=172.19.0.4:9866, 172.19.0.6:9866, 172.19.0.7:9866 for /user/hadoop/data/orders_x30.csv._COPYING_ BlockStateChange - BLOCK* addStoredBlock: 172.19.0.7:9866 is added to blk_1073741840_1016 (size=134217728) BlockStateChange - BLOCK* addStoredBlock: 172.19.0.6:9866 is added to blk_1073741840_1016 (size=134217728) BlockStateChange - BLOCK* addStoredBlock: 172.19.0.4:9866 is added to blk_1073741840_1016 (size=134217728) StateChange - BLOCK* allocate blk_1073741841_1017, replicas=172.19.0.4:9866, 172.19.0.7:9866, 172.19.0.6:9866 for …._COPYING_ StateChange - DIR* completeFile: /user/hadoop/data/orders_x30.csv._COPYING_ is closed by DFSClient_NONMAPREDUCE_2053878093_1 audit - allowed=true ugi=hadoop ip=/172.19.0.4 cmd=rename src=…/orders_x30.csv._COPYING_ dst=/user/hadoop/data/orders_x30.csv
create → allocate (with three chosen DataNodes) → each DataNode reports the finished replica (the last one in the pipeline finishes first) → next block → completeFile → rename. No data in any of these messages.
Each DataNode's side
```
docker logs lab5-datanode-1 | grep blk_1073741840     # and datanode-3, datanode-4
```
== datanode-1 (172.19.0.4) Receiving …:blk_1073741840_1016 src: /172.19.0.4:54570 dest: /172.19.0.4:9866 PacketResponder: …:blk_1073741840_1016, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[172.19.0.6:9866, 172.19.0.7:9866] terminating == datanode-3 (172.19.0.6) Receiving …:blk_1073741840_1016 src: /172.19.0.4:44414 dest: /172.19.0.6:9866 PacketResponder: …:blk_1073741840_1016, type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[172.19.0.7:9866] terminating == datanode-4 (172.19.0.7) Receiving …:blk_1073741840_1016 src: /172.19.0.6:49840 dest: /172.19.0.7:9866 PacketResponder: …:blk_1073741840_1016, type=LAST_IN_PIPELINE terminating
The chain is visible in the src addresses: datanode-3 received from datanode-1 (.4), datanode-4 from datanode-3 (.6). Each knows how many DataNodes are downstream of it.
Kill a DataNode in the middle of a block
```
# upload 422 MB from datanode-1; after packet 200 of the first block:
docker kill lab5-datanode-3
```
09:34:40,734 DEBUG DataStreamer - pipeline = [172.19.0.4:9866, 172.19.0.7:9866, 172.19.0.6:9866], blk_1073741847_1023 09:34:41,910 WARN DataStreamer - Exception for BP-…:blk_1073741847_1023 java.io.IOException: Bad response ERROR for BP-…:blk_1073741847_1023 from datanode DatanodeInfoWithStorage[172.19.0.6:9866,…] 09:34:42,010 WARN DataStreamer - Error Recovery for BP-…:blk_1073741847_1023 in pipeline [172.19.0.4:9866, 172.19.0.7:9866, 172.19.0.6:9866] … 09:34:42,127 DEBUG DataStreamer - pipeline = [172.19.0.4:9866, 172.19.0.7:9866], blk_1073741847_1023 NameNode: bumpBlockGenerationStamp(…:blk_1073741847_1024, client=DFSClient_NONMAPREDUCE_1852730810_1) success NameNode: updatePipeline(blk_1073741847_1023, newGS=1024, newLength=40061952, newNodes=[172.19.0.4:9866, 172.19.0.7:9866], …) NameNode: BLOCK* addStoredBlock: 172.19.0.7:9866 is added to blk_1073741847_1024 (size=134217728) NameNode: BLOCK* addStoredBlock: 172.19.0.4:9866 is added to blk_1073741847_1024 (size=134217728) NameNode: BLOCK* addStoredBlock: 172.19.0.3:9866 is added to blk_1073741847_1024 (size=134217728) (19 s later) EXIT=0
The upload succeeded. After 40,061,952 bytes the pipeline lost datanode-3; the client kept going with two DataNodes, the generation stamp changed from 1023 to 1024 (so the dead node's partial replica is recognised as stale), and the NameNode later copied the block to datanode-2 to get back to three replicas. (DatanodeInfo strings shortened.)
And a failure while setting up a pipeline
```
# another upload; datanode-4 killed just before the second block
```
WARN DataStreamer - Exception in createBlockOutputStream blk_1073741843_1019 java.io.IOException: Got error, status=ERROR, status message , ack with firstBadLink as 172.19.0.7:9866 WARN DataStreamer - Excluding datanode DatanodeInfoWithStorage[172.19.0.7:9866,…] DEBUG DataStreamer - pipeline = [172.19.0.4:9866, 172.19.0.3:9866, 172.19.0.6:9866], blk_1073741844_1020
Before any data flowed, datanode-1 reported that the next link (.7) was bad. The client abandoned block …843, told the NameNode to exclude .7, and got a new block …844 with datanode-2 (.3) instead. The final file has blocks …842, …844, …845, …846: block …843 never existed.

A write with replication 3 can succeed with fewer than 3 replicas if DataNodes fail during it; HDFS fixes the replica count afterwards. Data is still safe on the survivors, but check fsck for under-replicated blocks after cluster trouble, especially on small clusters.

5 See it happen

A block travels down the pipeline

The first block of orders_x30.csv, using the real log lines from step 1 to 3.

Writing blk_1073741840 6 steps

What you see

Behind the screen

6 Real-world use

The write pipeline at work

1. Bulk ingestion from edge nodes

Daily exports land on an edge node and are pushed with hdfs dfs -put or DistCp. The edge node's network link limits speed, but each byte leaves it only once; replication is the DataNodes' work.

Edge node→Pipeline→3 replicas

2. Logs written while still open

Log collectors and streaming sinks keep a file open for minutes and call hflush() regularly so readers can see recent lines before the file is closed.

Write→hflush→Visible to readers

3. One slow disk slows many writes

A pipeline moves at the speed of its slowest DataNode. Operations teams watch per-DataNode write latency and exclude or repair slow nodes, because one bad disk can delay every job that writes through it.

Slow disk→Slow acks→Slow jobs

4. Jobs survive DataNode failures

When a DataNode dies during a big job, thousands of task writes recover their pipelines exactly as in step 4 and carry on; the NameNode restores full replication afterwards. Users usually never notice.

Node dies→Pipelines recover→Re-replication later

Simplified; organisations differ in tools and policies.

7 Run the code

Simulate a pipeline with acknowledgements

This program sends packets down a three-DataNode pipeline. Each packet stays in the ack queue until every DataNode has stored it. Watch which packets are still unacknowledged at each step.

from collections import deque

pipeline = ["dn-1", "dn-3", "dn-4"]
stored = {dn: [] for dn in pipeline}
data_queue = deque(range(5))          # packet numbers 0..4
ack_queue = deque()

while data_queue or ack_queue:
    if data_queue:
        p = data_queue.popleft()
        for dn in pipeline:            # dn-1 stores and forwards, and so on
            stored[dn].append(p)
        ack_queue.append(p)
        print(f"sent packet {p}  waiting for acks: {list(ack_queue)}")
    if len(ack_queue) > 1 or not data_queue:
        done = ack_queue.popleft()     # acks come back in order
        print(f"  ack for packet {done} from {len(pipeline)} DataNodes")

for dn, packets in stored.items():
    print(dn, "has", packets)

sent packet 0 waiting for acks: [0] sent packet 1 waiting for acks: [0, 1] ack for packet 0 from 3 DataNodes sent packet 2 waiting for acks: [1, 2] ack for packet 1 from 3 DataNodes sent packet 3 waiting for acks: [2, 3] ack for packet 2 from 3 DataNodes sent packet 4 waiting for acks: [3, 4] ack for packet 3 from 3 DataNodes ack for packet 4 from 3 DataNodes dn-1 has [0, 1, 2, 3, 4] dn-3 has [0, 1, 2, 3, 4] dn-4 has [0, 1, 2, 3, 4]

for dn in pipeline: # dn-1 stores and forwards, and so onIn real HDFS each DataNode writes the packet to disk and forwards it at the same time; the client only sends to dn-1.
ack_queue.append(p)The client keeps the packet in memory until it is acknowledged, so it can resend it if a DataNode fails.
done = ack_queue.popleft() # acks come back in orderAcks return in packet order, from the last DataNode back to the client.

Run it on your PC: trace your own write pipeline

This script uploads a 134 MB file from datanode-1 with client debug logging, then summarises the pipelines, packets and acknowledgements from the log. Save it as write_trace.py in the hdfs-lab folder (it needs canteen_orders.csv copied to /tmp inside datanode-1, as in Lesson 5.1).

Pipelines and packets of one upload

# write_trace.py: upload a file from datanode-1 with client debug logging,
# then summarise what the write pipeline did. Run in the hdfs-lab folder.
import re
import subprocess

LOG4J = """log4j.rootLogger=WARN, err
log4j.appender.err=org.apache.log4j.ConsoleAppender
log4j.appender.err.Target=System.err
log4j.appender.err.layout=org.apache.log4j.PatternLayout
log4j.appender.err.layout.ConversionPattern=%d{HH:mm:ss,SSS} %m%n
log4j.logger.org.apache.hadoop.hdfs.DataStreamer=DEBUG
"""
CLIENT = "lab5-datanode-1"           # a DataNode, so replica 1 is written locally

def sh(cmd):
    return subprocess.run(cmd, shell=True, capture_output=True, text=True)

sh(f"docker exec -i {CLIENT} bash -c 'cat > /tmp/w.properties' <<'X'\n{LOG4J}X")
sh(f"docker exec {CLIENT} bash -c 'for i in $(seq 30); do cat /tmp/canteen_orders.csv; done > /tmp/orders_x30.csv'")
log = sh(f"docker exec {CLIENT} bash -c 'HADOOP_CLIENT_OPTS=-Dlog4j.configuration=file:/tmp/w.properties "
         f"hdfs dfs -put -f /tmp/orders_x30.csv /user/hadoop/data/orders_x30.csv'").stderr

ip_to_name = {}
for n in sh("docker ps --format '{{.Names}}'").stdout.split():
    ip = sh("docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' " + n).stdout.strip()
    ip_to_name[ip] = n.replace("lab5-", "")

for m in re.finditer(r"pipeline = \[(.*?)\]\], (blk_\d+)", log):
    ips = re.findall(r"(\d+\.\d+\.\d+\.\d+):9866", m.group(1))
    print(m.group(2), "pipeline:", " -> ".join(ip_to_name.get(ip, ip) for ip in ips))
packets = len(re.findall(r"sending packet seqno", log))
acks = len(re.findall(r"reply: SUCCESS reply: SUCCESS reply: SUCCESS", log))
first = re.search(r"seqno: 0 offsetInBlock: 0 lastPacketInBlock: false lastByteOffsetInBlock: (\d+)", log)
times = re.findall(r"^(\d\d:\d\d:\d\d,\d+)", log, re.M)
print(f"packets sent: {packets}, acknowledged by all 3 DataNodes: {acks}")
print(f"data bytes per full packet: {first.group(1)}")
print(f"first log line {times[0]}, last {times[-1]}")

# In the hdfs-lab folder:
#   docker cp canteen_orders.csv lab5-datanode-1:/tmp/
#   python3 write_trace.py

blk_1073741860 pipeline: datanode-1 -> datanode-3 -> datanode-4 blk_1073741861 pipeline: datanode-1 -> datanode-2 -> datanode-5 packets sent: 2186, acknowledged by all 3 DataNodes: 2186 data bytes per full packet: 64512 first log line 09:43:26,161, last 09:43:28,775

Real output, with 5 DataNodes. The first replica is always on datanode-1 (the writer). The pipeline order follows network distance from the writer, so block 2 visits datanode-2 (same rack) before datanode-5 (other rack); the replicas still cover both racks.

8 Line by line

Pipeline recovery, one line at a time

Follow the client when the middle DataNode fails: unacknowledged packets go back to the front of the data queue and the generation stamp changes.

Worked examples: real problems, solved and traced in memory

Three write-path calculations engineers make: upload time, network load on racks, and how much data is at risk while a pipeline runs short. Each runs live and is traced in memory.

Example 1BeginnerPlanning · ingestion

How long will this upload take?

Problem statement

An edge node with a 10 Gbit/s link (about 1,100 MB/s usable) uploads 2 TB of logs with replication 3. DataNode disks are not the bottleneck. Estimate the time, and compare with a client that had to send all 3 copies itself.

Input

2 TB · 1,100 MB/s · replication 3

Expected output

pipeline: 0.5 h   sending 3 copies itself: 1.6 h

Approach

Pipeline: the client sends the data once.
Time = size / link speed.
Without a pipeline it would send 3×.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 2IntermediateNetwork · racks

Cross-rack traffic for a day of writes

Problem statement

Jobs write 30 TB per day with replication 3 from tasks running on DataNodes. With the default placement, each block crosses racks once. How much traffic crosses the core switch per day, and the average rate in Gbit/s?

Input

30 TB/day · 1 cross-rack copy per block

Expected output

30 TB/day across racks = 3.1 Gbit/s on average

Approach

Cross-rack bytes = data written × 1.
Average rate = bytes / 86,400 s.
Convert to Gbit/s.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 3AdvancedRisk · recovery

Data with fewer than 3 replicas during failures

Problem statement

During a busy hour, 200 pipelines are writing at once, each with on average 60 MB of its current block written. A DataNode that is part of 40 of these pipelines dies. Those pipelines continue with 2 replicas. How much just-written data briefly has only 2 replicas, and how long until re-replication at 500 MB/s cluster-wide?

Input

40 affected pipelines · 64 MB average per block when complete · 500 MB/s

Expected output

~5,120 MB with 2 replicas, fixed in ~10 s

Approach

Affected blocks finish with 2 replicas: about 40 blocks of up to 128 MB.
Data at reduced redundancy ≈ 40 × 128 MB.
Time to fix = data / repair rate.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

9 Quiz and revision

Check yourself

Quick revision

Lesson 6.1 in one screen

Write = create (NameNode) → addBlock (DataNode list) → pipeline DN1 → DN2 → DN3 → complete.
Client sends each byte once; DataNodes forward. Replicas are written in parallel.
Packets: ~64 KB (64,512 data bytes = 126 × 512-byte chunks, each + 4-byte checksum).
Acks flow back; packets stay in the ack queue until all DataNodes confirm.
Block full → close pipeline → addBlock again. File visible under ._COPYING_ until renamed (put).
DataNode fails mid-block → exclude it, bump generation stamp, continue on survivors; re-replicate later.
Setup failure → “firstBadLink” → abandon block, exclude node, new block.
Pipeline order = network distance from the writer; replicas still span racks.

Interview and viva questions

How does a client write a file to HDFS?

It asks the NameNode to create the file, then for each block asks for a list of DataNodes. It streams the block as packets to the first DataNode, which stores and forwards them to the second, which forwards to the third. Acknowledgements come back along the pipeline. When all data is written, the client tells the NameNode the file is complete.

Does the data pass through the NameNode during a write?

No. The NameNode only handles metadata: creating the file, allocating blocks and recording finished replicas. Data flows from the client to DataNodes and between DataNodes.

What happens if a DataNode fails during a write?

The client detects the error, removes that DataNode from the pipeline, moves unacknowledged packets back to the data queue, gets a new generation stamp for the block from the NameNode, and continues with the remaining DataNodes (possibly adding a replacement). The NameNode re-replicates the block later.

What is the generation stamp?

A version number attached to each block (the part after the last underscore in blk_1073741847_1024). It is increased during pipeline recovery or append so that replicas left behind on failed DataNodes are recognised as stale and deleted.

When can other clients read a file that is being written?

They can always see the file, and data in completed blocks. Data in the block being written becomes visible after hflush() (or when the block or file is closed). Only one client can write to a file at a time.

Why is data sent in packets instead of whole blocks?

So that DataNodes can start forwarding while still receiving, keeping all three copies moving at once, and so that a failure only requires resending a few small packets, not a whole 128 MB block.

Practice problems

Pipeline exercises built on the numbers from the real logs.

P1Easy

Chunk arithmetic

A packet has 126 chunks of 512 data bytes, each with a 4-byte checksum. How many data bytes and total bytes per packet?

Input

126 chunks · 512 + 4 bytes

Expected output

64512 data bytes, 65016 bytes with checksums

Show solution and explanation

print(126 * 512, "data bytes,", 126 * 516, "bytes with checksums")

64512 data bytes, 65016 bytes with checksums

How it works: 64,512 data bytes and 65,016 bytes in total, exactly the packetSize in the client log.

P2Easy

Packets per block

How many full 64,512-byte packets fill a 134,217,728-byte block?

Input

134,217,728 / 64,512

Expected output

Show solution and explanation

import math
print(math.ceil(134217728 / 64512))

2081

How it works: 2,081 data packets (the last one partly full). The log's sequence numbers for block 1 run 0 to 2082 because it also counts small control packets, such as the empty last packet that closes the block.

P3Easy

Second block size

The file is 140,760,270 bytes with 128 MB blocks. How big is the second block?

Input

140,760,270 − 134,217,728

Expected output

Show solution and explanation

print(140760270 - 134217728)

6542542

How it works: 6,542,542 bytes, matching addStoredBlock … (size=6542542).

P4Medium

Order of the chain

From the DataNode “Receiving … src: … dest: …” lines, reconstruct the pipeline order.

Input

src/dest pairs from the real logs

Expected output

172.19.0.4 -> 172.19.0.6 -> 172.19.0.7

Show solution and explanation

pairs = [("172.19.0.6", "172.19.0.7"), ("172.19.0.4", "172.19.0.4"), ("172.19.0.4", "172.19.0.6")]
nxt = {s: d for s, d in pairs if s != d}
node, chain = "172.19.0.4", ["172.19.0.4"]
while node in nxt:
    node = nxt[node]; chain.append(node)
print(" -> ".join(chain))

172.19.0.4 -> 172.19.0.6 -> 172.19.0.7

How it works: 172.19.0.4 → 172.19.0.6 → 172.19.0.7: datanode-1 → datanode-3 → datanode-4.

P5Medium

Where did recovery happen?

updatePipeline reported newLength=40061952. How many packets had been fully acknowledged, and what fraction of the block was written?

Input

40,061,952 bytes · 64,512 per packet · 128 MB block

Expected output

621 packets; 30% of the block

Show solution and explanation

print(40061952 // 64512, "packets;", f"{40061952 / 134217728:.0%} of the block")

621 packets; 30% of the block

How it works: 621 full packets, about 30% of the block, before datanode-3 died. The rest was written to two DataNodes.

P6Medium

Abandoned block IDs

The final file has blocks …842, …844, …845, …846. Which block ID was abandoned?

Input

842 · 844 · 845 · 846

Expected output

[843]

Show solution and explanation

ids = [842, 844, 845, 846]
print([i for i in range(ids[0], ids[-1] + 1) if i not in ids])

[843]

How it works: …843, the block whose pipeline setup failed with firstBadLink 172.19.0.7.

P7Business problem

Network cost of naive copying

A company ingests 5 TB a day from a branch office over a 1 Gbit/s link (about 110 MB/s). With a pipeline the link carries the data once. How many hours a day would the link be busy if it had to carry 3 copies?

Input

5 TB/day · 110 MB/s

Expected output

once: 13.2 h/day, three copies: 39.7 h/day

Show solution and explanation

hours = 5 * 1024 * 1024 / 110 / 3600
print(f"once: {hours:.1f} h/day, three copies: {hours * 3:.1f} h/day")

once: 13.2 h/day, three copies: 39.7 h/day

How it works: About 13 hours a day once, 40 hours for three copies: impossible. The pipeline makes remote ingestion feasible.

P8Advanced

Simulate recovery with resend

Simulate 6 packets where dn-3 fails after acknowledging packet 2. Count packets each surviving DataNode stores, including resent ones (stored once each).

Input

6 packets · dn-3 fails after packet 2

Expected output

{'dn-1': [0, 1, 2, 3, 4, 5], 'dn-3': [0, 1, 2], 'dn-4': [0, 1, 2, 3, 4, 5]}

Show solution and explanation

pipeline = ["dn-1", "dn-3", "dn-4"]
stored = {d: set() for d in pipeline}
for p in range(6):
    if p == 3:
        pipeline.remove("dn-3")
    for d in pipeline:
        stored[d].add(p)
print({d: sorted(s) for d, s in stored.items()})

{'dn-1': [0, 1, 2, 3, 4, 5], 'dn-3': [0, 1, 2], 'dn-4': [0, 1, 2, 3, 4, 5]}

How it works: dn-1 and dn-4 hold all 6 packets; dn-3 only 0–2, a stale partial replica that the generation stamp lets HDFS discard.

Real-world scenario questions (asked in company interviews)

The write path is a favourite deep-dive question. Real log lines make your answer stand out.

HDFS · internalsFresher

“Walk me through what happens when I run hdfs dfs -put”

The classic anatomy-of-a-write question.

The interviewer is checking: the order of steps and who talks to whom.

Show a strong answer

Client asks the NameNode to create file._COPYING_; NameNode checks permissions and existence and grants a lease.
For each block the NameNode returns a block ID and three DataNodes (local, other rack, same rack as the second).
Client streams ~64 KB packets to DN1, which stores and forwards to DN2, then DN3; acks return; packets wait in the ack queue until all confirm.
After the last block the client calls complete; the NameNode closes the file; put renames it to the final name.

HDFS · failureExperienced

A DataNode crashes during a large write. Does the job fail?

A Spark job is writing 2 TB when a worker node reboots.

The interviewer is checking: pipeline recovery details.

Show a strong answer

Each affected writer detects the failure (bad ack or timeout), removes the DataNode, puts unacknowledged packets back in its data queue and asks the NameNode to update the pipeline.
The NameNode bumps the block's generation stamp so the dead node's partial replica is stale; writing continues on the survivors (a replacement may be added depending on the policy).
The NameNode re-replicates the block afterwards. The job continues; only tasks running on the rebooted node itself are retried by YARN/Spark.

Performance · writesExperienced

Writes became slow cluster-wide after one disk started failing

Job write stages are 3× slower than last week. No DataNode is dead.

The interviewer is checking: that you know a pipeline runs at the speed of its slowest member.

Show a strong answer

Every write that includes the slow DataNode waits for its acks, so one bad disk slows many pipelines.
Look at DataNode metrics/logs for slow writes (e.g. “Slow BlockReceiver write data to disk” warnings) and the NameNode's slow-node reports.
Fix: mark the failing volume bad or decommission the node, replace the disk; consider enabling slow-node detection and exclusion.

Design · consistencyFresher

“Can two programs write to the same HDFS file at once?”

A student wants two scripts to append to one log file in HDFS.

The interviewer is checking: single-writer semantics and leases.

Show a strong answer

No. HDFS gives one writer a lease on a file; a second writer gets an exception until the lease is released or expires.
Design instead: each writer writes its own file (e.g. one per task or per hour) in the same directory, and readers process the directory.
This is exactly how MapReduce and Spark output works: part-00000, part-00001, …

How a file is written: the write pipeline

One upload, three copies, one trip

The HDFS write pipeline

Words you'll meet in this lesson

Write facts (Hadoop 3.5.0, measured in the lab)

The write pipeline at three levels

Beginner A bucket chain

Intermediate Step by step

Advanced Failures and guarantees

Who talks to whom during a write

Ways to write to HDFS

hdfs dfs -put

FileSystem.create()

Jobs (MapReduce, Spark, Hive)

append / hflush

Read the pipeline in real logs

Upload with debug logging

The NameNode's side

Each DataNode's side

Kill a DataNode in the middle of a block

And a failure while setting up a pipeline

A block travels down the pipeline

The write pipeline at work

1. Bulk ingestion from edge nodes

2. Logs written while still open

3. One slow disk slows many writes

4. Jobs survive DataNode failures

Simulate a pipeline with acknowledgements

Run it on your PC: trace your own write pipeline

Pipelines and packets of one upload

Pipeline recovery, one line at a time

Worked examples: real problems, solved and traced in memory

How long will this upload take?

Cross-rack traffic for a day of writes

Data with fewer than 3 replicas during failures

Check yourself

Quick revision

Lesson 6.1 in one screen

Interview and viva questions

Practice problems

Chunk arithmetic

Packets per block

Second block size

Order of the chain

Where did recovery happen?

Abandoned block IDs

Network cost of naive copying

Simulate recovery with resend

Real-world scenario questions (asked in company interviews)

“Walk me through what happens when I run hdfs dfs -put”

A DataNode crashes during a large write. Does the job fail?

Writes became slow cluster-wide after one disk started failing

“Can two programs write to the same HDFS file at once?”