The life of a file and a job, end to end

1 The story

Two commands, two hundred messages

Riya types two commands. The first puts a 300 MB file of canteen orders into HDFS. The second runs a job that totals the orders per item. Both finish in under a minute, and her screen shows only a few lines. “That was easy,” she says. Arjun Sir smiles and turns her laptop around. “In that minute, about a dozen machines sent each other a few hundred messages on your behalf. Let's follow every one of them.”

Riya

A few hundred messages? For one file?

Arjun Sir

For a 300 MB file: three blocks, each with three replicas, sent as thousands of small packets, with acknowledgements coming back for each. Plus the calls to the NameNode to create the file, get each block, and close it.

Kabir

And the job is even more?

Arjun Sir

The job starts its own manager, the ApplicationMaster, which asks for containers, runs map tasks next to the blocks, moves their output to the reducers and finally writes a new file into HDFS. Which then has its own life!

Riya

I want to see all of it in order.

Arjun Sir

That's today. This lesson is the bridge between the big picture (Lesson 3.1) and the deep dives in Modules 5 to 11. If you can narrate both lives, step by step, you can answer half the questions in any Hadoop interview.

2 Definition

The write path, the read path and the job lifecycle

Formal definition (write this in exams)

The HDFS write path: the client asks the NameNode to create the file; for each block it calls addBlock, receives a block ID and a list of target DataNodes chosen by the placement policy, and streams the data as packets through a pipeline of those DataNodes, each storing and forwarding the packet and returning acknowledgements; finally the client calls complete. The read path: the client calls open, receives the block locations sorted by network distance, and reads each block directly from the closest DataNode, verifying checksums. The job lifecycle on YARN: the client computes input splits, copies the job resources to HDFS and submits the application to the ResourceManager; the RM launches an ApplicationMaster, which requests containers (preferably on nodes holding each split), runs map tasks, lets reducers shuffle and merge map outputs, runs reduce tasks that write output to HDFS, commits the output and unregisters from the RM.

In simple words

Writing a file: ask the NameNode “where should block 1 go?”, send it to three DataNodes in a chain, repeat for each block, then tell the NameNode “done”. Reading: ask the NameNode “where are the blocks?”, then fetch each block from the nearest DataNode. Running a job: hand the job to the ResourceManager, which starts a small manager for your job; that manager runs map tasks next to the data, gathers their results to reducers, and writes the answer back into HDFS.

Words you'll meet in this lesson

PipelineThe chain of DataNodes (usually 3) a block is written through: client → DN1 → DN2 → DN3.

PacketA small piece of a block (64 KB by default) sent through the pipeline. Each packet is acknowledged.

LeaseA lock the NameNode gives one writer, so only one client writes a file at a time.

Staging directoryA folder in HDFS where the client puts the job's jar, configuration and split list before submitting.

MRAppMasterMapReduce's ApplicationMaster: plans and supervises one job's tasks.

SpillWhen a map task's in-memory buffer fills (80% of 100 MB by default), its sorted contents are written to local disk.

Output committerMoves task outputs from a temporary folder to the final output folder only when tasks succeed.

_SUCCESSAn empty file written into the output directory when a MapReduce job completes successfully.

Numbers in the life of a file and a job (defaults)

Block size	128 MB (`dfs.blocksize`)
Packet size	64 KB (`dfs.client-write-packet-size`)
Checksum	CRC32C for every 512 bytes, stored next to each block replica
Map output sort buffer	100 MB (`mapreduce.task.io.sort.mb`), spilled to disk at 80%
Reducers start fetching	after 5% of maps finish (`mapreduce.job.reduce.slowstart.completedmaps` = 0.05)
Task attempts	up to 4 per task
Output files	`part-r-00000`, `part-r-00001`… (one per reducer) + `_SUCCESS`

The two lives at three levels

Pick your level. New to this? Start at Beginner and climb.

Beginner A courier and a census

Writing a file is like sending a parcel by courier with three copies: the head office (NameNode) tells you which three branches, you hand it to the first branch, and each branch passes a copy on to the next.
Reading a file: ask head office which branches have your parcels, and collect each from the nearest one.
Running a job is like a census: the government (ResourceManager) appoints a supervisor (ApplicationMaster), the supervisor sends counters (map tasks) to every area, area totals go to district officers (reducers), and the final report is filed (written to HDFS).

Intermediate The RPC calls you would see in a trace

Write: create() → addBlock() (per block) → data streamed to DataNodes → complete().
Read: getBlockLocations() (in batches) → data read from DataNodes → checksums verified on the client.
Job: getNewApplication() → upload jar/conf/splits to the staging directory → submitApplication() → RM starts the AM → AM registerApplicationMaster() → repeated allocate() calls (container requests and grants) → startContainers() on NodeManagers → finishApplicationMaster().
The NameNode only ever sees small metadata calls; the ResourceManager only sees resource requests. All heavy data moves between DataNodes, tasks and NodeManagers.

Advanced Where things go wrong, and how each step recovers

A DataNode in the write pipeline dies: the client removes it, finishes the block on the remaining nodes with a new generation stamp, and the NameNode re-replicates later.
A replica is corrupt during a read: checksum mismatch → the client reads another replica and reports the bad one.
The client dies mid-write: its lease expires; the NameNode recovers the last block and closes the file.
A map or reduce task fails: the AM retries it (up to 4 attempts). The AM itself fails: YARN restarts it (MapReduce allows 2 AM attempts by default) and it recovers completed tasks.
The output directory already exists: the job is rejected at submission (FileAlreadyExistsException) to protect old results.

Exam tip: “Explain the anatomy of a file write in HDFS” and “anatomy of a file read” are among the most frequent university questions. Number the steps (create → addBlock → pipeline + acks → complete; open → getBlockLocations → read nearest → checksum) and draw client, NameNode and three DataNodes with arrows. For “job execution flow”, list submit → AM → splits → containers → map → shuffle/sort → reduce → commit.

3 Types and ways to use it

Three journeys compared

	Writing a file	Reading a file	Running a job
Starts with	`hdfs dfs -put` or `FileSystem.create()`	`hdfs dfs -cat` or `FileSystem.open()`	`hadoop jar`, `spark-submit`, a Hive query
Master asked	NameNode: create, addBlock, complete	NameNode: block locations	ResourceManager: new app, submit
Who moves the data	Client → DataNode pipeline	Nearest DataNode → client	Tasks read blocks locally; shuffle between nodes
Parallelism	One block at a time per writer	One block at a time per reader (many readers in parallel)	Many tasks at once
Safety mechanism	Acks from every DataNode, pipeline recovery	Checksums, switch replica	Task retries, AM restart, output committer
Ends with	File closed, visible to readers	Bytes delivered	`part-*` files + `_SUCCESS`

Ways to start each journey

Shell

hdfs dfs -put / -get / -cat and hadoop jar job.jar …. Best for learning and scripts.

Command line

Java API

FileSystem, FSDataOutputStream, FSDataInputStream, and the MapReduce Job class (Modules 6 and 8).

Programs

Engines on top

Hive, Spark and others use the same write/read paths and submit their own applications to YARN.

Higher level

Ingestion tools

Pipelines (NiFi, Kafka Connect, Spark jobs) write files into HDFS through the same client library.

Pipelines

You can watch the job lifecycle live: run a job, then open the ResourceManager UI (port 8088). You'll see the application go from ACCEPTED (waiting for its ApplicationMaster container) to RUNNING to FINISHED, and you can click through to every map and reduce attempt.

4 Step by step

Riya's 300 MB file and her job, every step

Write: create the file
```
hdfs dfs -put canteen_orders.csv /data/
```
The client calls create("/data/canteen_orders.csv") on the NameNode, which checks permissions and that the file doesn't exist, adds it to the namespace as “under construction” and gives the client a lease.
Write: three blocks, three pipelines
```
import math
size_mb, block_mb = 300, 128
print(math.ceil(size_mb / block_mb), "blocks; last block", size_mb - 2 * block_mb, "MB")
```
3 blocks; last block 44 MB
For each block the client calls addBlock; the NameNode picks 3 DataNodes on 2 racks. The data streams as 64 KB packets: client → DN1 → DN2 → DN3, with acks flowing back. The last block uses only 44 MB of disk, not 128.
Write: close

After the last packet is acknowledged, the client calls complete(). The file is now visible and readable.
The DataNodes also tell the NameNode “I received block X” so its block map is up to date.
Job: submit
```
hadoop jar orders.jar ItemTotals /data/canteen_orders.csv /out/item_totals
```
The client checks that /out/item_totals does not exist, computes 3 input splits (one per block), uploads the jar, configuration and split list to a staging directory in HDFS, and submits the application to the ResourceManager.
Job: the ApplicationMaster takes over

The ResourceManager starts the MRAppMaster in a container on some worker. The AM reads the split list and asks for 3 map containers, naming the nodes that hold each block.
The RM grants containers (ideally node-local); NodeManagers launch the map tasks. Each map sorts its output in memory, spills to local disk, and merges the spills into one partitioned file.
Job: shuffle, reduce, commit

Reducers fetch their partition from every map's NodeManager, merge-sort them, run the reduce function and write part-r-00000 into a temporary folder.
When all tasks succeed, the output committer moves the files into /out/item_totals and writes _SUCCESS. The AM unregisters and exits; the job's history goes to the JobHistory Server. The output file is written through the same pipeline as step 2.

5 See it happen

The life of a file: what Riya types, and every message behind it

Riya writes a 300 MB file and reads it back. Left: her terminal. Right: the client, the NameNode and four DataNodes.

Write and read a file 7 steps

What you see

Behind the screen

Try it yourself: follow a file and a job

Upload a file at 1× speed and read the Event log as it happens; then run a job and watch it read your blocks.

6 Real-world use

The life of a job, and where these journeys matter

Real-time scenario

The life of Riya's job, from submit to _SUCCESS

Follow hadoop jar orders.jar ItemTotals through YARN and MapReduce, one stage at a time.

The life of a job 6 steps

Animated example

Where knowing these journeys pays off

1. Reading error messages correctly

Each step has typical errors: permission denied at create, “could only be written to 0 of the 1 minReplication nodes” at addBlock (no healthy DataNodes), FileAlreadyExistsException at job submission. Knowing the step tells you where to look.

Error→Which step failed?→Right fix

2. A job stuck in ACCEPTED

An application that stays ACCEPTED is waiting for a container for its ApplicationMaster: the queue is full or the AM's memory request doesn't fit any node. Understanding the lifecycle turns a mystery into a queue-capacity question.

Stuck job→No AM container yet→Check queue / memory

3. Pipelines that wait for _SUCCESS

Workflow tools (Airflow, Oozie) often start the next step only when the previous job's output folder contains _SUCCESS, so half-written output is never consumed.

Job A→_SUCCESS appears→Job B starts

4. Tuning ingestion speed

Write throughput depends on the pipeline: the slowest of the three DataNodes and the cross-rack hop. Ingestion teams write many files in parallel from several clients to keep all DataNodes busy.

Many writers→Many pipelines→Full cluster bandwidth

Simplified. Error texts vary slightly between versions.

7 Run the code

Simulate the write path in Python

This program does what the HDFS client does when you upload a file: cut it into blocks, ask a (pretend) NameNode for three DataNodes per block following the rack rule, and count the 64 KB packets that will travel through each pipeline.

import math

racks = {"r1": ["dn1", "dn2"], "r2": ["dn3", "dn4"]}

def add_block(n):
    """Pretend NameNode: replica 1 on any node, 2 on the other rack, 3 next to 2."""
    first_rack = "r1" if n % 2 == 0 else "r2"
    other = "r2" if first_rack == "r1" else "r1"
    return [racks[first_rack][n % 2], racks[other][0], racks[other][1]]

size_mb, block_mb, packet_kb = 300, 128, 64
blocks = math.ceil(size_mb / block_mb)
for n in range(blocks):
    mb = min(block_mb, size_mb - n * block_mb)
    pipeline = add_block(n)
    packets = math.ceil(mb * 1024 / packet_kb)
    print(f"block {n + 1}: {mb:>3} MB -> {' -> '.join(pipeline)} ({packets} packets)")

block 1: 128 MB -> dn1 -> dn3 -> dn4 (2048 packets) block 2: 128 MB -> dn4 -> dn1 -> dn2 (2048 packets) block 3: 44 MB -> dn1 -> dn3 -> dn4 (704 packets)

def add_block(n):Plays the NameNode's addBlock call: returns the pipeline of DataNodes for the next block.
return [racks[first_rack][n % 2], racks[other][0], racks[other][1]]Default placement: one replica on one rack, the next two together on another rack.
packets = math.ceil(mb * 1024 / packet_kb)A 128 MB block travels as 2,048 packets of 64 KB, each acknowledged by every DataNode in the pipeline.

8 Line by line

Splitting a file into blocks, one line at a time

Watch the client cut a 300 MB file into blocks, asking for a new pipeline each time. Notice how the last block is smaller.

Worked examples: real problems, solved and traced in memory

Three calculations that come straight from the write path and the job lifecycle: how much traffic a write creates, how long a write takes, and a full job timeline. Each runs live and is traced in memory.

Example 1BeginnerHDFS · write traffic

How much data crosses the network when you upload 1 GB?

Problem statement

A client outside the cluster uploads a 1 GB file with replication 3. Every replica is sent once over the network (client → DN1, DN1 → DN2, DN2 → DN3). The second and third replicas are on another rack from the first. How many GB travel in total, and how many cross between racks?

Input

1 GB · replication 3 · replica 1 on rack A, replicas 2–3 on rack B

Expected output

Total sent: 3 GB
Crossing racks or entering the cluster: 2 GB
  client -> DN1 into rack A
  DN1 -> DN2   rack A -> rack B
  DN2 -> DN3   inside rack B

Approach

Total sent = size × replication (each replica crosses one link).
Cross-rack: client → DN1 (the client is outside), DN1 → DN2 (rack A to B).
DN2 → DN3 stays inside rack B.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 2IntermediateHDFS · write time

How long does a pipeline write take?

Problem statement

A pipeline streams packets: DN1 forwards each packet to DN2 while receiving the next one, so the three copies are written at the same time, limited by the slowest link. Estimate the time to upload 10 GB when the slowest link is 200 MB/s, versus writing the three copies one after another.

Input

10 GB · slowest link 200 MB/s · 3 replicas

Expected output

Pipelined : 51 s
Sequential: 154 s

Approach

Pipelined: time ≈ size ÷ slowest link speed.
Sequential (one copy after another): 3 × that.
Print both.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 3AdvancedMapReduce · job timeline

Estimate a whole job, stage by stage

Problem statement

A job reads 40 GB (128 MB splits). The cluster can run 60 map tasks at once. Each map takes 40 s. Starting the ApplicationMaster takes 15 s; after the maps, the shuffle and reduce take 90 s; committing takes 5 s. Estimate the total time.

Input

40 GB · 128 MB splits · 60 slots · 40 s/map · AM 15 s · shuffle+reduce 90 s · commit 5 s

Expected output

start AM         :   15 s
maps             :  240 s
shuffle + reduce :   90 s
commit           :    5 s
Total: 350 s for 320 maps in 6 waves

Approach

Splits = 40 GB ÷ 128 MB.
Map waves = splits ÷ 60, rounded up; map time = waves × 40 s.
Total = AM + map time + shuffle/reduce + commit.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

9 Quiz and revision

Check yourself

Quick revision

Lesson 3.2 in one screen

Write: create (lease) → per block addBlock → 64 KB packets through DN1 → DN2 → DN3, acks back → complete.
New pipeline for each block; last block only as big as needed.
Read: getBlockLocations (sorted by distance) → read nearest DataNode directly → verify CRC → next block.
Masters never carry data. Pipeline failure → continue with remaining DataNodes; NameNode re-replicates later.
Job: client checks output dir, computes splits, uploads jar/conf/splits to staging, submits to RM.
RM starts MRAppMaster (ACCEPTED → RUNNING); AM requests containers near each split.
Map → sort buffer (100 MB) → spill to local disk → partitions. Reducers fetch (shuffle) after 5% of maps.
Reduce → write to HDFS → output committer → part-r-* + _SUCCESS; AM unregisters; history to JobHistory.

Interview and viva questions

Explain the anatomy of a file write in HDFS.

The client calls create() on the DistributedFileSystem, which makes an RPC to the NameNode; the NameNode checks permissions and that the file does not exist, records it and grants a lease. The client's output stream splits data into packets in a data queue. A DataStreamer asks the NameNode for a new block (addBlock), which returns a block ID and a pipeline of DataNodes chosen by the placement policy. Packets are sent to the first DataNode, which stores and forwards them to the second, and so on; acknowledgements return along the pipeline and packets are removed from the ack queue. This repeats for each block. On close(), remaining packets are flushed and acknowledged, and the client calls complete() on the NameNode.

Explain the anatomy of a file read in HDFS.

The client calls open(); the DistributedFileSystem asks the NameNode for the locations of the first blocks, sorted by network distance from the client. The input stream connects to the closest DataNode for the first block and streams the data, verifying checksums. At the end of the block it connects to the best DataNode for the next block, fetching more locations from the NameNode as needed. If a DataNode fails or a checksum does not match, it tries the next replica and reports the problem.

What happens if a DataNode fails during a write?

The pipeline is closed, packets in the ack queue are put back at the front of the data queue, the failed DataNode is removed, the current block on the good DataNodes gets a new generation stamp (so the partial replica on the failed node will be discarded later), and writing continues with the remaining DataNodes. The NameNode notices the block is under-replicated and creates another replica afterwards.

Describe the steps in running a MapReduce job on YARN.

The client checks the output path, computes input splits, copies the job jar, configuration and split information to a staging directory in HDFS, and submits the application to the ResourceManager. The RM allocates a container and a NodeManager launches the MRAppMaster. The AM requests containers for map tasks (with locality preferences) and later reduce tasks; NodeManagers launch them. Maps write partitioned, sorted output to local disk; reducers shuffle and merge it, run the reduce function and write output to HDFS through the output committer. The AM then cleans up, writes _SUCCESS, unregisters from the RM, and job history is available from the JobHistory Server.

Why are map outputs written to local disk and not to HDFS?

They are intermediate and short-lived; replicating them to HDFS would triple the write cost and network traffic for data that is deleted once the job finishes. The trade-off is that if a node is lost before reducers have fetched its map outputs, those map tasks must be re-run.

Practice problems

Practise the two journeys with numbers and small simulations, from counting blocks to building a job timeline.

P1Easy

Blocks and last block

A 1,000 MB file is written with 128 MB blocks. How many blocks, and how big is the last one?

Input

1000 MB · 128 MB

Expected output

8 blocks; last block 104 MB

Show solution and explanation

import math
n = math.ceil(1000 / 128)
print(n, "blocks; last block", 1000 - (n - 1) * 128, "MB")

8 blocks; last block 104 MB

How it works: 8 blocks; the last holds 104 MB. HDFS does not pad it to 128 MB on disk.

P2Easy

Packets per block

How many 64 KB packets make up one full 128 MB block?

Input

128 MB · 64 KB

Expected output

2048 packets

Show solution and explanation

print(128 * 1024 // 64, "packets")

2048 packets

How it works: 2,048 packets, each acknowledged by every DataNode in the pipeline.

P3Easy

Replicas written

A 2 GB file with replication 3: how many block replicas are written in total (128 MB blocks)?

Input

2 GB · 128 MB · ×3

Expected output

48 block replicas

Show solution and explanation

print(2 * 1024 // 128 * 3, "block replicas")

48 block replicas

How it works: 16 blocks × 3 = 48 replicas, each one a file (plus a checksum file) on some DataNode's disk.

P4Medium

Nearest replica for every block

For a client on dn2, print the best replica of each block using distance 0/2/4.

Input

racks r1: dn1, dn2 · r2: dn3, dn4 · B1 [dn3, dn4, dn1] · B2 [dn4, dn3, dn2]

Expected output

B1 -> dn1
B2 -> dn2

Show solution and explanation

rack = {"dn1": "r1", "dn2": "r1", "dn3": "r2", "dn4": "r2"}
d = lambda a, b: 0 if a == b else 2 if rack[a] == rack[b] else 4
for b, reps in {"B1": ["dn3", "dn4", "dn1"], "B2": ["dn4", "dn3", "dn2"]}.items():
    print(b, "->", min(reps, key=lambda r: d("dn2", r)))

B1 -> dn1 B2 -> dn2

How it works: B1 is read from dn1 (same rack), B2 from dn2 itself (same node, distance 0).

P5Medium

Map waves for a job

A job reads 25 GB in 128 MB splits on a cluster with 50 map slots. How many map tasks and waves?

Input

25 GB · 128 MB · 50 slots

Expected output

200 maps in 4 waves

Show solution and explanation

import math
maps = math.ceil(25 * 1024 / 128)
print(maps, "maps in", math.ceil(maps / 50), "waves")

200 maps in 4 waves

How it works: 200 maps in 4 waves. Doubling the slots would halve the map phase, but not the AM start-up or the reduce phase.

P6Medium

Output files of a job

Print the names of the output files of a job with 4 reducers (Hadoop naming), plus the success marker.

Input

4 reducers

Expected output

['part-r-00000', 'part-r-00001', 'part-r-00002', 'part-r-00003', '_SUCCESS']

Show solution and explanation

files = [f"part-r-{i:05d}" for i in range(4)] + ["_SUCCESS"]
print(files)

['part-r-00000', 'part-r-00001', 'part-r-00002', 'part-r-00003', '_SUCCESS']

How it works: One file per reducer: part-r-00000… Map-only jobs produce part-m-* files instead.

P7Business problem

Bulk load: client outside vs on the cluster

A team uploads 5 TB per night. From an edge node outside the racks, every replica crosses the network. From a DataNode, replica 1 is written locally. How many TB of network traffic does each approach create?

Input

5 TB · replication 3

Expected output

From an edge node : 15 TB over the network
From a DataNode   : 10 TB over the network

Show solution and explanation

tb = 5
outside = tb * 3
on_node = tb * 2
print("From an edge node :", outside, "TB over the network")
print("From a DataNode   :", on_node, "TB over the network")

From an edge node : 15 TB over the network From a DataNode : 10 TB over the network

How it works: 15 TB vs 10 TB a night. Loading from cluster nodes (or with distributed loaders) saves a third of the network traffic, at the cost of the first node filling faster unless writes are spread out.

P8Business problem

SLA check for a nightly job

The nightly job must finish within 30 minutes. Using the stage times below, does it meet the SLA, and how much slack is left?

Input

queue wait 4 min · AM start 0.5 min · maps 14 min · shuffle+reduce 8 min · commit 0.5 min

Expected output

Total 27.0 min; SLA met: True; slack 3.0 min

Show solution and explanation

stages = {"queue wait": 4, "AM start": 0.5, "maps": 14, "shuffle + reduce": 8, "commit": 0.5}
total = sum(stages.values())
print(f"Total {total} min; SLA met: {total <= 30}; slack {30 - total} min")

Total 27.0 min; SLA met: True; slack 3.0 min

How it works: 27 minutes, 3 minutes of slack. The queue wait (time in ACCEPTED) is often forgotten in SLA estimates.

Real-world scenario questions (asked in company interviews)

Interviewers love asking you to narrate a write, a read or a job, and then break one step. Practise the narration and the failure cases.

Any company · conceptsFresher

“A DataNode dies while I'm writing a file. What happens?”

You have just explained the write pipeline. The interviewer asks what happens if DN2 crashes halfway through block 3.

The interviewer is checking: do you know pipeline recovery, not just the happy path?

Show a strong answer

The client notices missing acknowledgements and closes the pipeline.
Unacknowledged packets are put back in the queue so nothing is lost.
DN2 is removed; the block on DN1 and DN3 gets a new generation stamp so DN2's partial copy will later be recognised as stale and deleted.
Writing continues to DN1 and DN3. The write succeeds as long as at least the minimum replication (1 by default) is met.
The NameNode later sees block 3 has 2 replicas and schedules a third.

Avoid: “The write fails and the user must retry.”

Operations · troubleshootingFresher

“could only be written to 0 of the 1 minReplication nodes”

A student's freshly installed cluster accepts hdfs dfs -mkdir but every -put fails with this message.

The interviewer is checking: can you map an error to the write step where it happens?

Show a strong answer

mkdir only talks to the NameNode, so metadata works. The error comes at addBlock: the NameNode found no DataNode it could place a block on.
Check jps: is a DataNode running? Check hdfs dfsadmin -report: live DataNodes and free space.
Common causes: the DataNode failed to start (often a cluster-ID mismatch after re-formatting the NameNode), the disk is full, or the DataNode can't reach the NameNode (hostname/firewall).

Avoid: reformatting the NameNode again; that wipes metadata and often causes the ID mismatch in the first place.

Data platform · operations1–3 years

Jobs stay in ACCEPTED for an hour

Every morning, newly submitted jobs sit in ACCEPTED state for a long time before running, although the cluster “isn't that busy”.

The interviewer is checking: do you know what ACCEPTED means in the job lifecycle?

Show a strong answer

ACCEPTED means the RM has the application but has not yet started its ApplicationMaster container.
Check the queue: it may be at its capacity or at its limit for AMs (maximum-am-resource-percent), so new AMs wait even if some memory is free.
Check whether the AM container size fits on any node, and whether many small jobs are each holding an AM.
Fixes: adjust queue capacities or AM limits, schedule big jobs at different times, combine many tiny jobs.

Avoid: killing and resubmitting jobs; they go back to the end of the queue.

Data engineering · pipelines1–3 years

The downstream job read half-written output

A reporting job starts every day at 6 am and reads /out/daily_sales. Some days its numbers are too low. The upstream MapReduce job sometimes finishes after 6 am.

The interviewer is checking: do you understand output commit and how pipelines should wait for it?

Show a strong answer

Successful task output is only moved into the final folder at commit, but a downstream job can still start before or during it, or read an old partial folder.
Make the reporting job depend on the upstream job's completion: in Airflow or Oozie, wait for the _SUCCESS file (or the job status) before starting.
Write each day to its own dated folder, so readers never see a folder that is being rewritten.

Avoid: “Just start the report later.” Time-based guesses break on slow days.

The life of a file and a job, end to end

Two commands, two hundred messages

The write path, the read path and the job lifecycle

Words you'll meet in this lesson

Numbers in the life of a file and a job (defaults)

The two lives at three levels

Beginner A courier and a census

Intermediate The RPC calls you would see in a trace

Advanced Where things go wrong, and how each step recovers

Three journeys compared

Ways to start each journey

Shell

Java API

Engines on top

Ingestion tools

Riya's 300 MB file and her job, every step

Write: create the file

Write: three blocks, three pipelines

Write: close

Job: submit

Job: the ApplicationMaster takes over

Job: shuffle, reduce, commit

The life of a file: what Riya types, and every message behind it

Try it yourself: follow a file and a job

The life of a job, and where these journeys matter

The life of Riya's job, from submit to _SUCCESS

Where knowing these journeys pays off

1. Reading error messages correctly

2. A job stuck in ACCEPTED

3. Pipelines that wait for _SUCCESS

4. Tuning ingestion speed

Simulate the write path in Python

Splitting a file into blocks, one line at a time

Worked examples: real problems, solved and traced in memory

How much data crosses the network when you upload 1 GB?

How long does a pipeline write take?

Estimate a whole job, stage by stage

Check yourself

Quick revision

Lesson 3.2 in one screen

Interview and viva questions

Practice problems

Blocks and last block

Packets per block

Replicas written

Nearest replica for every block

Map waves for a job

Output files of a job

Bulk load: client outside vs on the cluster

SLA check for a nightly job

Real-world scenario questions (asked in company interviews)

“A DataNode dies while I'm writing a file. What happens?”

“could only be written to 0 of the 1 minReplication nodes”

Jobs stay in ACCEPTED for an hour

The downstream job read half-written output