Blocks (and how they differ from splits)

1 The story

A cut through the middle of an order

Kabir stored the canteen file with 1 MB blocks and opened the end of the first block on a DataNode. The last line reads 71484,poha,2. The first line of the next block is just 5. Order 71484 cost 25 rupees, and HDFS has sliced it in half. “Our MapReduce job will read a poha for 2 rupees!” he says. But the job's answer was correct. Why?

Arjun Sir

Two different ideas are hiding in your question. A block is how HDFS stores a file: fixed-size pieces of bytes, cut wherever the byte count says, even in the middle of a word. HDFS doesn't know or care what a “line” is.

Riya

And the job?

Arjun Sir

The job works with input splits: one split per map task, usually one per block. But a split is logical. Its reader finishes the last record even if that means reading a few bytes from the next block, and the next reader skips the broken first line. Every order is read exactly once, whole.

Kabir

So the 2-rupee poha never exists.

Arjun Sir

Never. Today we'll prove it on the real cluster: count the records each map task read, and check that they add up to the whole file.

2 Definition

HDFS blocks (and input splits)

Formal definition (write this in exams)

An HDFS block is the unit in which a file is stored and replicated: a file is divided into blocks of a fixed size (dfs.blocksize, default 128 MB in Hadoop 2 and 3), chosen per file by the writing client; every block except the last is full, and the last block occupies only as much space as its data. Each block is replicated independently and placed on DataNodes by the NameNode. An input split is a logical division of the input for MapReduce: one split is processed by one map task. FileInputFormat creates splits that by default match block boundaries, and the RecordReader (e.g. LineRecordReader) ensures records crossing a boundary are read whole by exactly one split.

In simple words

Blocks are how HDFS chops a file for storage: by size, not by meaning. Splits are how a job divides the work: about one per block, but always in whole records.

Words you'll meet in this lesson

dfs.blocksizeBlock size for new files, set by the writer. Default 134,217,728 bytes (128 MB).

Last blockUsually smaller; uses only the disk space of its data (no padding).

Input splitThe piece of input one map task processes: path, start offset, length, preferred hosts.

RecordReaderTurns a split's bytes into records; finishes boundary records, skips partial first lines.

Block IDUnique number, e.g. blk_1073741826, plus a generation stamp _1002.

Seek vs transferLarge blocks keep disk seek time small compared with reading time.

Block facts (Hadoop 3.5.0 defaults)

Default block size	134,217,728 bytes = 128 MB (`dfs.blocksize`; Hadoop 1 used 64 MB)
Minimum block size	1,048,576 bytes = 1 MB (`dfs.namenode.fs-limits.min-block-size`)
Who chooses	The client writing the file; can differ per file (`-D dfs.blocksize=…`)
Last block	Stores only real data: our 4,692,009-byte file uses one 4,692,009-byte block file, not 128 MB
Splits by default	One per block (`mapreduce.input.fileinputformat.split.minsize`/`maxsize` can change this)

Blocks and splits at three levels

Pick your level. New to this? Start at Beginner and climb.

Beginner A long train and its coaches

A file is a long train. HDFS cuts it into coaches of equal length (blocks), even if a passenger's seat gets split by the cut.
Each coach is parked in three different stations (replicas).
Ticket checkers (map tasks) each take a coach. A checker who finds a passenger half in the next coach walks over and checks the whole passenger; the next checker ignores that half.

Intermediate Why big blocks?

A disk spends a few milliseconds seeking before it can read. With large blocks, reading takes much longer than seeking, so disks work at full speed.
Fewer blocks mean less NameNode memory and fewer map tasks to schedule.
Too big and you get too few map tasks (little parallelism) for medium files.
So 128 MB is a balance; 256 MB is common for very large tables, smaller sizes are rarely useful.

Advanced Splits in detail

Split size = max(minSize, min(maxSize, blockSize)); with defaults it equals the block size.
A small last piece (up to 10% larger than a split) is merged into the previous split (SPLIT_SLOP = 1.1).
Non-splittable inputs (e.g. a gzip file) become one split however many blocks they have (Lesson 10.3).
Each split carries the hosts of its block, so YARN can run the map task where the data is (locality).
Reading the end of a boundary record can be a small remote read from another DataNode.

Exam tip: “Differentiate HDFS block and input split”: block = physical division for storage, fixed size (128 MB), may cut a record, replicated; split = logical division for processing, one per map task, respects record boundaries, usually equals one block. Draw a record crossing a block boundary and show which split reads it.

3 Types and ways to use it

Block vs split, and choosing a block size

	Block	Input split
Kind	Physical: bytes on disks	Logical: a range of the input
Belongs to	HDFS	MapReduce / Spark / Hive input formats
Size set by	`dfs.blocksize` when writing	Block size and split min/max settings when reading
Respects records?	No, cuts at a byte count	Yes, via the RecordReader
Count for our file	5 blocks (1 MB) or 1 block (128 MB)	5 splits or 1 split (real job output)
Replicated?	Yes, 3 copies	No; it just points to data

Common block sizes

64 MB

Hadoop 1 default. Still seen on old clusters; more blocks, more NameNode memory.

Legacy

128 MB

Hadoop 2/3 default. Good for most data.

Default

256 MB

Very large tables scanned in full; fewer, longer map tasks.

Big tables

1 MB

The minimum allowed. Only for demos like this lesson: terrible for real data.

Demo only

Block size is per file and set when the file is written. To change it for existing data, rewrite the data (for example with distcp or by re-running the job that produced it with a different dfs.blocksize).

4 Step by step

See blocks and splits on the real cluster

Uses the Module 5 lab cluster and the two copies of canteen_orders.csv from Lesson 5.1.

Compare block sizes of the two copies
```
docker compose exec namenode hdfs dfs -stat "%o %b %n" data/canteen_orders.csv data/canteen_1mb.csv
docker compose exec namenode hdfs fsck /user/hadoop/data/canteen_1mb.csv -files -blocks | grep blk_
```
134217728 4692009 canteen_orders.csv 1048576 4692009 canteen_1mb.csv 0. BP-1507681208-…:blk_1073741826_1002 len=1048576 Live_repl=3 1. BP-1507681208-…:blk_1073741827_1003 len=1048576 Live_repl=3 2. BP-1507681208-…:blk_1073741828_1004 len=1048576 Live_repl=3 3. BP-1507681208-…:blk_1073741829_1005 len=1048576 Live_repl=3 4. BP-1507681208-…:blk_1073741830_1006 len=497705 Live_repl=3
4 full blocks of 1,048,576 bytes and a last block of 497,705 bytes: 4 × 1,048,576 + 497,705 = 4,692,009. The 128 MB copy has a single block.
Check the real files on a DataNode
```
find /tmp/hadoop-hadoop/dfs/data -name "blk_*" | grep -v meta | sort | xargs ls -l
```
-rw-r--r-- 1 hadoop users 4692009 … blk_1073741825 -rw-r--r-- 1 hadoop users 1048576 … blk_1073741826 -rw-r--r-- 1 hadoop users 1048576 … blk_1073741827 -rw-r--r-- 1 hadoop users 1048576 … blk_1073741828 -rw-r--r-- 1 hadoop users 1048576 … blk_1073741829 -rw-r--r-- 1 hadoop users 497705 … blk_1073741830
blk_1073741825 belongs to a file with 128 MB blocks, yet the file on disk is 4,692,009 bytes. A block is a maximum size, not a reservation.
Find the record cut by the boundary
```
tail -c 60 blk_1073741826
head -c 60 blk_1073741827
```
ha,25 71481,poha,25 71482,poha,25 71483,dosa,45 71484,poha,2 5 71485,idli,30 71486,idli,30 71487,dosa,45 71488,vada pav,2
The 1,048,576th byte falls between the “2” and the “5” of order 71484. HDFS stores bytes; it has no idea this is a CSV.
Run a job and count splits
```
docker compose exec namenode hadoop jar \
  /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.5.0.jar wordcount \
  -D mapreduce.framework.name=local data/canteen_1mb.csv out/wc1
```
INFO FileInputFormat:303 - Total input files to process : 1 INFO JobSubmitter:203 - number of splits:5 Map input records=70485 Map input records=67321 Map input records=65498 Map input records=65574 Map input records=31123 INFO Job:1765 - Job job_local479046282_0001 completed successfully Map input records=300001
5 blocks → 5 splits → 5 map tasks (run locally inside the client here, reading from HDFS). Their records add up to 300,001: the header plus 3 lakh orders, each read exactly once.
Same job on the 128 MB copy
```
docker compose exec namenode hadoop jar \
  /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.5.0.jar wordcount \
  -D mapreduce.framework.name=local data/canteen_orders.csv out/wc128
```
INFO JobSubmitter:203 - number of splits:1 Map input records=300001
One block, one split, one map task reading all 300,001 lines. Same data, different parallelism, decided by how the file was written.

Don't shrink the block size to “get more map tasks” for big data. Smaller blocks multiply NameNode objects and task overhead. Tune the split size of the job instead, or use a format and engine that parallelise well.

5 See it happen

How map 0 and map 1 share order 71484

The real bytes around the first block boundary of canteen_1mb.csv, and what each map task's RecordReader does with them.

Blocks cut bytes, splits keep records 6 steps

What you see

Behind the screen

Try it yourself: block size in the playground

Change Block before each upload and compare how many blocks the same file becomes.

6 Real-world use

Blocks and splits at work

1. Bigger blocks for huge tables

Teams writing multi-terabyte fact tables often use 256 MB blocks (and matching Parquet row groups), halving block counts and NameNode load while keeping enough parallelism.

Huge table→256 MB blocks→Fewer objects

2. Tuning splits, not blocks

When a CPU-heavy job needs more parallel tasks, engineers lower the job's maximum split size instead of rewriting data with smaller blocks.

Slow job→Smaller splits→More map tasks

3. Unsplittable files

A 10 GB gzip log is 80 blocks but one split, so one map task reads it all. Ingestion teams store logs in splittable formats (bzip2, or Parquet/ORC with Snappy) to avoid this.

One .gz→One mapper→Use splittable formats

4. Same idea in Spark

When Spark reads files from HDFS, its initial partitions come from the same Hadoop input splits, so block size also affects how many tasks a Spark stage starts with.

HDFS blocks→Splits→Spark partitions

Simplified; organisations differ in tools and policies.

7 Run the code

Cut a file into blocks, then read it in splits

This program stores a small CSV in 32-byte “blocks” (as HDFS would with a tiny block size), shows how lines get cut, then reads each split with Hadoop's two rules: skip the partial first line unless you are split 0 and finish the record that starts inside your split.

data = "1001,samosa,18\n1002,vada pav,20\n1003,poha,25\n1004,dosa,45\n1005,chai,15\n"
BLOCK = 32
blocks = [data[i:i + BLOCK] for i in range(0, len(data), BLOCK)]
for i, b in enumerate(blocks):
    print(f"block {i}: {b!r}")

for i in range(len(blocks)):
    start, end = i * BLOCK, min((i + 1) * BLOCK, len(data))
    pos = start if i == 0 else data.index("\n", start - 1) + 1
    records = []
    while pos < end:
        nl = data.index("\n", pos)
        records.append(data[pos:nl])
        pos = nl + 1
    print(f"split {i}: {records}")

block 0: '1001,samosa,18\n1002,vada pav,20\n' block 1: '1003,poha,25\n1004,dosa,45\n1005,c' block 2: 'hai,15\n' split 0: ['1001,samosa,18', '1002,vada pav,20'] split 1: ['1003,poha,25', '1004,dosa,45', '1005,chai,15'] split 2: []

blocks = [data[i:i + BLOCK] for i in range(0, len(data), BLOCK)]Blocks are cut every 32 characters, wherever that falls. Look for cut-off orders in the output.
pos = start if i == 0 else data.index("\n", start - 1) + 1Rule 1: every split except the first skips to just after the first newline at or after its start (looking one byte back, in case the split starts exactly at a line).
while pos < end:Rule 2: a record that starts before the end of the split belongs to it, even if it ends in the next block.

Run it on your PC: predict Hadoop's split counters exactly

This script applies HDFS block cutting and Hadoop's LineRecordReader rules to the real canteen_orders.csv (from make_canteen.py, Lesson 4.3) with 1 MB blocks. Save it as blocks_and_splits.py next to the CSV and run it. Then compare with the job counters in Part 4.

Blocks, splits and the cut record, without Hadoop

# blocks_and_splits.py: cut a file into HDFS-style blocks, then count records per split
# the way Hadoop's LineRecordReader does (no Hadoop needed).

FILE = "canteen_orders.csv"
BLOCK = 1024 * 1024                       # same as -D dfs.blocksize=1048576
data = open(FILE, "rb").read()
size = len(data)
starts = list(range(0, size, BLOCK))
print(f"{FILE}: {size:,} bytes -> {len(starts)} blocks of up to {BLOCK:,} bytes")

for i, start in enumerate(starts):
    end = min(start + BLOCK, size)
    pos = start
    if start != 0:                        # not the first split: skip the partial first line
        pos = data.index(b"\n", start - 1) + 1
    records = 0
    while pos < end:                      # a record that STARTS inside the split belongs to it
        nl = data.find(b"\n", pos)
        pos = size if nl == -1 else nl + 1
        records += 1
    print(f"split {i}: bytes {start:>7}-{end - 1:>7}  records {records:>6}")

cut = BLOCK                               # show the line broken by the first block boundary
line_start = data.rindex(b"\n", 0, cut) + 1
line_end = data.index(b"\n", cut)
print("record cut by block 0/1:", data[line_start:cut], "+", data[cut:line_end])

# In a terminal:
#   python3 blocks_and_splits.py

canteen_orders.csv: 4,692,009 bytes -> 5 blocks of up to 1,048,576 bytes split 0: bytes 0-1048575 records 70485 split 1: bytes 1048576-2097151 records 67321 split 2: bytes 2097152-3145727 records 65498 split 3: bytes 3145728-4194303 records 65574 split 4: bytes 4194304-4692008 records 31123 record cut by block 0/1: b'71484,poha,2' + b'5'

Every count matches the five “Map input records” counters of the real Hadoop job, and the cut record is the same one found in the block files on the DataNode.

8 Line by line

Reading split 1, one line at a time

Follow the reader of the second split. Watch how it skips the partial line and where it stops.

Worked examples: real problems, solved and traced in memory

Three block calculations that come up in real projects: how many blocks and map tasks a data set produces, how a block-size change affects the NameNode, and how splits are formed with the 10% slop rule. Each runs live and is traced in memory.

Example 1BeginnerPlanning · tasks

How many map tasks will this job start?

Problem statement

A job reads three files: 1 GB, 300 MB and 50 MB, all with 128 MB blocks. How many blocks, and how many map tasks (one per split, splits = blocks by default)?

Input

1024 MB · 300 MB · 50 MB · 128 MB blocks

Expected output

sales.csv     1024 MB -> 8 block(s)
returns.csv    300 MB -> 3 block(s)
stores.csv      50 MB -> 1 block(s)
map tasks: 12

Approach

Blocks per file = ceil(size / 128).
Every file starts its own splits; small files still need one.
Sum them.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 2IntermediateNameNode · objects

What does doubling the block size save?

Problem statement

A 600 TB data warehouse is stored with 128 MB blocks. How many blocks is that, and how much NameNode heap would 256 MB blocks save, at 150 bytes per block object?

Input

600 TB · 128 vs 256 MB · 150 B per block

Expected output

128 MB: 4,915,200 blocks   256 MB: 2,457,600 blocks
heap saved ~0.34 GB

Approach

Blocks = data / block size.
Heap for blocks = blocks × 150 bytes.
Difference between the two sizes.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 3AdvancedMapReduce · splits

Form splits with the 10% slop rule

Problem statement

FileInputFormat keeps cutting full splits while the remaining bytes are more than 1.1 × the split size; whatever remains becomes the last split. Compute the splits for a 270 MB file and a 260 MB file with 128 MB split size.

Input

270 MB and 260 MB · split 128 MB · slop 1.1

Expected output

270 MB -> [128, 128, 14]
260 MB -> [128, 132]

Approach

While remaining / split > 1.1: cut one full split.
Then the remainder (if any) is one split.
Compare with a naive ceil(size / 128).

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

9 Quiz and revision

Check yourself

Quick revision

Lesson 5.2 in one screen

Block = physical unit of storage and replication; default 128 MB (min 1 MB).
Cut by byte count: records can be split across blocks.
Last block stores only real data (4,692,009-byte file → 4,692,009-byte block file).
Block size is per file, chosen by the writer: -D dfs.blocksize=….
Split = logical unit of work, one per map task, usually one per block.
RecordReader: skip partial first line (except split 0); finish the record that starts in your split.
Real proof: 5 splits read 70485 + 67321 + 65498 + 65574 + 31123 = 300001 records.
Bigger blocks: less seek overhead, less NameNode memory, fewer tasks. Unsplittable formats (gzip) = 1 split.

Interview and viva questions

What is a block in HDFS?

The unit in which HDFS stores and replicates a file. Files are divided into blocks of dfs.blocksize bytes (128 MB by default); each block is stored on several DataNodes. Only the last block of a file may be smaller, and it uses only the space it needs.

Why is the HDFS block size so large?

To make reading efficient on disks (the time to transfer a block is much larger than the time to seek to it) and to keep the number of blocks, and so the NameNode's memory and the number of tasks, small.

What is the difference between a block and an input split?

A block is a physical division of a file for storage. An input split is a logical division of the input for processing by one map task. Splits usually match blocks, but the RecordReader makes sure records crossing block boundaries are read whole by exactly one split.

Does a small file occupy a full 128 MB block?

No. A 1 MB file uses about 1 MB of disk per replica. It does, however, cost one file object and one block object in NameNode memory, which is why many small files are a problem.

Can I change the block size of an existing file?

Not in place. Block size is fixed when the file is written; to change it, rewrite the file (for example copy it with -D dfs.blocksize=… or with distcp).

How many map tasks will my job have?

By default one per input split, and splits follow blocks: a 1 GB file with 128 MB blocks gives 8 map tasks. Split size settings, the 10% slop rule, unsplittable compression and combined input formats can change this.

Practice problems

Block and split exercises, several using the exact numbers from the real cluster.

P1Easy

Blocks for a file

How many 128 MB blocks does a 1,000 MB file need, and how big is the last block?

Input

1000 MB · 128 MB

Expected output

8 blocks, last block 104 MB

Show solution and explanation

import math
n = math.ceil(1000 / 128)
print(n, "blocks, last block", 1000 - (n - 1) * 128, "MB")

8 blocks, last block 104 MB

How it works: 8 blocks: 7 full ones and a last block of 104 MB.

P2Easy

Check the real numbers

Verify that 4 blocks of 1,048,576 bytes plus 497,705 bytes equal the file size 4,692,009.

Input

4 × 1,048,576 + 497,705

Expected output

True

Show solution and explanation

print(4 * 1048576 + 497705 == 4692009)

True

How it works: True: the fsck lengths add up to the file size exactly.

P3Easy

Add up the counters

Add the five real Map input records counters and compare with the file's line count (300,001).

Input

70485 · 67321 · 65498 · 65574 · 31123

Expected output

Show solution and explanation

print(sum([70485, 67321, 65498, 65574, 31123]))

300001

How it works: 300,001: every line read exactly once, despite five block boundaries.

P4Medium

Where is byte N?

For 1 MB blocks, which block contains byte offsets 0, 1,048,575, 1,048,576 and 4,692,008, and at what offset inside the block?

Input

offsets 0 · 1048575 · 1048576 · 4692008

Expected output

0 -> block 0 offset 0
1048575 -> block 0 offset 1048575
1048576 -> block 1 offset 0
4692008 -> block 4 offset 497704

Show solution and explanation

B = 1048576
for off in [0, 1048575, 1048576, 4692008]:
    print(off, "-> block", off // B, "offset", off % B)

0 -> block 0 offset 0 1048575 -> block 0 offset 1048575 1048576 -> block 1 offset 0 4692008 -> block 4 offset 497704

How it works: Block = offset // block size, position = offset % block size. Byte 1,048,576 is the first byte of block 1, the “5” of order 71484.

P5Medium

Splits for many files

A folder has 20 files of 200 MB and 1,000 files of 2 MB. How many map tasks with one split per block (128 MB)?

Input

20 × 200 MB · 1000 × 2 MB

Expected output

Show solution and explanation

import math
print(20 * math.ceil(200 / 128) + 1000 * math.ceil(2 / 128))

1040

How it works: 40 tasks for the big files and 1,000 for the tiny ones: 1,040 tasks, most of them doing almost nothing.

P6Medium

Seek share

With 8 ms seek and 200 MB/s transfer, what block size keeps seeking under 1% of the time?

Input

8 ms · 200 MB/s · 1%

Expected output

about 256 MB

Show solution and explanation

seek, rate = 0.008, 200
mb = 1
while seek / (seek + mb / rate) > 0.01:
    mb *= 2
print("about", mb, "MB")

about 256 MB

How it works: About 256 MB (the first power of two that brings seeking under 1%); with 128 MB it is about 1.2%.

P7Business problem

Mapper hours wasted

A nightly job reads 5,000 small files, one map task each. Each task has 3 s of start-up overhead. Merging them into 40 files of 128 MB reduces it to 40 tasks. How many task-hours of overhead are saved per month (30 runs)?

Input

5000 vs 40 tasks · 3 s · 30 runs

Expected output

124 task-hours per month

Show solution and explanation

saved_s = (5000 - 40) * 3 * 30
print(f"{saved_s / 3600:.0f} task-hours per month")

124 task-hours per month

How it works: About 124 task-hours a month of pure start-up overhead, before counting NameNode load.

P8Advanced

Record reader for a boundary

Implement the two RecordReader rules for a list of split boundaries and return records per split for the given text.

Input

"aa\nbbbb\ncc\ndddddd\ne\n" · splits at 0, 5, 12

Expected output

split 0 [0,5): ['aa', 'bbbb']
split 1 [5,12): ['cc', 'dddddd']
split 2 [12,20): ['e']

Show solution and explanation

data = "aa\nbbbb\ncc\ndddddd\ne\n"
bounds = [0, 5, 12, len(data)]
for i in range(len(bounds) - 1):
    start, end = bounds[i], bounds[i + 1]
    pos = start if start == 0 else data.index("\n", start - 1) + 1
    recs = []
    while pos < end:
        nl = data.index("\n", pos)
        recs.append(data[pos:nl]); pos = nl + 1
    print(f"split {i} [{start},{end}):", recs)

split 0 [0,5): ['aa', 'bbbb'] split 1 [5,12): ['cc', 'dddddd'] split 2 [12,20): ['e']

How it works: Split 0 gets aa and bbbb (bbbb crosses byte 5), split 1 gets cc and dddddd, split 2 gets e. Each record appears once.

Real-world scenario questions (asked in company interviews)

Block and split questions appear in almost every Hadoop interview. Use exact numbers when you can.

HDFS · conceptsFresher

“If a line is split across two blocks, is the record corrupted?”

An interviewer draws a CSV row crossing a block boundary.

The interviewer is checking: block vs split, and the RecordReader.

Show a strong answer

The block boundary is purely physical; the bytes are all stored, just in two blocks.
The map task whose split contains the start of the line reads past its end (possibly from another DataNode) to finish it.
The next split's reader skips everything up to the first newline. So each record is processed exactly once.
Real example: order 71484 cut between “2” and “5”; the job still read all 300,001 lines.

Performance · tuningExperienced

“The job uses only 4 mappers for 1 TB”

A job over 1 TB of logs starts only 4 map tasks and takes hours.

The interviewer is checking: what controls the number of splits.

Show a strong answer

1 TB at 128 MB should give about 8,000 splits, so something prevents splitting.
Most likely the input is 4 large gzip files (unsplittable). Check the file extensions and the job's “number of splits”.
Fix: store data in a splittable format (bzip2, or Parquet/ORC with Snappy), or split files at ingestion. Also check any custom InputFormat and split min-size settings.

Design · storageExperienced

Choose a block size for a new 2 PB table

A team will write a 2 PB fact table, scanned mostly in full each night.

The interviewer is checking: trade-offs, not a memorised number.

Show a strong answer

Large, fully scanned data favours bigger blocks, e.g. 256 MB: half the block objects, longer efficient reads.
Check parallelism: 2 PB / 256 MB is still about 80 lakh blocks, plenty of tasks.
Align file format units (Parquet row groups / ORC stripes) with the block size so a task's data sits in one block.
Keep file sizes large too; block size does not help if writers produce small files.

Storage · basicsFresher

“A 1 KB file wastes 128 MB, right?”

A student claims small files waste disk because each takes a full block.

The interviewer is checking: what really costs with small files.

Show a strong answer

No: a block file is only as big as its data (plus a tiny checksum file). The 4.5 MB file used 4.5 MB per replica, not 128 MB.
The real cost is metadata: every file and block is an object in NameNode memory, and each small file usually becomes its own map task.
So small files are a NameNode and scheduling problem, not a disk problem.

Blocks (and how they differ from splits)

A cut through the middle of an order

HDFS blocks (and input splits)

Words you'll meet in this lesson

Block facts (Hadoop 3.5.0 defaults)

Blocks and splits at three levels

Beginner A long train and its coaches

Intermediate Why big blocks?

Advanced Splits in detail

Block vs split, and choosing a block size

Common block sizes

64 MB

128 MB

256 MB

1 MB

See blocks and splits on the real cluster

Compare block sizes of the two copies

Check the real files on a DataNode

Find the record cut by the boundary

Run a job and count splits

Same job on the 128 MB copy

How map 0 and map 1 share order 71484

Try it yourself: block size in the playground

Blocks and splits at work

1. Bigger blocks for huge tables

2. Tuning splits, not blocks

3. Unsplittable files

4. Same idea in Spark

Cut a file into blocks, then read it in splits

Run it on your PC: predict Hadoop's split counters exactly

Blocks, splits and the cut record, without Hadoop

Reading split 1, one line at a time

Worked examples: real problems, solved and traced in memory

How many map tasks will this job start?

What does doubling the block size save?

Form splits with the 10% slop rule

Check yourself

Quick revision

Lesson 5.2 in one screen

Interview and viva questions

Practice problems

Blocks for a file

Check the real numbers

Add up the counters

Where is byte N?

Splits for many files

Seek share

Mapper hours wasted

Record reader for a boundary

Real-world scenario questions (asked in company interviews)

“If a line is split across two blocks, is the record corrupted?”

“The job uses only 4 mappers for 1 TB”

Choose a block size for a new 2 PB table

“A 1 KB file wastes 128 MB, right?”