Input splits, InputFormat and RecordReader

1 The story

A line cut in half

Riya stores the canteen file again, this time with tiny 1 MB blocks, so her job gets five map tasks instead of one. Then a doubt hits her. HDFS cuts blocks at exactly 1,048,576 bytes; it has no idea where a line ends. She looks at the bytes around the cut: the order 71484,poha,25 starts at byte 1,048,564 and ends at byte 1,048,578. Fourteen bytes, twelve in block 1 and two in block 2. Will map task 1 see 71484,poha,2 and map task 2 see a lonely 5?

Arjun Sir

Good doubt, and the answer is no. Blocks are cut by bytes; records are not. The piece of code that reads a split, the RecordReader, follows one rule: finish the line you started, even if it means reading a few bytes from the next block, and skip the half line you start in, because the previous reader already took it.

Riya

So the split boundary is just a suggestion?

Kabir

It's a byte range. The records are decided by the reader. We checked it on the cluster: map 0's last line was 71484,poha,25, complete, and map 1's first record started at byte 1,048,578. Five maps, 300,001 records in total, not one lost or doubled.

Arjun Sir

Today: who decides how many splits a job gets (the InputFormat), how big they are, why a 2-block file got only 1 map task, and how each record is read.

2 Definition

Input splits, InputFormat and RecordReader

Formal definition (write this in exams)

An InputFormat describes how a MapReduce job reads its input. It has two responsibilities: getSplits(), run by the client at job submission, divides the input into InputSplits, logical byte ranges (for files, a FileSplit of path, start offset, length and the hosts that store the data), each processed by exactly one map task; and createRecordReader(), run inside each map task, returns a RecordReader that turns the bytes of one split into key-value records for the mapper. For FileInputFormat the split size is max(minSize, min(maxSize, blockSize)), and a final piece of up to 10% extra (the split slop, 1.1) is merged into the previous split. The default TextInputFormat uses a LineRecordReader that yields the byte offset of each line as key (LongWritable) and the line as value (Text); it completes a line that crosses the end of the split and skips the partial first line of a split that does not start at offset 0, so every line is read exactly once.

In simple words

Before the job starts, Hadoop divides the input into pieces (splits), one per map task, usually one per block. Inside each map task a reader cuts its piece into records, carefully handling lines that cross the edges.

Words you'll meet in this lesson

InputSplitA logical chunk of input for one map task. A byte range, not a copy of data.

FileSplitThe split for files: path, start, length, hosts.

InputFormatDecides the splits and creates the record reader.

RecordReaderTurns a split's bytes into (key, value) records.

TextInputFormatThe default: key = byte offset, value = the line.

Split sizemax(minSize, min(maxSize, blockSize)); normally the block size.

Split slopA last piece up to 10% of a split is joined to the one before (1.1 rule).

SplittableA file a reader can start in the middle of. Plain text yes, gzip no.

From our lab (Hadoop 3.5.0, SplitPeek job)

canteen_orders.csv, 4.7 MB	1 block → 1 split, 300,001 records
Same file, 1 MB blocks	5 blocks → 5 splits: 70,485 + 67,321 + 65,498 + 65,574 + 31,123 = 300,001 records
1 block, split.maxsize = 1 MB	The same 5 splits and identical record counts: splits are logical
orders_x30.csv, 2 blocks	Blocks of 134,217,728 + 6,541,912 bytes → only 1 split (the slop rule), 9,000,000 records
orders_x90.csv, 422 MB	4 splits; with split.minsize = 256 MB: 2 splits
Boundary line	Split 0 ends at byte 1,048,576 but its last record starts at 1,048,564 (`71484,poha,25`); split 1's first record starts at 1,048,578

Splits and readers at three levels

Pick your level. New to this? Start at Beginner and climb.

Beginner Cutting a notebook for five friends

You tear a 100-page register into five bundles by page number: that's the split. You didn't read it, you just counted pages.
A sentence can run from page 20 onto page 21. Friend 1 reads on until the sentence ends; friend 2 ignores the half sentence at the top of page 21 and starts at the next full one: that's the RecordReader.
Nobody misses a sentence and nobody reads one twice.

Intermediate What the client and the task each do

Client, at submit time: InputFormat.getSplits() lists files, asks the NameNode for block locations and builds FileSplits. Our log: “Total input files to process : 1”, “number of splits:5”. The split list is uploaded with the job, and the MRAppMaster requests one container per split, preferring its hosts.
Map task: createRecordReader() opens the file, seeks to the split's start and calls map() once per record.
TextInputFormat records: key = byte offset of the line (LongWritable), value = the line without the newline (Text). The offsets in our SplitPeek output are exactly these keys.
Size: splitSize = max(minSize, min(maxSize, blockSize)). Default minSize 1 byte, maxSize unlimited, so splitSize = block size.

Advanced Edge rules that explain odd counts

Slop 1.1: while the remaining bytes are more than 1.1 × splitSize, cut a full split; the rest becomes the last split. 140,759,640 / 134,217,728 = 1.049, so orders_x30's two blocks gave one split, reading 6.2 MB of block 2 remotely or locally wherever the split ran.
maxsize smaller than a block gives more maps than blocks (more parallelism, more overhead). minsize bigger than a block gives fewer, bigger maps that span blocks and lose some locality.
LineRecordReader: a split not starting at 0 throws away everything up to the first newline; every reader keeps reading while the line it is about to start begins at or before its end offset. Together this assigns each line to exactly one split.
Non-splittable input (gzip, or isSplitable() returning false): one split per file, whatever its size. Module 10 compares codecs and splittability.
Splits are computed per file: 10,000 small files give at least 10,000 splits (Lesson 7.5), unless you use CombineTextInputFormat.

Exam tip: For “Difference between a block and an input split”, write: a block is the physical unit HDFS stores (128 MB, replicated); a split is the logical unit one map task processes, computed by the InputFormat, usually equal to a block but adjustable by min/max split size, and records that cross a block boundary are read whole by the RecordReader of the split where they start.

3 Types and ways to use it

InputFormats you will meet

InputFormat	Key	Value	Splits	Use it for
TextInputFormat (default)	byte offset (LongWritable)	line (Text)	per block	CSV, logs, any text
KeyValueTextInputFormat	text before the first tab	rest of the line	per block	Output of an earlier job (key TAB value)
NLineInputFormat	byte offset	line	every N lines	Each line is a heavy task (e.g. a URL to fetch)
SequenceFileInputFormat	stored key	stored value	per block (sync markers)	Binary key-value files between jobs
CombineTextInputFormat	offset	line	many files per split	Lots of small files
DBInputFormat	row number	a DBWritable row	row ranges	Reading a small database table

The settings that change the number of maps

Block size

Decided when the file is written (-D dfs.blocksize=…). Our 1 MB-block copy gave 5 maps.

physical

split.maxsize

mapreduce.input.fileinputformat.split.maxsize: cap below the block size for more maps. 1 MB gave the same 5 splits.

more maps

split.minsize

mapreduce.input.fileinputformat.split.minsize: raise above the block size for fewer maps. 256 MB turned 4 splits into 2.

fewer maps

Compression

A gzip file can't be entered in the middle: always 1 split per file. bzip2 and container formats can be split.

splittable?

You cannot set the number of map tasks directly in the new API: it follows the splits. To change it, change the split size or the block size. The number of reduce tasks, on the other hand, is simply mapreduce.job.reduces (Lesson 8.5).

4 Step by step

Watching splits with SplitPeek

SplitPeek.java is a map-only job whose mapper counts the records it gets and, in cleanup(), writes one line per split: its start and length, the first and last record offsets, the last line and the host it ran on. Run with splits_lab.sh on the Module 8 lab (in “Run it on your PC”).

One small file, one split
```
hadoop jar mr.jar SplitPeek /user/hadoop/data/canteen_orders.csv /out/peek_a
hdfs dfs -cat /out/peek_a/part-m-*
```
number of splits:1 split start=0 length=4692009 | records=300001 first_offset=0 last_offset=4691994 last_line=301000,idli,30 | ran_on=worker2
4.7 MB is less than one 128 MB block, so one split covers the whole file. The last line starts at byte 4,691,994: 301000,idli,30 plus the newline is 15 bytes, ending exactly at 4,692,009.
Same data, 1 MB blocks
```
hdfs dfs -D dfs.blocksize=1048576 -put -f canteen_orders.csv /user/hadoop/data/orders_1mb_blocks.csv
hadoop jar mr.jar SplitPeek /user/hadoop/data/orders_1mb_blocks.csv /out/peek_b
```
Total blocks (validated): 5 (avg. block size 938401 B) number of splits:5 split start=0 length=1048576 | records=70485 first_offset=0 last_offset=1048564 last_line=71484,poha,25 | ran_on=worker2 split start=1048576 length=1048576 | records=67321 first_offset=1048578 last_offset=2097145 last_line=138805,idli,30 | ran_on=worker3 split start=2097152 length=1048576 | records=65498 first_offset=2097160 last_offset=3145721 last_line=204303,chai,15 | ran_on=worker1 split start=3145728 length=1048576 | records=65574 first_offset=3145736 last_offset=4194301 last_line=269877,samosa,18 | ran_on=worker2 split start=4194304 length=497705 | records=31123 first_offset=4194318 last_offset=4691994 last_line=301000,idli,30 | ran_on=worker1
Five blocks, five splits, five maps on all three workers. No split's first record starts exactly at its start: each began by skipping a partial line. The record counts add up to 300,001.
Read the boundary
```
split 0 ends at        1,048,576
its last line starts   1,048,564   "71484,poha,25\n" = 14 bytes
so that line ends at   1,048,578   (2 bytes inside block 2)
split 1 first record   1,048,578
```
Map 0 read 2 bytes past its split to finish the line; map 1 threw away those 2 bytes (“5” and the newline) and started at the next line. The same happens at every boundary: 2,097,145 + 15 = 2,097,160, the first record of split 2.
Splits without changing blocks
```
hadoop jar mr.jar SplitPeek -D mapreduce.input.fileinputformat.split.maxsize=1048576 \
    /user/hadoop/data/canteen_orders.csv /out/peek_c
```
number of splits:5 split start=0 length=1048576 | records=70485 first_offset=0 last_offset=1048564 last_line=71484,poha,25 | ran_on=worker3 … split start=4194304 length=497705 | records=31123 first_offset=4194318 last_offset=4691994 last_line=301000,idli,30 | ran_on=worker3
The file is still one 4.7 MB block, yet the job got the same 5 splits with identical record counts. A split is only a byte range; it does not need to match a block.
Two blocks, one split
```
hdfs fsck /user/hadoop/data/orders_x30.csv -files -blocks
hadoop jar mr.jar SplitPeek /user/hadoop/data/orders_x30.csv /out/peek_d
```
0. …blk_1073741827_1003 len=134217728 Live_repl=3 1. …blk_1073741828_1004 len=6541912 Live_repl=3 number of splits:1 split start=0 length=140759640 | records=9000000 first_offset=0 last_offset=140759625 last_line=301000,idli,30 | ran_on=worker1
140,759,640 bytes is only 1.049 times the 128 MB split size, below the 1.1 slop, so FileInputFormat makes one split instead of a 128 MB split plus a 6.2 MB one. A tiny second map task would cost more than it saves.
Fewer, bigger splits
```
hadoop jar mr.jar SplitPeek /user/hadoop/data/orders_x90.csv /out/peek_e
hadoop jar mr.jar SplitPeek -D mapreduce.input.fileinputformat.split.minsize=268435456 \
    /user/hadoop/data/orders_x90.csv /out/peek_f
```
number of splits:4 number of splits:2
With minSize 256 MB, splitSize = max(256 MB, min(∞, 128 MB)) = 256 MB: two map tasks, each reading two blocks. Fewer task start-ups, but part of each split may be read from another node.

Don't “fix” a slow job by setting split.maxsize very low. Our 4.7 MB file in five 1 MB splits started five JVMs for work one map does in seconds. Smaller splits help only when each map does heavy work per record.

5 See it happen

From a file to records

Six steps: the client computes splits, two map tasks read across the boundary at byte 1,048,576.

Splits and the boundary line 6 steps

What you see

Behind the screen

6 Real-world use

Splits at work

1. The single giant gzip file

A vendor sends a 40 GB .gz file daily. The job gets one map task that runs for hours while the rest of the cluster sits idle. Teams recompress to a splittable format (bzip2, or Parquet/ORC with block compression) or ask for many smaller files.

40 GB .gz→1 split→Recompress

2. NLineInputFormat for heavy rows

A file of 5,000 lines, each naming a large image or a model to score. As one 50 KB split it would be one map; NLineInputFormat with N = 50 gives 100 maps working in parallel.

5,000 lines→N = 50→100 maps

3. Combining small files

Daily folders with thousands of small log files are read with CombineTextInputFormat and a max split size of 256 MB, so a job runs a few dozen maps instead of thousands.

8,000 files→CombineTextInputFormat→~40 maps

4. Tuning map count for CPU-heavy jobs

A parsing job spends 90% of its time on CPU. Lowering split.maxsize to 64 MB doubles the number of maps and uses every core; for an I/O-bound job the same change only adds overhead.

128 MB splits→maxsize 64 MB→2× maps

Simplified; organisations differ in tools and policies.

7 Run the code

Compute splits like FileInputFormat

This is the core of FileInputFormat.getSplits() for one file: the split size formula and the 1.1 slop rule. The file sizes are the real ones from our lab, and the printed counts match the job logs.

MB = 1024 * 1024
SLOP = 1.1

def get_splits(length, block=128 * MB, min_size=1, max_size=2**63 - 1):
    split_size = max(min_size, min(max_size, block))
    splits, start, remaining = [], 0, length
    while remaining / split_size > SLOP:
        splits.append((start, split_size))
        start += split_size
        remaining -= split_size
    if remaining:
        splits.append((start, remaining))
    return splits

files = {"canteen_orders.csv": 4_692_009, "orders_x30.csv": 140_759_640, "orders_x90.csv": 422_278_920}
print("A", len(get_splits(files["canteen_orders.csv"])), "split(s)")
print("B", get_splits(files["canteen_orders.csv"], block=1 * MB))
print("D", len(get_splits(files["orders_x30.csv"])), "split(s)")
print("E", len(get_splits(files["orders_x90.csv"])), "split(s)")
print("F", len(get_splits(files["orders_x90.csv"], min_size=256 * MB)), "split(s) with minsize 256 MB")

A 1 split(s) B [(0, 1048576), (1048576, 1048576), (2097152, 1048576), (3145728, 1048576), (4194304, 497705)] D 1 split(s) E 4 split(s) F 2 split(s) with minsize 256 MB

split_size = max(min_size, min(max_size, block))The formula from FileInputFormat.computeSplitSize(). With default min and max it is simply the block size.
while remaining / split_size > SLOP:Keep cutting full splits only while more than 1.1 splits are left; the rest becomes the last split.
print("D", len(get_splits(files["orders_x30.csv"])), "split(s)")140,759,640 / 134,217,728 = 1.049, below 1.1: two blocks, one split, exactly as the job log said.

Run it on your PC: peek inside every split

Use the Module 8 lab from Lesson 8.1 (lab8). SplitPeek.java is a map-only job that writes one line per split. splits_lab.sh runs it six ways. It takes about 3 minutes.

A job that reports its own split

// SplitPeek.java: a map-only job that reports, for every input split, where it starts,
// how long it is, which hosts hold it, the first and last record offsets, and the record count.
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.FileSplit;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class SplitPeek {
  public static class PeekMapper extends Mapper<LongWritable, Text, NullWritable, Text> {
    private long first = -1, last = -1, records = 0;
    private String lastLine = "";

    @Override
    protected void map(LongWritable offset, Text line, Context ctx) {
      if (first < 0) first = offset.get();
      last = offset.get();
      lastLine = line.toString();
      records++;
    }

    @Override
    protected void cleanup(Context ctx) throws IOException, InterruptedException {
      FileSplit s = (FileSplit) ctx.getInputSplit();
      String host = java.net.InetAddress.getLocalHost().getHostName();
      ctx.write(NullWritable.get(), new Text(String.format(
          "split start=%d length=%d | records=%d first_offset=%d last_offset=%d last_line=%s | ran_on=%s",
          s.getStart(), s.getLength(), records, first, last, lastLine, host)));
    }
  }

  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    String[] rest = new GenericOptionsParser(conf, args).getRemainingArgs();
    Job job = Job.getInstance(conf, "split peek");
    job.setJarByClass(SplitPeek.class);
    job.setMapperClass(PeekMapper.class);
    job.setNumReduceTasks(0);                     // map-only: one output file per map task
    job.setOutputKeyClass(NullWritable.class);
    job.setOutputValueClass(Text.class);
    FileInputFormat.addInputPath(job, new Path(rest[0]));
    FileOutputFormat.setOutputPath(job, new Path(rest[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

// In the mr-lab folder:
//   docker cp SplitPeek.java lab8-namenode-1:/tmp/mr/
//   docker exec -it lab8-namenode-1 bash
// Inside the container (canteen_orders.csv and orders_x30/x90 from Lesson 8.1 are already in HDFS):
//   cd /tmp/mr
//   mkdir -p classes && javac -cp $(hadoop classpath) -d classes *.java && jar cf mr.jar -C classes .
//   hadoop jar mr.jar SplitPeek /user/hadoop/data/canteen_orders.csv /out/peek_a
//   hdfs dfs -cat /out/peek_a/part-m-*

== A: canteen_orders.csv (4.7 MB, 1 block) number of splits:1 split start=0 length=4692009 | records=300001 first_offset=0 last_offset=4691994 last_line=301000,idli,30 | ran_on=worker2

Real output. The key the mapper receives is the byte offset of each line, so first_offset and last_offset are simply the first and last keys it saw.

Six ways to split the same data

# splits_lab.sh: how many input splits (map tasks) does Hadoop make, and where do records start?
cd /tmp/mr
run() {  # run SplitPeek with extra options, print split count and one line per split
  out=/out/peek_$1; shift
  hdfs dfs -rm -r -f -skipTrash $out > /dev/null 2>&1
  hadoop jar mr.jar SplitPeek "$@" $out 2>&1 | grep -o "number of splits:[0-9]*"
  hdfs dfs -cat $out/part-m-* 2>/dev/null
}
echo "== A: canteen_orders.csv (4.7 MB, 1 block)"
run a /user/hadoop/data/canteen_orders.csv
echo "== B: same data stored with 1 MB blocks"
hdfs dfs -D dfs.blocksize=1048576 -put -f canteen_orders.csv /user/hadoop/data/orders_1mb_blocks.csv
hdfs fsck /user/hadoop/data/orders_1mb_blocks.csv -blocks 2>/dev/null | grep "Total blocks"
run b /user/hadoop/data/orders_1mb_blocks.csv
echo "== C: 1 block, but split.maxsize = 1 MB"
run c -D mapreduce.input.fileinputformat.split.maxsize=1048576 /user/hadoop/data/canteen_orders.csv
echo "== D: orders_x30.csv (2 blocks: 128 MB + 6.2 MB)"
run d /user/hadoop/data/orders_x30.csv
echo "== E: orders_x90.csv (4 blocks), then split.minsize = 256 MB"
run e /user/hadoop/data/orders_x90.csv | head -1
run f -D mapreduce.input.fileinputformat.split.minsize=268435456 /user/hadoop/data/orders_x90.csv | head -1

# Inside the namenode container:
#   bash /tmp/mr/splits_lab.sh

== A: canteen_orders.csv (4.7 MB, 1 block) number of splits:1 split start=0 length=4692009 | records=300001 first_offset=0 last_offset=4691994 last_line=301000,idli,30 | ran_on=worker2 == B: same data stored with 1 MB blocks Total blocks (validated): 5 (avg. block size 938401 B) number of splits:5 split start=0 length=1048576 | records=70485 first_offset=0 last_offset=1048564 last_line=71484,poha,25 | ran_on=worker2 split start=1048576 length=1048576 | records=67321 first_offset=1048578 last_offset=2097145 last_line=138805,idli,30 | ran_on=worker3 split start=2097152 length=1048576 | records=65498 first_offset=2097160 last_offset=3145721 last_line=204303,chai,15 | ran_on=worker1 split start=3145728 length=1048576 | records=65574 first_offset=3145736 last_offset=4194301 last_line=269877,samosa,18 | ran_on=worker2 split start=4194304 length=497705 | records=31123 first_offset=4194318 last_offset=4691994 last_line=301000,idli,30 | ran_on=worker1 == C: 1 block, but split.maxsize = 1 MB number of splits:5 split start=0 length=1048576 | records=70485 first_offset=0 last_offset=1048564 last_line=71484,poha,25 | ran_on=worker3 split start=1048576 length=1048576 | records=67321 first_offset=1048578 last_offset=2097145 last_line=138805,idli,30 | ran_on=worker1 split start=2097152 length=1048576 | records=65498 first_offset=2097160 last_offset=3145721 last_line=204303,chai,15 | ran_on=worker2 split start=3145728 length=1048576 | records=65574 first_offset=3145736 last_offset=4194301 last_line=269877,samosa,18 | ran_on=worker3 split start=4194304 length=497705 | records=31123 first_offset=4194318 last_offset=4691994 last_line=301000,idli,30 | ran_on=worker3 == D: orders_x30.csv (2 blocks: 128 MB + 6.2 MB) number of splits:1 split start=0 length=140759640 | records=9000000 first_offset=0 last_offset=140759625 last_line=301000,idli,30 | ran_on=worker1 == E: orders_x90.csv (4 blocks), then split.minsize = 256 MB number of splits:4 number of splits:2

Real output. Which worker ran each split will differ on your machine; split boundaries, offsets and record counts will be identical because the data is generated with a fixed seed.

8 Line by line

Who reads the line that crosses the boundary?

Two readers share 39 bytes, cut at byte 20. Follow pos in each: the second reader jumps past the first newline, the first reads past its end.

Worked examples: real problems, solved and traced in memory

Three split calculations: from block layout to map count, a boundary check, and choosing a split size.

Example 1BeginnerSplits · count

How many map tasks will this job get?

Problem statement

A folder holds three files: 300 MB, 130 MB and 20 MB, with 128 MB blocks and default split settings. How many map tasks run?

Input

300 MB · 130 MB · 20 MB

Expected output

[3, 1, 1] -> 5 map tasks

Approach

Splits are computed per file.
Apply the 1.1 slop rule to each file.
Add them up.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 2IntermediateRecords · boundary

Check a boundary from the job output

Problem statement

SplitPeek reported for split 1: last_offset=2097145 last_line=138805,idli,30, and split 2 started at byte 2,097,152 with first_offset=2097160. Show that no line was lost between them.

Input

2,097,145 · "138805,idli,30" · 2,097,160

Expected output

line ends at 2097160 | split 2 first record 2097160 | match: True
bytes read past the split end: 8

Approach

The last line of split 1 ends at its offset + its length + 1 (newline).
That must be exactly where split 2's first record starts.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 3AdvancedTuning · size

Pick a split size for a 10 GB job

Problem statement

A 10 GB text file, 128 MB blocks, on a cluster with 40 free map containers. Each map has a fixed 6 s start-up and processes 50 MB/s. Compare 128 MB splits, 256 MB splits (split.minsize) and 64 MB splits (split.maxsize).

Input

10 GB · 40 containers · 6 s start · 50 MB/s

Expected output

 64 MB: 160 maps, 4 waves, ~29 s
128 MB:  80 maps, 2 waves, ~17 s
256 MB:  40 maps, 1 waves, ~11 s

Approach

Maps = size / split size; waves = maps / 40.
Time per map = 6 s + split / 50 MB/s.
Total ≈ waves × time per map.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

9 Quiz and revision

Check yourself

Quick revision

Lesson 8.2 in one screen

InputFormat: getSplits() on the client, createRecordReader() in each map task.
One map task per split; a split is a byte range (FileSplit: path, start, length, hosts), not a copy of data.
splitSize = max(minSize, min(maxSize, blockSize)); last piece up to 1.1 × is merged (slop).
Lab: 4.7 MB → 1 split; 1 MB blocks or maxsize 1 MB → 5; 134 MB in 2 blocks → 1; 422 MB → 4, minsize 256 MB → 2.
TextInputFormat: key = byte offset (LongWritable), value = line (Text).
LineRecordReader: skip the partial first line (unless start = 0), read past the end to finish the last line; each line read once.
Non-splittable files (gzip) → 1 split per file; many small files → use CombineTextInputFormat.

Interview and viva questions

What is the difference between an HDFS block and an input split?

A block is the physical unit HDFS stores and replicates (128 MB by default). An input split is a logical byte range computed by the job's InputFormat; one map task processes one split. By default a split equals a block, but split.minsize and split.maxsize change that, and the RecordReader reads records that cross block boundaries whole, so a split's records may extend slightly into the next block.

How is the number of mappers decided in Hadoop?

By the number of input splits that the InputFormat computes. For FileInputFormat it depends on the file sizes, block size, mapreduce.input.fileinputformat.split.minsize and split.maxsize, the 1.1 split slop, and whether files are splittable. You cannot set the number of map tasks directly in the new API.

How does Hadoop handle records that cross block boundaries?

The LineRecordReader of a split that does not start at byte 0 skips everything up to the first newline, because that partial line belongs to the previous split. Every reader keeps reading while the next line starts at or before the end of its split, so it finishes a line that crosses the boundary by reading a few bytes of the next block. Each line is therefore processed exactly once.

What does TextInputFormat pass to the mapper?

For each line, a LongWritable key holding the byte offset where the line starts in the file, and a Text value holding the line without its line terminator. The offset is often ignored by the mapper.

Why does my job have only one mapper for a large file?

Usually because the file is compressed with a non-splittable codec such as gzip, the InputFormat marks it as not splittable, or split.minsize is set larger than the file. A file just slightly bigger than one block (up to 10% more) also gets a single split because of the split slop.

Practice problems

Split and record-reader exercises using the real offsets and sizes from our lab.

P1Easy

Records add up

Add the record counts of the five splits and compare with the single-split run.

Input

70,485 · 67,321 · 65,498 · 65,574 · 31,123

Expected output

300001 True

Show solution and explanation

parts = [70485, 67321, 65498, 65574, 31123]
print(sum(parts), sum(parts) == 300001)

300001 True

How it works: 300,001 in both runs: the boundary rule loses and duplicates nothing.

P2Easy

Last split length

The file is 4,692,009 bytes and splits are 1 MB. What is the length of the last split?

Input

4,692,009 · 1,048,576

Expected output

Show solution and explanation

print(4692009 - 4 * 1048576)

497705

How it works: 497,705 bytes, exactly what SplitPeek printed.

P3Easy

Slop ratio

Compute the ratio that decided orders_x30.csv's single split.

Input

140,759,640 / 134,217,728

Expected output

1.049 <= 1.1 -> one split

Show solution and explanation

print(round(140759640 / 134217728, 3), "<= 1.1 -> one split")

1.049 <= 1.1 -> one split

How it works: 1.049: below 1.1, so no separate split for the 6.2 MB tail.

P4Medium

Boundary at split 3

Split 3 ended at byte 4,194,304; its last line started at 4,194,301 (269877,samosa,18). Where does split 4 start reading?

Input

4,194,301 + line + newline

Expected output

Show solution and explanation

print(4194301 + len("269877,samosa,18") + 1)

4194318

How it works: 4,194,318, the first_offset of split 4. Map 3 read 14 bytes past its end.

P5Medium

KeyValueTextInputFormat

Parse output lines of Lesson 8.1 the way KeyValueTextInputFormat does: key before the first tab, value after.

Input

part-r-00000 lines

Expected output

'chai' -> '50170\t752550'
'vada pav' -> '49986\t999720'

Show solution and explanation

lines = ["chai\t50170\t752550", "vada pav\t49986\t999720"]
for l in lines:
    key, value = l.split("\t", 1)
    print(repr(key), "->", repr(value))

'chai' -> '50170\t752550' 'vada pav' -> '49986\t999720'

How it works: Only the first tab separates; the value keeps the rest. Useful for chaining jobs.

P6Medium

NLineInputFormat

5,000 lines, N = 50 lines per split. How many maps, and which lines does map 7 get?

Input

5,000 lines · N = 50

Expected output

100 maps; map 7 gets lines 351 to 400

Show solution and explanation

n = 50
print(5000 // n, "maps; map 7 gets lines", 7 * n + 1, "to", 8 * n)

100 maps; map 7 gets lines 351 to 400

How it works: 100 maps; map 7 (counting from 0) gets lines 351–400.

P7Business problem

The gzip bottleneck

A daily 60 GB gzip file takes one map 2 hours. Recompressed to splittable 128 MB pieces on a cluster with 100 free containers, each piece takes 45 s plus 5 s start-up. Estimate the new map time.

Input

60 GB · 100 containers · 50 s per map

Expected output

480 maps, 5 waves, about 4 minutes instead of 120

Show solution and explanation

import math
maps = math.ceil(60 * 1024 / 128)
waves = math.ceil(maps / 100)
print(maps, "maps,", waves, "waves, about", round(waves * 50 / 60), "minutes instead of 120")

480 maps, 5 waves, about 4 minutes instead of 120

How it works: 480 maps in 5 waves: about 4 minutes instead of 2 hours (illustrative numbers).

P8Advanced

A full LineRecordReader check

Generate 1,000 order lines, cut them into 7 equal byte ranges, read each with the boundary rule, and verify that every line is read exactly once.

Input

1,000 lines · 7 splits

Expected output

1000 lines read; True

Show solution and explanation

import random
random.seed(1)
items = ["chai", "dosa", "poha", "idli"]
data = "".join(f"{1000 + i},{random.choice(items)},{random.randint(10, 50)}\n" for i in range(1000))
size = len(data) // 7 + 1
seen = []
for start in range(0, len(data), size):
    end = start + size
    pos = 0 if start == 0 else data.index("\n", start) + 1
    while pos <= end and pos < len(data):
        nl = data.index("\n", pos)
        seen.append(data[pos:nl]); pos = nl + 1
print(len(seen), "lines read;", len(set(seen)) == 1000 and len(seen) == 1000)

1000 lines read; True

How it works: 1000 lines read exactly once, whatever the cut points. Try other split counts.

Real-world scenario questions (asked in company interviews)

Splits come up whenever an interviewer asks how many mappers a job will have, or why a job is slow.

Concepts · basicsFresher

“Block vs input split?”

A classic Hadoop interview question.

The interviewer is checking: physical vs logical.

Show a strong answer

Block: physical storage unit of HDFS, fixed size, replicated, cut by bytes.
Split: logical unit of work for one map task, computed by the InputFormat at submission; usually one block.
Records crossing a block edge are read whole by the split they start in; min/max split size can make splits differ from blocks.

Internals · recordsFresher

“A line is split across two blocks. Does the mapper get half a line?”

Follow-up to the first question.

The interviewer is checking: the LineRecordReader rule.

Show a strong answer

No. The reader of the first split reads past its end to finish the line (a remote read of a few bytes if needed).
The reader of the next split skips its partial first line, because it belongs to the previous split.
So each line is processed exactly once; only the split starting at byte 0 doesn't skip.

Performance · tuningExperienced

A job on one 50 GB file runs with a single mapper for 3 hours

The cluster is mostly idle while it runs.

The interviewer is checking: splittability diagnosis.

Show a strong answer

Check “number of splits” in the client log; one split for 50 GB almost always means a non-splittable codec (gzip) or a custom isSplitable() returning false.
Fix upstream: write splittable files (bzip2, or Parquet/ORC/Avro with block compression) or many smaller gzip files.
Short term: decompress once into HDFS as plain text, or repartition with a one-off job.

Tuning · map countExperienced

“How do you increase or decrease the number of mappers?”

Often asked right after “how is the number of mappers decided”.

The interviewer is checking: knowing which knob does what.

Show a strong answer

More maps: lower mapreduce.input.fileinputformat.split.maxsize (or write with a smaller block size).
Fewer maps: raise split.minsize, or use CombineTextInputFormat for many small files.
Mention the trade-off: more maps add start-up overhead; fewer, bigger maps lose locality and parallelism.

Input splits, InputFormat and RecordReader

A line cut in half

Input splits, InputFormat and RecordReader

Words you'll meet in this lesson

From our lab (Hadoop 3.5.0, SplitPeek job)

Splits and readers at three levels

Beginner Cutting a notebook for five friends

Intermediate What the client and the task each do

Advanced Edge rules that explain odd counts

InputFormats you will meet

The settings that change the number of maps

Block size

split.maxsize

split.minsize

Compression

Watching splits with SplitPeek

One small file, one split

Same data, 1 MB blocks

Read the boundary

Splits without changing blocks

Two blocks, one split

Fewer, bigger splits

From a file to records

Splits at work

1. The single giant gzip file

2. NLineInputFormat for heavy rows

3. Combining small files

4. Tuning map count for CPU-heavy jobs

Compute splits like FileInputFormat

Run it on your PC: peek inside every split

A job that reports its own split

Six ways to split the same data

Who reads the line that crosses the boundary?

Worked examples: real problems, solved and traced in memory

How many map tasks will this job get?

Check a boundary from the job output

Pick a split size for a 10 GB job

Check yourself

Quick revision

Lesson 8.2 in one screen

Interview and viva questions

Practice problems

Records add up

Last split length

Slop ratio

Boundary at split 3

KeyValueTextInputFormat

NLineInputFormat

The gzip bottleneck

A full LineRecordReader check

Real-world scenario questions (asked in company interviews)

“Block vs input split?”

“A line is split across two blocks. Does the mapper get half a line?”

A job on one 50 GB file runs with a single mapper for 3 hours

“How do you increase or decrease the number of mappers?”