Custom Writable types

1 The story

The “efficient” version that was bigger

Kabir wants, for every student and every day, three numbers: how many orders, how much money, and the biggest single order. His first version glues everything into strings: the key is "S0001\t2026-09-01", the value "1,20,20". Riya says strings are wasteful and writes proper Java types instead: a key class with two fields and a value class with an int, a long and an int. They run both on 5 lakh orders. Riya's version moves 40% more bytes through the shuffle.

Riya

Sir, I used real types instead of strings. How can mine be bigger?

Arjun Sir

Look at what your write() does. writeUTF("2026-09-10") writes a 2-byte length and 10 characters: 12 bytes for a date. Then writeInt is always 4 bytes and writeLong always 8, even when the number is 1 or 45. Kabir's text "1,45,45" is only 7 characters.

Kabir

So Writable is just “you decide the bytes”. Decide badly and you pay in the shuffle.

Arjun Sir

Exactly. Today we write a composite key and a value type, see what Hadoop needs from them (and what happens when you forget the empty constructor), then shrink the same record to 8 bytes with variable-length numbers: 4.3 times smaller than the first try.

2 Definition

Writable and WritableComparable

Formal definition (write this in exams)

Writable (org.apache.hadoop.io.Writable) is Hadoop's serialization interface. A type implements write(DataOutput out), which writes its fields as bytes, and readFields(DataInput in), which reads them back in the same order into an existing object. MapReduce uses Writables for all keys and values that are transferred between tasks or stored in SequenceFiles. Keys must implement WritableComparable, which adds compareTo(): it defines the order in which keys are sorted during the shuffle. The default partitioner uses the key's hashCode() to choose a reducer, so equal keys must have equal hash codes. Because the framework creates instances by reflection and reuses them for many records, a Writable must have a public no-argument constructor and readFields() must overwrite every field. A RawComparator can compare keys directly in their serialized form, avoiding deserialization during sorting.

In simple words

A Writable is a class that knows how to turn itself into bytes and back. You choose which bytes. A key also needs to know how to compare itself, because the shuffle sorts by it.

Words you'll meet in this lesson

SerializationTurning an object into bytes to send or store; deserialization is the reverse.

write() / readFields()The two methods of Writable. Same fields, same order, or the data is garbage.

Composite keyA key made of several fields, e.g. (student, day).

compareTo()Sort order of keys in the shuffle and in the output.

hashCode()Used by HashPartitioner to choose the reducer.

VInt / VLongVariable-length numbers: 1 byte for -112 to 127, up to 5 (int) or 9 (long).

No-arg constructorHadoop creates keys and values with new X() by reflection.

RawComparatorCompares two keys as bytes, without building objects.

From our lab (Hadoop 3.5.0, 5 lakh orders → 59,946 student-days)

Writable, writeUTF + int/long/int	Map output bytes 17,500,000 (35 per record), materialized 5,604,593, CPU 8,360 ms
Text key and value	12,500,000 (25 per record), materialized 4,132,541, CPU 7,290 ms
Writable, VInt/VLong, student and day as numbers	4,068,724 (about 8 per record), materialized 1,601,326, CPU 8,600 ms
Results	All three wrote the same 59,946 rows, e.g. `S0001 2026-09-01 124 3224 80`; each job took 24–25 s
No-arg constructor removed	`NoSuchMethodException: StudentDayNoCtor$StudentDayKey.<init>()`, “Initialization of all the collectors failed”, job FAILED

Writables at three levels

Pick your level. New to this? Start at Beginner and climb.

Beginner Packing a parcel

Sending a parcel by courier: you decide how to pack it. Every map output record is packed (write), sent through the shuffle, and unpacked (readFields) on the other side.
The receiver unpacks in the same order you packed: first the student, then the day. Change the order on one side only and you get nonsense.
A big box for a small thing still costs postage. A 4-byte int for the number 1 is a big box.

Intermediate What Hadoop needs from your class

Value types implement Writable: write() and readFields().
Key types implement WritableComparable: also compareTo() (sort order), and override hashCode() and equals() (partitioning).
A public no-argument constructor. Without it the map task cannot even start its output collector: our job failed on every attempt.
toString() decides how the type looks in TextOutputFormat output: our key prints S0001<TAB>2026-09-01.
Objects are reused: readFields() must set every field, and never store the object itself (Lesson 8.5).

Advanced Making records small and sorting fast

Shuffle cost is per byte: map output is buffered, spilled, compressed, copied and merged. Smaller records mean fewer spills (Lesson 8.4).
Encode the information, not the text: an ID like S1413 is the number 1413; a date in one month is a day-of-year number; small counts fit in a VInt.
Fixed sizes (writeInt) are simpler and slightly faster to decode; VInts are smaller for small numbers. Our compact job used a little more CPU (8,600 ms vs 8,360) for 4.3× fewer bytes.
Sorting calls compareTo() millions of times. The default WritableComparator deserializes both keys first; a RawComparator that compares the bytes directly (as Text and IntWritable do) is faster. Register it with WritableComparator.define() or job.setSortComparatorClass().

Exam tip: For “Why does Hadoop use Writable instead of Java Serializable?”, give three points: compact (no class names or metadata per object, only the fields), fast (objects are reused, no new object per record), and comparable in serialized form (RawComparator for sorting). Mention that keys must be WritableComparable.

3 Types and ways to use it

Built-in Writables and their sizes

Type	Java value	Bytes on the wire	Use it for
Text	String (UTF-8)	VInt length + UTF-8 bytes	Words, IDs, lines
IntWritable / LongWritable	int / long	always 4 / 8	Counts, offsets (TextInputFormat key)
VIntWritable / VLongWritable	int / long	1 to 5 / 1 to 9	Mostly small numbers
DoubleWritable, BooleanWritable	double, boolean	8, 1	Measurements, flags
NullWritable	nothing	0	Key-only or value-only output
BytesWritable	byte[]	4-byte length + bytes	Binary blobs
ArrayWritable, MapWritable	arrays, maps of Writables	count + elements (MapWritable adds type ids)	Quick prototypes; a custom class is usually smaller

Ways to serialize records in Hadoop

Text with separators

Easiest to write and debug; numbers become characters, and you parse them again in the reducer.

simple

Custom Writable

Exactly the fields you choose. As small as you make it: 35 B or 8 B for the same record in our lab.

this lesson

Java Serializable

Writes class metadata and creates new objects per record. Hadoop does not use it for keys and values.

avoid

Avro (and Parquet)

Schema-based binary records that other tools and languages can read; used for files that outlive one job (Module 10).

data files

Before writing a custom type, check the counter Map output bytes of the Text version. It tells you the bytes per record (divide by Map output records), so you can see whether a Writable would actually save anything.

4 Step by step

A composite key, a value type, and three sizes

Run on the Module 8 lab cluster with writable_lab.sh and student_orders.csv (5 lakh orders, 2,000 students, September 2026). Files are in “Run it on your PC”.

The key: (student, day)

public static class StudentDayKey implements WritableComparable<StudentDayKey> {
  private String student = "", day = "";
  public StudentDayKey() { }                       // Hadoop needs this
  public void write(DataOutput out) throws IOException { out.writeUTF(student); out.writeUTF(day); }
  public void readFields(DataInput in) throws IOException { student = in.readUTF(); day = in.readUTF(); }
  public int compareTo(StudentDayKey o) {
    int c = student.compareTo(o.student);
    return c != 0 ? c : day.compareTo(o.day);       // sort by student, then day
  }
  public int hashCode() { return student.hashCode() * 31 + day.hashCode(); }
  public String toString() { return student + "\t" + day; }
}

compareTo makes the output come out sorted by student and then by day. hashCode decides the reducer. Both must agree with equals: equal keys, equal hash.

The value: orders, total, max

public static class SpendWritable implements Writable {
  private int orders; private long total; private int max;
  public SpendWritable() { }
  public void write(DataOutput out) throws IOException { out.writeInt(orders); out.writeLong(total); out.writeInt(max); }
  public void readFields(DataInput in) throws IOException { orders = in.readInt(); total = in.readLong(); max = in.readInt(); }
}

The reducer adds orders and totals and keeps the maximum, which is also safe as a combiner (Lesson 9.1). Combine output records: 151,475 instead of 500,000.

Run the Writable version and the Text version
```
hadoop jar mr9.jar StudentDay     /user/hadoop/data/student_orders.csv /out/StudentDay
hadoop jar mr9.jar StudentDayText /user/hadoop/data/student_orders.csv /out/StudentDayText
```
== StudentDay 24815 ms Map output records=500000 Map output bytes=17500000 Map output materialized bytes=5604593 == StudentDayText 24363 ms Map output records=500000 Map output bytes=12500000 Map output materialized bytes=4132541
17,500,000 / 500,000 = 35 bytes per record for the Writable, 25 for Text. Key: 2+5 (S1413) + 2+10 (the date) = 19 bytes; value: 4+8+4 = 16 bytes. Text: 1+16 for the key and 1+7 for "1,45,45".
Shrink it: numbers instead of strings, VInts instead of fixed sizes
```
private int student;   // "S1413" -> 1413
private int day;       // 2026-09-10 -> day of year 253
public void write(DataOutput out) throws IOException {
  WritableUtils.writeVInt(out, student);   // 1413 -> 3 bytes
  WritableUtils.writeVInt(out, day);       // 253  -> 2 bytes (256 and above need 3)
}
// value: writeVInt(orders), writeVLong(total), writeVInt(max) -> 1 + 1 + 1 bytes for small amounts
```
== StudentDayCompact 25126 ms Map output records=500000 Map output bytes=4068724 Map output materialized bytes=1601326
About 8 bytes per record (6 to 9, depending on the student number and the date): 4.3 times smaller than the first Writable and 3 times smaller than Text. Students S0001–S0127 even fit in 1 byte. CPU time barely moved (8,600 ms).
Check that all three agree
```
hdfs dfs -cat /out/StudentDay/part-r-00000 | head -3
diff <(hdfs dfs -cat /out/StudentDay/part-r-00000) <(hdfs dfs -cat /out/StudentDayCompact/part-r-00000)
```
S0001 2026-09-01 124 3224 80 S0001 2026-09-02 134 3411 80 S0001 2026-09-03 124 3064 80 compact: same rows text: same rows
Same 59,946 rows. The compact key's toString() turns the numbers back into S0001 and a date, so the output looks identical.
Forget the no-argument constructor
```
# replace  public StudentDayKey() { }  with  public StudentDayKey(String s, String d) { … }
hadoop jar nc.jar StudentDayNoCtor /user/hadoop/data/student_orders.csv /out/nc
```
Caused by: java.lang.NoSuchMethodException: StudentDayNoCtor$StudentDayKey.<init>() Error: java.io.IOException: Initialization of all the collectors failed. … Job job_1792643020992_0034 failed with state FAILED due to: Task failed task_1792643020992_0034_m_000000
It compiles fine; it fails only on the cluster. The map output collector creates key objects by reflection (ReflectionUtils.newInstance), which needs new StudentDayKey().

write() and readFields() must handle the same fields in the same order and with the same methods. If write uses writeVInt and readFields uses readInt, nothing fails at compile time; the reducer silently reads wrong numbers or crashes with an EOFException.

5 See it happen

One record, packed and unpacked

Six steps: one record from map() to reduce(), first in the naive Writable, then in the compact one.

write() and readFields() 6 steps

What you see

Behind the screen

6 Real-world use

Custom types at work

1. Composite keys for daily aggregates

Telecom and payments jobs aggregate per (customer, day) or (merchant, hour). A composite WritableComparable keeps the key typed and sorts output by customer and then by time.

Transactions→key = (customer, day)→Daily totals

2. Statistics values for combiners

A value type holding sum, count, min and max lets one job compute several statistics with a correct combiner, instead of one job per statistic.

(1, amount, amount)→combine: add, max→avg, max per key

3. Intermediate data between chained jobs

When one job feeds another, output is often written as a SequenceFile of the same Writables, so the next job reads binary records without parsing text.

Job 1→SequenceFile of Writables→Job 2

4. Inside the ecosystem

Hive, HBase and Mahout define their own Writables (HBase's ImmutableBytesWritable, Hive's row Writables). Knowing the interface helps when reading their code or their errors.

Your job→Writable interface→Hive, HBase types

Simplified; organisations differ in tools and policies.

7 Run the code

Count the bytes of one record, three ways

Python's struct module can write the same bytes as Java's DataOutput. This code serializes one map output record the way each of our three jobs did, and also implements Hadoop's VInt encoding (WritableUtils.writeVLong).

import struct

def write_utf(s):                      # DataOutput.writeUTF: 2-byte length + UTF-8 bytes
    b = s.encode()
    return struct.pack(">H", len(b)) + b

def write_vint(i):                     # WritableUtils.writeVInt / writeVLong (for i >= 0)
    if -112 <= i <= 127:
        return bytes([i & 0xFF])
    data = i.to_bytes((i.bit_length() + 7) // 8, "big")
    return bytes([(-112 - len(data)) & 0xFF]) + data

def text(s):                           # Text: VInt length + UTF-8 bytes
    b = s.encode()
    return write_vint(len(b)) + b

student, day, amount = "S1413", "2026-09-10", 45
naive = write_utf(student) + write_utf(day) + struct.pack(">iqi", 1, amount, amount)
as_text = text(student + "\t" + day) + text(f"1,{amount},{amount}")
compact = write_vint(1413) + write_vint(253) + write_vint(1) + write_vint(amount) + write_vint(amount)

for name, b in [("Writable (UTF)", naive), ("Text", as_text), ("Writable (VInt)", compact)]:
    print(f"{name:16} {len(b):2} bytes  {b.hex(' ')}")

Writable (UTF) 35 bytes 00 05 53 31 34 31 33 00 0a 32 30 32 36 2d 30 39 2d 31 30 00 00 00 01 00 00 00 00 00 00 00 2d 00 00 00 2d Text 25 bytes 10 53 31 34 31 33 09 32 30 32 36 2d 30 39 2d 31 30 07 31 2c 34 35 2c 34 35 Writable (VInt) 8 bytes 8e 05 85 8f fd 01 2d 2d

return struct.pack(">H", len(b)) + bwriteUTF always spends 2 bytes on the length: "2026-09-10" costs 12 bytes.
naive = write_utf(student) + write_utf(day) + struct.pack(">iqi", 1, amount, amount)">iqi" = int, long, int, big-endian: 4 + 8 + 4 = 16 bytes, mostly zeros.
return bytes([(-112 - len(data)) & 0xFF]) + dataOne marker byte saying how many bytes follow, then only those bytes: 1413 → 8e 05 85.

Run it on your PC: three encodings of the same job

Use the Module 8 lab cluster (Lesson 8.1) and student_orders.csv from make_student_orders.py (Lesson 9.1). Copy the three Java files and writable_lab.sh into /tmp/mr9 in the namenode container, compile them into mr9.jar, and run the script. It takes about 2 minutes.

Composite key and value with writeUTF and fixed-size numbers

// StudentDay.java: a composite WritableComparable key (student, day) and a custom Writable value.
// Job: for every student and day, the number of orders, total spend and the biggest order.
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class StudentDay {

  /** Key: (student, day). Sorted by student, then day. */
  public static class StudentDayKey implements WritableComparable<StudentDayKey> {
    private String student = "";
    private String day = "";
    public StudentDayKey() { }                                   // Hadoop needs a no-argument constructor
    public void set(String s, String d) { student = s; day = d; }
    @Override public void write(DataOutput out) throws IOException { out.writeUTF(student); out.writeUTF(day); }
    @Override public void readFields(DataInput in) throws IOException { student = in.readUTF(); day = in.readUTF(); }
    @Override public int compareTo(StudentDayKey o) {
      int c = student.compareTo(o.student);
      return c != 0 ? c : day.compareTo(o.day);
    }
    @Override public int hashCode() { return student.hashCode() * 31 + day.hashCode(); }   // used by HashPartitioner
    @Override public boolean equals(Object o) {
      return o instanceof StudentDayKey && compareTo((StudentDayKey) o) == 0;
    }
    @Override public String toString() { return student + "\t" + day; }
  }

  /** Value: orders, total and max amount. Small and fixed size: 3 numbers. */
  public static class SpendWritable implements Writable {
    private int orders; private long total; private int max;
    public SpendWritable() { }
    public void set(int o, long t, int m) { orders = o; total = t; max = m; }
    @Override public void write(DataOutput out) throws IOException { out.writeInt(orders); out.writeLong(total); out.writeInt(max); }
    @Override public void readFields(DataInput in) throws IOException { orders = in.readInt(); total = in.readLong(); max = in.readInt(); }
    @Override public String toString() { return orders + "\t" + total + "\t" + max; }
  }

  public static class M extends Mapper<LongWritable, Text, StudentDayKey, SpendWritable> {
    private final StudentDayKey key = new StudentDayKey();
    private final SpendWritable val = new SpendWritable();
    @Override
    protected void map(LongWritable k, Text line, Context ctx) throws IOException, InterruptedException {
      String[] f = line.toString().split(",");        // order_id,student_id,time,item,amount
      int amount = Integer.parseInt(f[4]);
      key.set(f[1], f[2].substring(0, 10));          // "2026-09-20 08:36" -> "2026-09-20"
      val.set(1, amount, amount);
      ctx.write(key, val);
    }
  }

  public static class R extends Reducer<StudentDayKey, SpendWritable, StudentDayKey, SpendWritable> {
    private final SpendWritable out = new SpendWritable();
    @Override
    protected void reduce(StudentDayKey k, Iterable<SpendWritable> vals, Context ctx) throws IOException, InterruptedException {
      int orders = 0, max = 0; long total = 0;
      for (SpendWritable v : vals) { orders += v.orders; total += v.total; max = Math.max(max, v.max); }
      out.set(orders, total, max);
      ctx.write(k, out);
    }
  }

  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    String[] rest = new GenericOptionsParser(conf, args).getRemainingArgs();
    Job job = Job.getInstance(conf, "student day");
    job.setJarByClass(StudentDay.class);
    job.setMapperClass(M.class);
    job.setCombinerClass(R.class);                    // orders, total and max are all safe to combine
    job.setReducerClass(R.class);
    job.setOutputKeyClass(StudentDayKey.class);
    job.setOutputValueClass(SpendWritable.class);
    FileInputFormat.addInputPath(job, new Path(rest[0]));
    FileOutputFormat.setOutputPath(job, new Path(rest[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

// Inside the namenode container, in /tmp/mr9:
//   mkdir -p classes && javac -cp $(hadoop classpath) -d classes *.java && jar cf mr9.jar -C classes .
//   hadoop jar mr9.jar StudentDay /user/hadoop/data/student_orders.csv /out/StudentDay

Map output records=500000 Map output bytes=17500000 Map output materialized bytes=5604593 Combine output records=151475 Reduce output records=59946

Real counters from the run in the next item. 35 bytes per record.

The same job with Text

// StudentDayText.java: the same job as StudentDay, but key and value travel as Text strings.
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class StudentDayText {
  public static class M extends Mapper<LongWritable, Text, Text, Text> {
    private final Text key = new Text(), val = new Text();
    @Override
    protected void map(LongWritable k, Text line, Context ctx) throws IOException, InterruptedException {
      String[] f = line.toString().split(",");
      key.set(f[1] + "\t" + f[2].substring(0, 10));
      val.set("1," + f[4] + "," + f[4]);
      ctx.write(key, val);
    }
  }
  public static class R extends Reducer<Text, Text, Text, Text> {
    @Override
    protected void reduce(Text k, Iterable<Text> vals, Context ctx) throws IOException, InterruptedException {
      int orders = 0, max = 0; long total = 0;
      for (Text v : vals) {
        String[] p = v.toString().split(",");
        orders += Integer.parseInt(p[0]); total += Long.parseLong(p[1]); max = Math.max(max, Integer.parseInt(p[2]));
      }
      ctx.write(k, new Text(orders + "," + total + "," + max));
    }
  }
  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    String[] rest = new GenericOptionsParser(conf, args).getRemainingArgs();
    Job job = Job.getInstance(conf, "student day text");
    job.setJarByClass(StudentDayText.class);
    job.setMapperClass(M.class);
    job.setCombinerClass(R.class);
    job.setReducerClass(R.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(Text.class);
    FileInputFormat.addInputPath(job, new Path(rest[0]));
    FileOutputFormat.setOutputPath(job, new Path(rest[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

Map output records=500000 Map output bytes=12500000 Map output materialized bytes=4132541 Combine output records=151475 Reduce output records=59946

25 bytes per record: the strings are short and every number here has two digits.

Compact encoding with VInt and VLong

// StudentDayCompact.java: the same job as StudentDay, with compact variable-length encodings:
// student "S0001" stored as the number 1, the day as a day-of-year number, every number as a VInt/VLong.
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.io.WritableUtils;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class StudentDayCompact {

  /** Key: (student, day). Sorted by student, then day. */
  public static class StudentDayKey implements WritableComparable<StudentDayKey> {
    private int student;                                         // "S0001" -> 1
    private int day;                                             // day of the year, 2026-09-01 -> 244
    public StudentDayKey() { }
    public void set(String s, String d) { student = Integer.parseInt(s.substring(1)); day = java.time.LocalDate.parse(d).getDayOfYear(); }
    @Override public void write(DataOutput out) throws IOException { WritableUtils.writeVInt(out, student); WritableUtils.writeVInt(out, day); }
    @Override public void readFields(DataInput in) throws IOException { student = WritableUtils.readVInt(in); day = WritableUtils.readVInt(in); }
    @Override public int compareTo(StudentDayKey o) {
      int c = Integer.compare(student, o.student);
      return c != 0 ? c : Integer.compare(day, o.day);
    }
    @Override public int hashCode() { return student * 31 + day; }
    @Override public boolean equals(Object o) {
      return o instanceof StudentDayKey && compareTo((StudentDayKey) o) == 0;
    }
    @Override public String toString() { return String.format("S%04d\t%s", student, java.time.LocalDate.ofYearDay(2026, day)); }
  }

  /** Value: orders, total and max amount, each written in as few bytes as the number needs. */
  public static class SpendWritable implements Writable {
    private int orders; private long total; private int max;
    public SpendWritable() { }
    public void set(int o, long t, int m) { orders = o; total = t; max = m; }
    @Override public void write(DataOutput out) throws IOException { WritableUtils.writeVInt(out, orders); WritableUtils.writeVLong(out, total); WritableUtils.writeVInt(out, max); }
    @Override public void readFields(DataInput in) throws IOException { orders = WritableUtils.readVInt(in); total = WritableUtils.readVLong(in); max = WritableUtils.readVInt(in); }
    @Override public String toString() { return orders + "\t" + total + "\t" + max; }
  }

  public static class M extends Mapper<LongWritable, Text, StudentDayKey, SpendWritable> {
    private final StudentDayKey key = new StudentDayKey();
    private final SpendWritable val = new SpendWritable();
    @Override
    protected void map(LongWritable k, Text line, Context ctx) throws IOException, InterruptedException {
      String[] f = line.toString().split(",");        // order_id,student_id,time,item,amount
      int amount = Integer.parseInt(f[4]);
      key.set(f[1], f[2].substring(0, 10));          // "2026-09-20 08:36" -> "2026-09-20"
      val.set(1, amount, amount);
      ctx.write(key, val);
    }
  }

  public static class R extends Reducer<StudentDayKey, SpendWritable, StudentDayKey, SpendWritable> {
    private final SpendWritable out = new SpendWritable();
    @Override
    protected void reduce(StudentDayKey k, Iterable<SpendWritable> vals, Context ctx) throws IOException, InterruptedException {
      int orders = 0, max = 0; long total = 0;
      for (SpendWritable v : vals) { orders += v.orders; total += v.total; max = Math.max(max, v.max); }
      out.set(orders, total, max);
      ctx.write(k, out);
    }
  }

  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    String[] rest = new GenericOptionsParser(conf, args).getRemainingArgs();
    Job job = Job.getInstance(conf, "student day compact");
    job.setJarByClass(StudentDayCompact.class);
    job.setMapperClass(M.class);
    job.setCombinerClass(R.class);                    // orders, total and max are all safe to combine
    job.setReducerClass(R.class);
    job.setOutputKeyClass(StudentDayKey.class);
    job.setOutputValueClass(SpendWritable.class);
    FileInputFormat.addInputPath(job, new Path(rest[0]));
    FileOutputFormat.setOutputPath(job, new Path(rest[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

Map output records=500000 Map output bytes=4068724 Map output materialized bytes=1601326 Combine output records=151475 Reduce output records=59946

About 8 bytes per record. toString() prints the student and date back in their usual form.

Run all three, compare, then break the constructor

# writable_lab.sh: a custom key and value vs the same job with Text; then forget the no-arg constructor
cd /tmp/mr9
D=/user/hadoop/data/student_orders.csv
for job in StudentDay StudentDayText StudentDayCompact; do
  hdfs dfs -rm -r -f -skipTrash /out/$job > /dev/null 2>&1
  start=$(date +%s%N)
  hadoop jar mr9.jar $job $D /out/$job > /tmp/$job.log 2>&1
  echo "== $job   $(( ($(date +%s%N) - start) / 1000000 )) ms"
  grep -E "Map output records|Map output bytes|Map output materialized bytes|Combine output records|Reduce output records|CPU time spent" /tmp/$job.log | sed 's/^\s*/  /'
done
echo "-- output (custom Writable), first lines and student S0001"
hdfs dfs -cat /out/StudentDay/part-r-00000 2>/dev/null | head -3
hdfs dfs -cat /out/StudentDay/part-r-00000 | grep -P "^S0001\t" 2>/dev/null | head -3
echo "-- the Text and Compact versions agree?"
diff <(hdfs dfs -cat /out/StudentDay/part-r-00000) <(hdfs dfs -cat /out/StudentDayCompact/part-r-00000) > /dev/null && echo "compact: same rows" || echo "compact: rows differ"
diff <(hdfs dfs -cat /out/StudentDay/part-r-00000 | tr '\t' ',' | sed -E 's/^([^,]*),([^,]*),/\1\t\2\t/') <(hdfs dfs -cat /out/StudentDayText/part-r-00000) > /dev/null && echo "text: same rows" || echo "text: rows differ"
echo "== remove the no-argument constructor of the key"
sed -e 's/StudentDay\b/StudentDayNoCtor/g' -e 's/public StudentDayKey() { }.*/public StudentDayKey(String s, String d) { set(s, d); }/' \
    -e 's/new StudentDayKey()/new StudentDayKey("", "")/' StudentDay.java > StudentDayNoCtor.java
mkdir -p nc && javac -cp $(hadoop classpath) -d nc StudentDayNoCtor.java && jar cf nc.jar -C nc .
hdfs dfs -rm -r -f -skipTrash /out/nc > /dev/null 2>&1
hadoop jar nc.jar StudentDayNoCtor $D /out/nc 2>&1 | grep -E "Error|Caused by|NoSuchMethod|failed with state" | sed -E 's/^[0-9-]+ [0-9:]+ INFO  //' | cut -c1-170 | sort -u | head -5

# Inside the namenode container:  bash /tmp/mr9/writable_lab.sh

== StudentDay 24815 ms Map output records=500000 Map output bytes=17500000 Map output materialized bytes=5604593 Combine output records=151475 Reduce output records=59946 CPU time spent (ms)=8360 == StudentDayText 24363 ms Map output records=500000 Map output bytes=12500000 Map output materialized bytes=4132541 Combine output records=151475 Reduce output records=59946 CPU time spent (ms)=7290 == StudentDayCompact 25126 ms Map output records=500000 Map output bytes=4068724 Map output materialized bytes=1601326 Combine output records=151475 Reduce output records=59946 CPU time spent (ms)=8600 -- output (custom Writable), first lines and student S0001 S0001 2026-09-01 124 3224 80 S0001 2026-09-02 134 3411 80 S0001 2026-09-03 124 3064 80 S0001 2026-09-01 124 3224 80 S0001 2026-09-02 134 3411 80 S0001 2026-09-03 124 3064 80 -- the Text and Compact versions agree? compact: same rows text: same rows == remove the no-argument constructor of the key Caused by: java.lang.NoSuchMethodException: StudentDayNoCtor$StudentDayKey.<init>() Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException: StudentDayNoCtor$StudentDayKey.<init>() Error: java.io.IOException: Initialization of all the collectors failed. Error in last collector was:java.lang.RuntimeException: java.lang.NoSuchMethodException: StudentD Job:1767 - Job job_1792643020992_0034 failed with state FAILED due to: Task failed task_1792643020992_0034_m_000000

Real output. The output lines for S0001 appear twice because the first rows of the file are S0001's. The constructor test is done with sed on a copy, so StudentDay.java is unchanged.

8 Line by line

write() then readFields() into a reused object

Follow one record through write and readFields. Notice that reading fills an existing object instead of creating a new one.

Worked examples: real problems, solved and traced in memory

Three design problems: sizing a record, choosing an encoding, and a key that partitions wrongly.

Example 1BeginnerSizing · bytes

How big is this record?

Problem statement

A key holds a city name written with writeUTF ("Mumbai") and a year as writeInt. The value is a writeLong total. How many bytes per record, and how many MB for 2 crore records?

Input

writeUTF("Mumbai") + int + long

Expected output

20 bytes per record, 381 MB

Approach

writeUTF = 2 + number of bytes.
int = 4, long = 8.
Multiply by the number of records.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 2IntermediateDesign · encoding

Pick an encoding for a phone-number key

Problem statement

A telecom job keys records by a 10-digit mobile number. Compare storing it as Text, as writeLong, and as a VLong.

Input

9876543210

Expected output

Text 11 · long 8 · VLong 6 bytes

Approach

Text: VInt length + 10 characters.
writeLong: always 8 bytes.
VLong: marker + the bytes the number needs.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 3AdvancedCorrectness · partitioning

A key whose hashCode breaks the job

Problem statement

A developer writes a composite key but forgets to override hashCode(), so Java's default (based on the object's identity) is used. Equal keys from two map tasks get different hash codes. What happens to the reducers' output?

Input

key (S0001, 2026-09-01) from two maps

Expected output

map 0 sends ('S0001', '2026-09-01') to reducer 3
map 1 sends ('S0001', '2026-09-01') to reducer 2
hash from the fields -> reducer 2 for both maps

Approach

HashPartitioner uses hashCode() % reducers.
Identity hash codes differ between objects and JVMs.
Simulate two maps sending the same key.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

9 Quiz and revision

Check yourself

Quick revision

Lesson 9.3 in one screen

Writable: write(DataOutput) + readFields(DataInput), same fields, same order.
Keys: WritableComparable (compareTo = sort order) and a consistent hashCode/equals (partitioning).
Public no-arg constructor required (reflection); readFields must overwrite every field (objects are reused).
Sizes: writeUTF 2 + bytes; int 4; long 8; VInt 1–5, VLong 1–9; Text = VInt length + bytes; NullWritable 0.
Lab: 35 B (writeUTF + fixed numbers) vs 25 B (Text) vs ≈8 B (VInt) per record; 17.5 / 12.5 / 4.1 MB for 5 lakh records; same 59,946 rows.
toString() controls how the type appears in TextOutputFormat output.
Faster sorting: a RawComparator compares serialized bytes. Avro/Parquet for data files shared with other tools.

Interview and viva questions

What is a Writable in Hadoop?

Writable is Hadoop's serialization interface. A class implements write(DataOutput) to write its fields as bytes and readFields(DataInput) to read them back. Hadoop uses Writables for all MapReduce keys and values and in SequenceFiles. Common built-in Writables are Text, IntWritable, LongWritable, DoubleWritable and NullWritable.

What is the difference between Writable and WritableComparable?

Writable only serializes and deserializes. WritableComparable also implements Comparable, so objects can be sorted. MapReduce sorts map output by key, so every key type must be WritableComparable; value types only need Writable.

Why does Hadoop not use Java serialization?

Java serialization writes class information and object metadata with every object and creates a new object when reading, which is slow and large for billions of small records. Writables write only the field values, can be reused for every record, and many can be compared directly in serialized form with a RawComparator.

How do you create a custom key in MapReduce?

Write a class that implements WritableComparable<YourKey>, with a public no-argument constructor, write() and readFields() handling the same fields in the same order, compareTo() defining the sort order, and hashCode() and equals() based on the same fields so the partitioner sends equal keys to the same reducer. Override toString() for readable text output.

Why does my custom Writable job fail with NoSuchMethodException <init>()?

Hadoop creates key and value objects by reflection with the no-argument constructor. If your class only has a constructor with parameters, Java does not generate the default one, and tasks fail at runtime with NoSuchMethodException: YourClass.<init>(). Add a public no-argument constructor (and make nested classes static).

Practice problems

Writable exercises using the sizes and counters measured in the lab.

P1Easy

Bytes per record

Compute bytes per record for the three jobs from Map output bytes and Map output records.

Input

17,500,000 · 12,500,000 · 4,068,724 / 500,000

Expected output

UTF 35.0
Text 25.0
VInt 8.14

Show solution and explanation

for name, b in [("UTF", 17_500_000), ("Text", 12_500_000), ("VInt", 4_068_724)]:
    print(name, round(b / 500_000, 2))

UTF 35.0 Text 25.0 VInt 8.14

How it works: 35, 25 and 8.14 bytes.

P2Easy

How much smaller?

How many times smaller is the compact Writable than the first Writable, and than Text?

Input

three byte counts

Expected output

4.3 3.1

Show solution and explanation

print(round(17_500_000 / 4_068_724, 1), round(12_500_000 / 4_068_724, 1))

4.3 3.1

How it works: 4.3× smaller than the naive Writable and 3.1× smaller than Text.

P3Easy

writeUTF cost

How many bytes does writeUTF use for "S0001" and for "2026-09-01"?

Input

two strings

Expected output

S0001 7
2026-09-01 12

Show solution and explanation

for s in ["S0001", "2026-09-01"]:
    print(s, 2 + len(s.encode()))

S0001 7 2026-09-01 12

How it works: 7 and 12: 19 bytes of key in the naive version.

P4Medium

Materialized vs raw

Map output materialized bytes are what is actually written to disk for the shuffle (with partition framing and checksums). Compute the ratio materialized / raw for each job. Which shrinks most?

Input

materialized and raw bytes

Expected output

UTF 0.32
Text 0.33
VInt 0.39

Show solution and explanation

jobs = {"UTF": (5_604_593, 17_500_000), "Text": (4_132_541, 12_500_000), "VInt": (1_601_326, 4_068_724)}
for k, (m, r) in jobs.items():
    print(k, round(m / r, 2))

UTF 0.32 Text 0.33 VInt 0.39

How it works: About 0.32, 0.33 and 0.39: the combiner shrank all three record counts from 500,000 to 151,475 before writing; the compact records had less redundancy to begin with.

P5Medium

VInt encoder

Write Hadoop's VInt encoding for non-negative numbers and print the hex for 45, 253, 1413 and 2000.

Input

4 numbers

Expected output

45 2d
253 8f fd
1413 8e 05 85
2000 8e 07 d0

Show solution and explanation

def vint(i):
    if i <= 127:
        return bytes([i])
    data = i.to_bytes((i.bit_length() + 7) // 8, "big")
    return bytes([(-112 - len(data)) & 0xFF]) + data
for n in [45, 253, 1413, 2000]:
    print(n, vint(n).hex(" "))

45 2d 253 8f fd 1413 8e 05 85 2000 8e 07 d0

How it works: 2d · 8f fd · 8e 05 85 · 8e 07 d0. The marker byte 8f means one data byte follows, 8e means two.

P6Medium

Composite key order

Sort these (student, day) keys the way compareTo does, and say which reducer call comes first.

Input

4 keys

Expected output

[('S0002', '2026-09-01'), ('S0002', '2026-09-30'), ('S0010', '2026-09-01'), ('S0100', '2026-09-05')]

Show solution and explanation

keys = [("S0010", "2026-09-01"), ("S0002", "2026-09-30"), ("S0002", "2026-09-01"), ("S0100", "2026-09-05")]
print(sorted(keys))

[('S0002', '2026-09-01'), ('S0002', '2026-09-30'), ('S0010', '2026-09-01'), ('S0100', '2026-09-05')]

How it works: S0002 first (both days in order), then S0010, then S0100: string order works because the IDs are zero-padded.

P7Business problem

Shuffle savings per year

A nightly job shuffles 800 GB with Text records. A compact Writable would cut the bytes by 3×. The network and disk cost is ₹2 per GB shuffled (illustrative). What is saved in a year?

Input

800 GB/night · 3× smaller · ₹2/GB

Expected output

Rs 389,333 per year

Show solution and explanation

saved_gb = 800 - 800 / 3
print(f"Rs {saved_gb * 2 * 365:,.0f} per year")

Rs 389,333 per year

How it works: About ₹3.9 lakh a year, plus faster jobs. Worth a day of a developer's time; not worth it for a 1 GB job.

P8Advanced

Zero-padding pitfall

If student IDs were not zero-padded (S1, S2, S10), a String compareTo sorts them as text. Show the order, then fix it by comparing the numbers.

Input

S1, S2, S10, S9

Expected output

as text:   ['S1', 'S10', 'S2', 'S9']
as number: ['S1', 'S2', 'S9', 'S10']

Show solution and explanation

ids = ["S1", "S2", "S10", "S9"]
print("as text:  ", sorted(ids))
print("as number:", sorted(ids, key=lambda s: int(s[1:])))

as text: ['S1', 'S10', 'S2', 'S9'] as number: ['S1', 'S2', 'S9', 'S10']

How it works: Text order gives S1, S10, S2, S9. Storing the number (as the compact key does) sorts correctly and is smaller.

Real-world scenario questions (asked in company interviews)

Writable questions test whether you understand what really travels between map and reduce.

Concepts · basicsFresher

“Why does Hadoop have its own IntWritable instead of using Integer?”

A common opener in Hadoop interviews.

The interviewer is checking: serialization basics.

Show a strong answer

Keys and values are sent between JVMs and written to disk, so they must be serialized.
Writable writes only the field bytes (4 for an int), with no class metadata, and objects are reused for every record.
Keys are WritableComparable so they can be sorted, often byte by byte with a RawComparator.

Debugging · runtimeFresher

The job runs in your IDE tests but fails on the cluster with NoSuchMethodException

The message ends with MyKey.<init>().

The interviewer is checking: reflection and constructors.

Show a strong answer

Hadoop instantiates keys and values by reflection with a no-argument constructor.
Adding a constructor with parameters removed Java's default one; add a public no-arg constructor.
For nested classes, make them static, or reflection cannot create them either.

Performance · shuffleExperienced

A job with a custom Writable shuffles more data than the old Text version

The team expected the opposite.

The interviewer is checking: encoding choices.

Show a strong answer

Compare Map output bytes / Map output records for both versions to get bytes per record.
Look at write(): writeUTF adds 2 bytes per string, writeInt/writeLong use 4/8 bytes even for tiny numbers.
Encode IDs and dates as numbers, use VInt/VLong, drop fields the reducer doesn't need; in our lab 35 → 8 bytes per record.

Correctness · keysExperienced

Output has duplicate keys across part files

The same (customer, day) appears in part-r-00001 and part-r-00003 with different totals.

The interviewer is checking: hashCode and equals contracts.

Show a strong answer

The key's hashCode() is not based on its fields (or is inconsistent with equals/compareTo), so equal keys go to different reducers.
Implement hashCode() from the same fields compareTo() uses; avoid Object.hashCode() and anything JVM-specific.
Rerun the job: each key then appears exactly once.

Custom Writable types

The “efficient” version that was bigger

Writable and WritableComparable

Words you'll meet in this lesson

From our lab (Hadoop 3.5.0, 5 lakh orders → 59,946 student-days)

Writables at three levels

Beginner Packing a parcel

Intermediate What Hadoop needs from your class

Advanced Making records small and sorting fast

Built-in Writables and their sizes

Ways to serialize records in Hadoop

Text with separators

Custom Writable

Java Serializable

Avro (and Parquet)

A composite key, a value type, and three sizes

The key: (student, day)

The value: orders, total, max

Run the Writable version and the Text version

Shrink it: numbers instead of strings, VInts instead of fixed sizes

Check that all three agree

Forget the no-argument constructor

One record, packed and unpacked

Custom types at work

1. Composite keys for daily aggregates

2. Statistics values for combiners

3. Intermediate data between chained jobs

4. Inside the ecosystem

Count the bytes of one record, three ways

Run it on your PC: three encodings of the same job

Composite key and value with writeUTF and fixed-size numbers

The same job with Text

Compact encoding with VInt and VLong

Run all three, compare, then break the constructor

write() then readFields() into a reused object

Worked examples: real problems, solved and traced in memory

How big is this record?

Pick an encoding for a phone-number key

A key whose hashCode breaks the job

Check yourself

Quick revision

Lesson 9.3 in one screen

Interview and viva questions

Practice problems

Bytes per record

How much smaller?

writeUTF cost

Materialized vs raw

VInt encoder

Composite key order

Shuffle savings per year

Zero-padding pitfall

Real-world scenario questions (asked in company interviews)

“Why does Hadoop have its own IntWritable instead of using Integer?”

The job runs in your IDE tests but fails on the cluster with NoSuchMethodException

A job with a custom Writable shuffles more data than the old Text version

Output has duplicate keys across part files