How Hadoop solved each problem

Riya's problem list, ticked off one by one

Hadoop's architecture as a set of answers

Words you'll meet in this lesson

Problem → solution map

The architecture at three levels

The four core modules, side by side

Hadoop's strengths and limits

A word count, and which Hadoop part handles each step

1. Store: put the file in HDFS

   hdfs dfs -put pages.txt /data/pages.txt

   HDFS splits the file into 128 MB blocks and writes 3 replicas of each to DataNodes on two racks. Solves: doesn't fit and disks die.

2. Map: each task reads one block

   def map_words(line):
       return [(w, 1) for w in line.lower().split()]
   print(map_words("Pune Delhi Pune"))

   [('pune', 1), ('delhi', 1), ('pune', 1)]
   Your only code. MapReduce runs one copy per block, in parallel, on nodes that hold the blocks (locality). Solves: too slow and network jammed.

3. Shuffle and sort: group by key

   pairs = [("pune", 1), ("delhi", 1), ("pune", 1)]
   groups = {}
   for k, v in sorted(pairs):
       groups.setdefault(k, []).append(v)
   print(groups)

   {'delhi': [1], 'pune': [1, 1]}
   Done entirely by the framework: map outputs are partitioned by key, copied over the network to reducers and sorted. Solves: too much plumbing.

4. Reduce: combine each key's values

   def reduce_count(word, ones):
       return word, sum(ones)
   print(reduce_count("pune", [1, 1]))

   ('pune', 2)
   Your second function. Reducers write results back to HDFS, replicated like any other file.

5. Recover: a node dies halfway

   A DataNode with three running map tasks loses power.

   YARN and the ApplicationMaster re-run those tasks on other nodes, reading other replicas. The job finishes. Solves: tasks fail mid-job.

Six problems, six fixes: the board and the cluster

Each frame takes one line from Riya's board. Left: the problem as a user feels it. Right: the Hadoop mechanism that fixes it.

Try it yourself: test every fix

This cluster has everything switched on, including High Availability for the NameNode. Work through the checklist to see each fix with your own eyes.

Which fix matters most, by industry

Examples are simplified descriptions of common patterns.

A tiny MapReduce, in 20 lines of Python

This is the whole MapReduce idea on one machine. map_fn and reduce_fn are the only parts you would write in Hadoop; run_job plays the framework: it splits the input, runs every map, shuffles and sorts by key, and runs every reduce.

def map_fn(line):                        # YOU write this
    city, amount = line.split(",")
    return [(city, int(amount))]

def reduce_fn(city, amounts):            # YOU write this
    return city, sum(amounts)

def run_job(lines, splits=2):            # HADOOP does this part
    parts = [lines[i::splits] for i in range(splits)]
    mapped = [kv for part in parts for line in part for kv in map_fn(line)]
    groups = {}
    for k, v in sorted(mapped):          # shuffle and sort by key
        groups.setdefault(k, []).append(v)
    return [reduce_fn(k, vs) for k, vs in groups.items()]

orders = ["Pune,240", "Delhi,120", "Pune,60", "Goa,300", "Delhi,80"]
print(run_job(orders))

• def map_fn(line):Map: turn one input record into zero or more (key, value) pairs. Here: (city, amount).
• parts = [lines[i::splits] for i in range(splits)]Split the input into pieces, like HDFS blocks. lines[i::2] takes every second line starting at i.
• for k, v in sorted(mapped):Shuffle and sort: all values with the same key end up together, in key order.
• return [reduce_fn(k, vs) for k, vs in groups.items()]Reduce: called once per key with the list of all its values.

Map, shuffle and reduce, one line at a time

Follow three orders through map (records → pairs), shuffle (pairs grouped by key) and reduce (one total per city). Watch the groups dictionary build up.

Worked examples: real problems, solved and traced in memory

Three problems that show each layer of Hadoop doing its job: a MapReduce on business data, the storage cost of replication versus erasure coding, and the gain from YARN sharing one cluster. Each runs live and is traced in memory.

["Kharghar,chai,20", "Vashi,samosa,15", "Kharghar,samosa,15", "Vashi,chai,20", "Kharghar,chai,20"]

('Kharghar', 55)
('Vashi', 35)

600 TB total · 60 TB hot · replication 3× · erasure coding 1.5×

All 3x replication: 1800 TB raw
Hot 3x + cold EC   : 990.0 TB raw
Saving: 810 TB (45%)

3 teams × 20 nodes · 90% busy 6 h, 10% busy 18 h · peaks at different times

Separate clusters: 30% average utilisation, 60 nodes
Shared YARN cluster: about 22 nodes busy at worst
With 25% spare: 28 nodes instead of 60

Check yourself

Quick revision

Interview and viva questions

Practice problems

Practise the MapReduce way of thinking (map → shuffle → reduce) and the arithmetic behind Hadoop's storage choices.

"Pune,UPI,450"

('UPI', 450)

def map_fn(line):
    city, mode, amount = line.split(",")
    return (mode, int(amount))
print(map_fn("Pune,UPI,450"))

[("a",1),("b",2),("a",3)]

{'a': [1, 3], 'b': [2]}

pairs = [("a", 1), ("b", 2), ("a", 3)]
groups = {}
for k, v in pairs:
    groups.setdefault(k, []).append(v)
print(groups)

250 TB

Replication: 750 TB | EC: 375.0 TB

print("Replication:", 250 * 3, "TB | EC:", 250 * 1.5, "TB")

["Pune,240", "Delhi,120", "Pune,60", "Delhi,80", "Goa,300"]

Delhi 100.0
Goa 300.0
Pune 150.0

orders = ["Pune,240", "Delhi,120", "Pune,60", "Delhi,80", "Goa,300"]
groups = {}
for city, amt in sorted((o.split(",")[0], int(o.split(",")[1])) for o in orders):
    groups.setdefault(city, []).append(amt)
for city, amts in groups.items():
    print(city, sum(amts) / len(amts))

["Hadoop stores data", "Spark caches data in memory"]

{'caches': 1, 'hadoop': 1, 'memory': 1, 'spark': 1, 'stores': 1}

lines = ["Hadoop stores data", "Spark caches data in memory"]
pairs = [(w.lower(), 1) for l in lines for w in l.split() if len(w) > 4]
counts = {}
for w, one in sorted(pairs):
    counts[w] = counts.get(w, 0) + one
print(counts)

scan 2 years of logs · show one user profile in 50 ms · nightly sales report · live fraud alert in 1 s

scan 2 years of logs            : yes
show one user profile in 50 ms  : no
nightly sales report            : yes
live fraud alert in 1 s         : no

workloads = {"scan 2 years of logs": "batch", "show one user profile in 50 ms": "lookup",
             "nightly sales report": "batch", "live fraud alert in 1 s": "stream"}
for w, kind in workloads.items():
    print(f"{w:32}: {'yes' if kind == 'batch' else 'no'}")

40 crore files × 200 KB · 150 B per object

Small files : ~112 GB of NameNode heap
Packed files: ~0.09 GB

files = 40_00_00_000
small_gb = files * 2 * 150 / 1024**3
data_mb = files * 200 / 1024
big_files = data_mb / 256
big_gb = big_files * 2 * 150 / 1024**3
print(f"Small files : ~{small_gb:.0f} GB of NameNode heap")
print(f"Packed files: ~{big_gb:.2f} GB")

900 TB · 3× → 1.5× · Rs 2,500/TB

Saved 1350 TB raw = Rs 33.8 lakh

saved_tb = 900 * (3 - 1.5)
print(f"Saved {saved_tb:.0f} TB raw = Rs {saved_tb * 2500 / 1e5:.1f} lakh")

Real-world scenario questions (asked in company interviews)

Interviewers often test architecture knowledge by giving you a problem and asking which part of Hadoop handles it.