Hadoop architecture: HDFS, YARN, MapReduce and Common

“What is actually running on these machines?”

Hadoop's architecture, daemon by daemon

Words you'll meet in this lesson

Every daemon, where it runs, and its default ports (Hadoop 3)

These are Apache Hadoop 3 defaults. Distributions and companies often change them; always check the cluster's configuration files.

The architecture at three levels

The four modules, their daemons and their configuration files

The most important settings in each file

Bringing a cluster up, daemon by daemon

1. Every node gets the same configuration

   import xml.etree.ElementTree as ET
   xml = """<configuration><property><name>fs.defaultFS</name>
   <value>hdfs://nn1:8020</value></property></configuration>"""
   for p in ET.fromstring(xml).iter("property"):
       print(p.findtext("name"), "=", p.findtext("value"))

   fs.defaultFS = hdfs://nn1:8020
   Every Hadoop configuration file is a list of <property> name–value pairs. Every daemon and client reads them at start-up, so they all agree on where the masters are.

2. Start HDFS: the NameNode first

   The admin runs start-dfs.sh. The NameNode loads its saved metadata (fsimage + edit log) and waits in safe mode.

   The NameNode knows the files and their blocks, but not yet where the blocks are. That information is never saved; it comes fresh from the DataNodes.

3. DataNodes start and report

   On each worker a DataNode starts, registers with the NameNode and sends a block report.

   When enough blocks have been reported (99.9% by default), the NameNode leaves safe mode and allows writes. Module 5 explains safe mode in detail.

4. Start YARN

   The admin runs start-yarn.sh. The ResourceManager starts; each NodeManager registers with how much memory and how many vcores it offers.

   The RM now knows total cluster capacity, for example 40 nodes × 192 GB. Its web UI on port 8088 shows it.

5. Start the JobHistory Server, then check

   mapred --daemon start historyserver
   jps

   On a single-machine (pseudo-distributed) test cluster, jps should list NameNode, DataNode, SecondaryNameNode, ResourceManager, NodeManager and JobHistoryServer. If one is missing, its log file says why. You'll do this yourself in Module 4.

A cluster starting up: what the admin sees, and which daemons appear

Follow a small cluster (one master, three workers) from power-on to ready. Left: the admin's terminal. Right: the daemons on each machine.

Try it yourself: find each daemon in the playground

The playground shows the same architecture: masters at the top, DataNodes (with their NodeManager containers) below.

Architecture knowledge in daily work

Examples are simplified. Port numbers are Apache defaults; distributions may differ.

Read a Hadoop configuration file

Every Hadoop setting lives in XML files like this hdfs-site.xml. This program reads it the way Hadoop does and turns it into a dictionary. Then try the challenges: find the NameNode, plan which daemons go on which host, and catch a configuration mistake.

import xml.etree.ElementTree as ET

hdfs_site = """<configuration>
  <property><name>dfs.replication</name><value>3</value></property>
  <property><name>dfs.namenode.name.dir</name><value>/data/nn</value></property>
  <property><name>dfs.datanode.data.dir</name><value>/data/1/dn,/data/2/dn</value></property>
  <property><name>dfs.blocksize</name><value>134217728</value></property>
</configuration>"""

conf = {p.findtext("name"): p.findtext("value")
        for p in ET.fromstring(hdfs_site).iter("property")}

print("Replication :", conf["dfs.replication"])
print("Block size  :", int(conf["dfs.blocksize"]) // 1024 // 1024, "MB")
print("DataNode disks:", conf["dfs.datanode.data.dir"].split(","))

• ET.fromstring(hdfs_site).iter("property")Parse the XML and visit every <property> element, as Hadoop's Configuration class does.
• conf = {p.findtext("name"): p.findtext("value") ...}A dictionary comprehension: property name → value. All values are strings.
• int(conf["dfs.blocksize"]) // 1024 // 1024134217728 bytes = 128 MB, the Hadoop 2/3 default.

Merging configuration files, one line at a time

Hadoop loads default settings first and then the site files, so later files override earlier ones. Watch the conf dictionary change as each layer is applied.

Worked examples: real problems, solved and traced in memory

Three tasks a new platform engineer is given in the first week: audit which daemons run where, check a cluster's port plan, and size the master's memory. Each runs live and is traced in memory.

master1: NameNode, ResourceManager · w1: DataNode, NodeManager · w2: DataNode · w3: NodeManager

w2 missing ['NodeManager']
w3 missing ['DataNode']

master daemons: NameNode, ResourceManager · worker daemons: DataNode, NodeManager

master: service [8020, 8030, 8031, 8032] | web UI [8088, 9870]
worker: service [9866, 9867, 13562] | web UI [8042, 9864]

40 M files × 1.5 blocks · 3 M directories · ~1 GB per million objects · +30%

Objects: 103 million
Heap with headroom: 134 GB -> set 136 GB

Check yourself

Quick revision

Interview and viva questions

Practice problems

Small programs about daemons, ports and configuration: the things you check every day on a real cluster.

[9870, 8088, 19888]

9870 -> NameNode web UI
8088 -> ResourceManager web UI
19888 -> JobHistory web UI

service = {9870: "NameNode web UI", 8088: "ResourceManager web UI", 19888: "JobHistory web UI",
           8020: "NameNode RPC", 9864: "DataNode web UI"}
for p in [9870, 8088, 19888]:
    print(p, "->", service[p])

2 × 3 + 40 × 2

86 daemons

print(2 * 3 + 40 * 2, "daemons")

["134217728", "268435456"]

134217728 = 128 MB
268435456 = 256 MB

for v in ["134217728", "268435456"]:
    print(v, "=", int(v) // (1024 * 1024), "MB")

["hdfs://nn1:8020", "hdfs://localhost:9000", "hdfs://mycluster"]

hdfs://nn1:8020 -> nn1 8020
hdfs://localhost:9000 -> localhost 9000
hdfs://mycluster -> mycluster 8020

from urllib.parse import urlparse
for v in ["hdfs://nn1:8020", "hdfs://localhost:9000", "hdfs://mycluster"]:
    u = urlparse(v)
    print(v, "->", u.hostname, u.port or 8020)

defaults {repl 3, block 128m} · site {repl 2} · job {block 256m}

{'dfs.replication': '2', 'dfs.blocksize': '256m'}

layers = [{"dfs.replication": "3", "dfs.blocksize": "128m"},
          {"dfs.replication": "2"},
          {"dfs.blocksize": "256m"}]
conf = {}
for layer in layers:
    conf.update(layer)
print(conf)

{dn1: 2, dn2: 700, dn3: 1, dn4: 640, dn5: 3}

Live: 3 | Dead: 2 ['dn2', 'dn4']

last = {"dn1": 2, "dn2": 700, "dn3": 1, "dn4": 640, "dn5": 3}
dead = [n for n, s in last.items() if s > 630]
print("Live:", len(last) - len(dead), "| Dead:", len(dead), dead)

256 GB · OS 8 · DN 4 · NM 2 · containers 4 GB

yarn.nodemanager.resource.memory-mb = 247808
4 GB containers per node: 60

total, os_gb, dn, nm = 256, 8, 4, 2
yarn_gb = total - os_gb - dn - nm
print("yarn.nodemanager.resource.memory-mb =", yarn_gb * 1024)
print("4 GB containers per node:", yarn_gb // 4)

120 M objects · +25%/year · +30%

Year 0: ~156 GB NameNode heap
Year 2: ~244 GB NameNode heap

objects_m = 120
for year in [0, 2]:
    need = objects_m * 1.25 ** year * 1.3
    print(f"Year {year}: ~{need:.0f} GB NameNode heap")

Real-world scenario questions (asked in company interviews)

Architecture questions in interviews are usually practical: what runs where, how they talk, and what you check when something breaks.