Node roles: master, worker, edge and gateway

“Don't log in to the NameNode!”

Node roles in a Hadoop cluster

Words you'll meet in this lesson

Typical hardware by role (2026, approximate)

Roles at three levels

The roles, side by side

Example layouts by cluster size

Designing the roles for a 30-node cluster

1. Decide what must be highly available

   TiffinGo's dashboards depend on HDFS all day, so they want NameNode and ResourceManager HA.

   HA means two NameNodes, two ResourceManagers, three JournalNodes and three ZooKeeper servers, spread over three machines so any one can fail.

2. Place the masters

   masters = {
       "m1": ["NameNode (active)", "ZKFC", "JournalNode", "ZooKeeper"],
       "m2": ["NameNode (standby)", "ZKFC", "JournalNode", "ZooKeeper", "ResourceManager"],
       "m3": ["JournalNode", "ZooKeeper", "ResourceManager (standby)", "JobHistoryServer"],
   }
   print(sum(1 for roles in masters.values() if "JournalNode" in roles), "JournalNodes")

   3 JournalNodes
   Put m1, m2 and m3 on different racks and power feeds. The active NameNode gets the lightest load of other services.

3. Add utility services

   Hive Metastore with its database and HiveServer2 go on a utility node (or on m3 in a small cluster).

   The metastore database holds every table definition: back it up and give it a standby, or every Hive and Spark SQL query stops.

4. Add edge nodes

   Two edge nodes with the client configuration, Hive Beeline, spark-submit, Jupyter and the Airflow workers.

   Two, so users can keep working while one is patched. Only these two are reachable from the office network.

5. Everything else is a worker

   total, masters, utility, edge = 30, 3, 1, 2
   print(total - masters - utility - edge, "workers (DataNode + NodeManager)")

   24 workers (DataNode + NodeManager)
   The workers provide all storage and compute, so capacity planning (Lesson 1.3) is done on these 24, not on all 30.

One Hive query: what Riya does on the edge node, and which role does what

Riya runs a SQL query from the edge node. Left: her screen. Right: the request travelling through each kind of node.

Try it yourself: roles in the playground

The playground's Client box is the edge node, the dark boxes are masters, and every DataNode box is a worker. Use it to see why each role matters.

Roles in real platforms

Simplified. Exact role layouts differ by distribution and company.

Plan the roles for a cluster

This program assigns roles to the machines of a cluster: three masters for HA, one utility node, two edge nodes, and workers for the rest. Then it prints the plan. Try the challenges to check a layout for mistakes.

def plan(total, ha=True):
    masters = 3 if ha else 1
    utility, edge = 1, 2 if total >= 20 else 1
    workers = total - masters - utility - edge
    roles = {}
    for i in range(1, masters + 1):
        roles[f"master{i}"] = "master"
    roles["util1"] = "utility"
    for i in range(1, edge + 1):
        roles[f"edge{i}"] = "edge"
    for i in range(1, workers + 1):
        roles[f"worker{i}"] = "worker"
    return roles

roles = plan(30)
summary = {}
for host, role in roles.items():
    summary[role] = summary.get(role, 0) + 1
print(summary)

• masters = 3 if ha else 1High availability needs three machines for the JournalNode and ZooKeeper majorities; without HA, one master.
• utility, edge = 1, 2 if total >= 20 else 1Tuple assignment: utility = 1, and edge is 2 for bigger clusters, 1 for small ones.
• workers = total - masters - utility - edgeEvery remaining machine is a worker: storage and compute.

Counting roles, one line at a time

Watch the loop turn a host-to-role map into a summary: a dictionary that counts how many hosts have each role.

Worked examples: real problems, solved and traced in memory

Three tasks from real cluster design reviews: lay out a cluster, find rule violations in someone else's layout, and cost the hardware by role. Each runs live and is traced in memory.

12 hosts · no HA

lab01 ['NameNode', 'SecondaryNameNode', 'ResourceManager', 'JobHistoryServer']
lab02 ['(clients only)']
lab03 ['DataNode', 'NodeManager']
lab12 ['DataNode', 'NodeManager']
Workers: 10

m1 (rack1): NameNode, ZooKeeper, DataNode · m2 (rack1): NameNode, ZooKeeper · m3 (rack2): ZooKeeper, ZooKeeper

Rule 1: m1 runs worker daemons
Rule 2: ZooKeeper on 4 servers but 3 hosts
Rule 3: both NameNodes on {'rack1'}

3 × 9 L · 1 × 5 L · 2 × 4 L · 34 × 12 L

master  :  3 x Rs 9 lakh = Rs 27 lakh
utility :  1 x Rs 5 lakh = Rs 5 lakh
edge    :  2 x Rs 4 lakh = Rs 8 lakh
worker  : 34 x Rs 12 lakh = Rs 408 lakh
Total Rs 448 lakh; workers are 91%

Check yourself

Quick revision

Interview and viva questions

Practice problems

Small programs and calculations for planning and checking node roles.

50 − 3 − 2 − 3

42 workers

print(50 - 3 - 2 - 3, "workers")

[3, 5]

3 servers: majority 2 | survives 1 failure(s)
5 servers: majority 3 | survives 2 failure(s)

for n in [3, 5]:
    majority = n // 2 + 1
    print(n, "servers: majority", majority, "| survives", n - majority, "failure(s)")

{"edge2": "edge", "worker7": "worker", "m1": "master"}

edge2: edge | worker7: worker

roles = {"edge2": "edge", "worker7": "worker", "m1": "master"}
print("edge2:", roles["edge2"], "| worker7:", roles["worker7"])

{"w2": "worker", "m1": "master", "e1": "edge", "w1": "worker", "m2": "master"}

edge ['e1']
master ['m1', 'm2']
worker ['w1', 'w2']

roles = {"w2": "worker", "m1": "master", "e1": "edge", "w1": "worker", "m2": "master"}
by_role = {}
for h, r in roles.items():
    by_role.setdefault(r, []).append(h)
for r in sorted(by_role):
    print(r, sorted(by_role[r]))

{"m1": "r1", "m2": "r2", "m3": "r2"}

Racks used: ['r1', 'r2'] | all different: False

rack = {"m1": "r1", "m2": "r2", "m3": "r2"}
used = set(rack.values())
print("Racks used:", sorted(used), "| all different:", len(used) == len(rack))

140 users · 25 per node · +1 spare

7 edge nodes

import math
print(math.ceil(140 / 25) + 1, "edge nodes")

20 core · 3× compute · 60 h · Rs 30/h

Add 40 task nodes; extra cost Rs 72000

core = 20
extra_nodes = core * 3 - core
cost = extra_nodes * 60 * 30
print("Add", extra_nodes, "task nodes; extra cost Rs", cost)

40 min · 120 analysts · Rs 900/h · 3 × Rs 50,000

Lost time: Rs 72,000 | SLA penalties: Rs 150,000 | Total: Rs 222,000

people = 120 * 900 * 40 / 60
sla = 3 * 50_000
print(f"Lost time: Rs {people:,.0f} | SLA penalties: Rs {sla:,} | Total: Rs {people + sla:,.0f}")

Real-world scenario questions (asked in company interviews)

Role questions are about judgement: where things should run, and what to do when someone breaks the rules.