Configuration files and web UIs

Who decides the value: the jar, the file, or you?

Hadoop configuration and web UIs

Words you'll meet in this lesson

Defaults checked in Hadoop 3.5.0's *-default.xml

Configuration at three levels

Kinds of settings, and the UIs that show them

The web UIs you'll use

Inspect and change configuration on a running cluster

All outputs are from the Docker practice cluster of Lesson 4.3 (Hadoop 3.5.0, 3 DataNodes). On the Path A install, drop the docker compose exec namenode prefix.

1. See the files

   ls $HADOOP_HOME/etc/hadoop

   capacity-scheduler.xml kms-log4j.properties configuration.xsl kms-site.xml container-executor.cfg log4j.properties core-site.xml mapred-env.cmd hadoop-env.cmd mapred-env.sh hadoop-env.sh mapred-queues.xml.template hadoop-metrics2.properties mapred-site.xml hadoop-policy.xml shellprofile.d hadoop-user-functions.sh.example ssl-client.xml.example hdfs-rbf-site.xml ssl-server.xml.example hdfs-site.xml user_ec_policies.xml.template httpfs-env.sh workers httpfs-log4j.properties yarn-env.cmd httpfs-site.xml yarn-env.sh kms-acls.xml yarn-site.xml kms-env.sh yarnservice-log4j.properties
   32 entries ship in etc/hadoop. The four site files, the env scripts and workers are the ones you'll edit; the rest configure optional services (KMS, HttpFS, router-based federation).

2. Ask for the value Hadoop really uses

   docker compose exec namenode hdfs getconf -confKey fs.defaultFS
   docker compose exec namenode hdfs getconf -confKey dfs.replication
   docker compose exec namenode hdfs getconf -confKey dfs.blocksize
   docker compose exec namenode hdfs getconf -confKey dfs.namenode.http-address
   docker compose exec namenode hdfs getconf -confKey dfs.heartbeat.interval

   hdfs://namenode 3 134217728 0.0.0.0:9870 3
   getconf prints the final, merged value: from your site file if set (fs.defaultFS, dfs.replication), otherwise the default (block size, address, heartbeat).

3. See where each value came from

   curl -s http://localhost:9870/conf | grep -o "<source>[^<]*</source>" | sort | uniq -c | sort -rn
   curl -s http://localhost:9870/conf | grep -o "<property><name>dfs.replication</name>[^/]*</value><final>[a-z]*</final><source>[^<]*</source>"

   542 <source>hdfs-default.xml</source> 523 <source>yarn-default.xml</source> 376 <source>core-default.xml</source> 194 <source>mapred-default.xml</source> 89 <source>hdfs-rbf-default.xml</source> 17 <source>programmatically</source> 5 <source>yarn-site.xml</source> 4 <source>mapred-site.xml</source> 2 <source>hdfs-site.xml</source> <property><name>dfs.replication</name><value>3</value><final>false</final><source>hdfs-site.xml</source>
   The running NameNode knows about 1,750 settings. All but a handful are defaults; only a few come from our site files. “programmatically” means the daemon set the value itself at start-up.

4. Override for one command with -D

   hdfs dfs -D dfs.replication=2 -put canteen_orders.csv data/copy2.csv
   hdfs dfs -D dfs.blocksize=1048576 -put canteen_orders.csv data/small_blocks.csv
   hdfs dfs -stat "%r %o %b %n" data/*
   hdfs fsck /user/hadoop/data/small_blocks.csv | grep "Total blocks"

   3 134217728 4692009 canteen_orders.csv 2 134217728 4692009 copy2.csv 3 1048576 4692009 small_blocks.csv Total blocks (validated): 5 (avg. block size 938401 B)
   %r = replication, %o = block size, %b = bytes. Same file, three different storage layouts: replication and block size are chosen by the writing client, per file. With 1 MB blocks the 4.5 MB file becomes 5 blocks. (Open a shell with docker compose exec namenode bash; the CSV was copied to /tmp in Lesson 4.3.)

5. Change an existing file, and test “final”

   hdfs dfs -setrep -w 3 data/copy2.csv
   # a copy of the configuration with dfs.replication marked final
   cp -r /opt/hadoop/etc/hadoop /tmp/fconf
   #   in /tmp/fconf/hdfs-site.xml: <value>3</value><final>true</final>
   HADOOP_CONF_DIR=/tmp/fconf hdfs dfs -D dfs.replication=2 -put canteen_orders.csv data/final_test.csv
   hdfs dfs -stat "%r %n" data/copy2.csv data/final_test.csv

   Replication 3 set: data/copy2.csv Waiting for data/copy2.csv .... done 3 copy2.csv 2 final_test.csv
   -setrep -w changes an existing file and waits until the new copies exist. The second line is the surprise from the story, tested for real: final blocks later configuration files, but -D sets the value directly in the client, so 2 still wins.

6. Open the web UIs

   In a browser on your PC: http://localhost:9870 (NameNode) and http://localhost:8088 (ResourceManager). Below are the real pages of the practice cluster, cropped.

   

   

   Screenshots show only the parts used in this course. The same data is available as JSON from /jmx (next step).

7. Read the same facts as JSON

   curl -s "http://localhost:9870/jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState" \
     | python3 -c "import sys,json; b=json.load(sys.stdin)['beans'][0]; [print(k,'=',b[k]) for k in ['FSState','NumLiveDataNodes','NumDeadDataNodes','FilesTotal','BlocksTotal','UnderReplicatedBlocks']]"

   FSState = Operational NumLiveDataNodes = 3 NumDeadDataNodes = 0 FilesTotal = 8 BlocksTotal = 8 UnderReplicatedBlocks = 0
   This is how monitoring systems watch a cluster: they poll /jmx and raise alerts when, for example, dead DataNodes or under-replicated blocks appear.

One property, four layers

Follow dfs.replication through Kabir's experiments. Each terminal line is from the real run in Part 4.

Try it yourself: settings in the playground

The upload controls are settings a client sends: Block is dfs.blocksize and Replication is dfs.replication. Change them per upload and compare the files in the NameNode metadata tab.

Configuration and UIs at work

Simplified; organisations differ in tools and policies.

Merge configuration layers like Hadoop does

This program loads three layers in order (defaults, site file, command line) and keeps the last value for each property, unless an earlier layer marked it final and the later layer is a file. It also records the source, like the /conf page.

layers = [
    ("hdfs-default.xml", "file", {"dfs.replication": "3", "dfs.blocksize": "134217728"}, set()),
    ("hdfs-site.xml", "file", {"dfs.replication": "1"}, {"dfs.replication"}),
    ("job.xml", "file", {"dfs.replication": "5", "dfs.blocksize": "268435456"}, set()),
]

conf, source, final = {}, {}, set()
for name, kind, props, finals in layers:
    for key, value in props.items():
        if key in final and kind == "file":
            print(f"ignored {key}={value} from {name} (final)")
            continue
        conf[key], source[key] = value, name
    final |= finals

for key in conf:
    print(f"{key:16} = {conf[key]:>10}  [{source[key]}]")

• ("hdfs-site.xml", "file", {"dfs.replication": "1"}, {"dfs.replication"}),The site file sets replication to 1 and marks it final (the last item is the set of final names).
• if key in final and kind == "file":A later file cannot override a final property. (Values set directly, like -D, are not files: try challenge 2.)
• final |= finalsFinals are added only after the layer is loaded, so they protect against later layers.

Run it on your PC: a configuration report from a live NameNode

With your cluster from Lesson 4.3 running, this program reads the NameNode's /conf page and reports how many settings come from each file, plus the value and source of four important ones. Save it as conf_report.py and run it on the same computer.

# conf_report.py: which settings is the NameNode really using, and from which file?
import urllib.request
import xml.etree.ElementTree as ET
from collections import Counter

URL = "http://localhost:9870/conf"           # NameNode web UI
root = ET.fromstring(urllib.request.urlopen(URL).read())

props = {}
for p in root.iter("property"):
    props[p.findtext("name")] = (p.findtext("value"), p.findtext("source"))

print(f"{len(props)} properties loaded")
for source, n in Counter(src for _, src in props.values()).most_common():
    print(f"{n:>5}  {source}")

print()
for key in ["fs.defaultFS", "dfs.replication", "dfs.blocksize", "dfs.namenode.name.dir"]:
    value, source = props.get(key, ("(not set)", "-"))
    print(f"{key:24} = {value}   [{source}]")

# In a terminal:
#   python3 conf_report.py

Which value wins, one line at a time

Follow dfs.replication through three layers and watch conf and src change in memory.

Worked examples: real problems, solved and traced in memory

Three everyday configuration tasks: turning human-friendly sizes into the bytes Hadoop expects, finding differences between two nodes' configurations, and sizing YARN memory for a worker. Each runs live and is traced in memory.

64, 128, 256 MB · 10 GB file

 64 MB =  67108864 bytes -> 160 blocks
128 MB = 134217728 bytes -> 80 blocks
256 MB = 268435456 bytes -> 40 blocks

good: replication 3, NM memory 8192, aux mapreduce_shuffle · bad: replication 3, NM memory 2048, no aux

yarn.nodemanager.aux-services          good=mapreduce_shuffle  bad=(missing)
yarn.nodemanager.resource.memory-mb    good=8192               bad=2048

128 GB RAM · OS 8 · DataNode 4 · NodeManager 2 · 4 GB containers

yarn.nodemanager.resource.memory-mb = 116736
4 GB containers per node: 28

Check yourself

Quick revision

Interview and viva questions

Practice problems

Exercises on reading, converting and checking configuration values. Each one runs in the browser.

fs.defaultFS · dfs.replication · yarn.resourcemanager.hostname · mapreduce.framework.name

fs.defaultFS -> core-site.xml
dfs.replication -> hdfs-site.xml
yarn.resourcemanager.hostname -> yarn-site.xml
mapreduce.framework.name -> mapred-site.xml

prefix = {"fs.": "core-site.xml", "dfs.": "hdfs-site.xml",
          "yarn.": "yarn-site.xml", "mapreduce.": "mapred-site.xml"}
for key in ["fs.defaultFS", "dfs.replication", "yarn.resourcemanager.hostname", "mapreduce.framework.name"]:
    print(key, "->", next(f for p, f in prefix.items() if key.startswith(p)))

NameNode, ResourceManager on nn1 · DataNode, NodeManager on w1

NameNode         http://nn1:9870
ResourceManager  http://nn1:8088
DataNode         http://w1:9864
NodeManager      http://w1:8042

uis = [("NameNode", "nn1", 9870), ("ResourceManager", "nn1", 8088),
       ("DataNode", "w1", 9864), ("NodeManager", "w1", 8042)]
for name, host, port in uis:
    print(f"{name:16} http://{host}:{port}")

2 134217728 4692009 copy2.csv

copy2.csv: 2 replicas, 128 MB blocks, 4.5 MB

r, block, size, name = "2 134217728 4692009 copy2.csv".split()
print(f"{name}: {r} replicas, {int(block) // 2**20} MB blocks, {int(size) / 2**20:.1f} MB")

4,692,009 bytes · 134217728 and 1048576

134217728 -> 1 block(s)
1048576 -> 5 block(s)

import math
for block in [134217728, 1048576]:
    print(block, "->", math.ceil(4692009 / block), "block(s)")

defaults {replication 3, blocksize 128m} · site {replication 2} · -D {blocksize 256m}

{'dfs.replication': '2', 'dfs.blocksize': '256m'}

conf = {}
for layer in [{"dfs.replication": "3", "dfs.blocksize": "128m"},
              {"dfs.replication": "2"},
              {"dfs.blocksize": "256m"}]:
    conf.update(layer)
print(conf)

<property><name>dfs.replication</name><value>3</value>…<source>hdfs-site.xml</source>

{'name': 'dfs.replication', 'value': '3', 'final': 'false', 'source': 'hdfs-site.xml'}

import re
line = "<property><name>dfs.replication</name><value>3</value><final>false</final><source>hdfs-site.xml</source>"
fields = dict(re.findall(r"<(name|value|final|source)>([^<]*)</", line))
print(fields)

20 nodes × 12 × 4 TB · 10%

raw 960 TB, top 10% = 96 TB

raw_tb = 20 * 12 * 4
print(f"raw {raw_tb} TB, top 10% = {raw_tb * 0.10:.0f} TB")

128m · 1g · 512k · 134217728

      128m -> 134,217,728 bytes
        1g -> 1,073,741,824 bytes
      512k -> 524,288 bytes
 134217728 -> 134,217,728 bytes

def to_bytes(v):
    units = {"k": 1, "m": 2, "g": 3, "t": 4}
    if v[-1].lower() in units:
        return int(v[:-1]) * 1024 ** units[v[-1].lower()]
    return int(v)
for v in ["128m", "1g", "512k", "134217728"]:
    print(f"{v:>10} -> {to_bytes(v):,} bytes")

Real-world scenario questions (asked in company interviews)

Configuration questions are about reasoning: where does a value come from, and how do you prove it?