Security: Kerberos and Ranger

One environment variable, and the folder was gone

Hadoop security: Kerberos and Ranger

Words you'll meet in this lesson

From our lab (Hadoop 3.5.0, MIT Kerberos from Ubuntu 24.04)

Architecture at a glance

A secured cluster as most companies run it. Green borders are what our lab really ran; dashed grey boxes are explained but were not installed.

Security at three levels

The layers of Hadoop security

HDFS ACL or Ranger policy?

Animated example: an intern and the salary table

A bank with Kerberos, Ranger and audit in place. The story is illustrative; the messages in the first two steps are the ones our lab printed.

From “anyone can be hadoop” to tickets and ACLs

Part 1 runs on the Module 8 lab. Part 2 uses three new containers (kerberos/docker-compose.yaml in “Run it on your PC”): an Ubuntu KDC, a Kerberos-secured NameNode and a client.

1. Unsecured: claim to be the owner

   HADOOP_USER_NAME=riya   hdfs dfs -cat /secure_demo/salaries.csv
   HADOOP_USER_NAME=hadoop hdfs dfs -rm -r -skipTrash /secure_demo

   Permission denied: user=riya, access=EXECUTE, inode="/secure_demo":hadoop:supergroup:drwx------ Deleted /secure_demo
   With hadoop.security.authentication=simple the client simply sends a user name, taken from the OS or from HADOOP_USER_NAME. HDFS checks permissions against that name faithfully, which protects against mistakes, never against intent.

2. Build a KDC and secure the NameNode

   kdb5_util create -s -r LAB.MYPHATSHALA
   kadmin.local -q "addprinc -randkey nn/nn-secure@LAB.MYPHATSHALA"
   kadmin.local -q "addprinc -pw riya123 riya@LAB.MYPHATSHALA"
   kadmin.local -q "ktadd -k /keytabs/nn.keytab nn/nn-secure@LAB.MYPHATSHALA HTTP/nn-secure@LAB.MYPHATSHALA"
   CORE-SITE.XML_hadoop.security.authentication=kerberos
   HDFS-SITE.XML_dfs.namenode.kerberos.principal=nn/nn-secure@LAB.MYPHATSHALA
   HDFS-SITE.XML_dfs.namenode.keytab.file=/keytabs/nn.keytab

   INFO UserGroupInformation - Login successful for user nn/nn-secure@LAB.MYPHATSHALA using keytab file nn.keytab INFO Server - IPC Server listener on 8020: starting
   Our first start failed for real: the image formats the NameNode when the container starts, before the KDC had written the keytab, giving KerberosAuthException: failure to login … Unable to obtain password from user. The fix was to wait until kinit -kt works, then format and start.

3. No ticket, no entry

   hdfs dfs -ls /
   HADOOP_USER_NAME=nn hdfs dfs -ls /

   AccessControlException Client cannot authenticate via:[TOKEN, KERBEROS] Client cannot authenticate via:[TOKEN, KERBEROS]
   The client offered neither a Kerberos ticket nor a delegation token, so the NameNode refused the connection before any file was checked. The environment variable is ignored in secure mode.

4. Log in as riya and look at the ticket

   kinit riya@LAB.MYPHATSHALA
   klist
   hdfs dfs -touchz /user/riya/notes.txt
   hdfs dfs -ls /user/riya

   Default principal: riya@LAB.MYPHATSHALA Valid starting Expires Service principal 10/22/2026 10:08:23 10/22/2026 20:08:23 krbtgt/LAB.MYPHATSHALA@LAB.MYPHATSHALA -rw-r--r-- riya /user/riya/notes.txt
   The TGT lasts 10 hours (ticket_lifetime = 10h in krb5.conf). Hadoop's Java client then asks the KDC for a service ticket for nn/nn-secure and keeps it in memory. We used touchz because the secure lab has no DataNode: secure DataNodes need privileged ports or SASL with HTTPS.

5. Try the payroll folder, and try to pretend

   hdfs dfs -cat /payroll/salaries.csv
   HADOOP_USER_NAME=nn hdfs dfs -cat /payroll/salaries.csv
   kdestroy; kinit riya@LAB.MYPHATSHALA      # with a wrong password

   Permission denied: user=riya, access=EXECUTE, inode="/payroll":nn:supergroup:drwx------ Permission denied: user=riya, access=EXECUTE, inode="/payroll":nn:supergroup:drwx------ kinit: Password incorrect while getting initial credentials
   Now the user name comes from the ticket, so both attempts are riya. Note the owner of /payroll is nn: the short name of nn/nn-secure@LAB.MYPHATSHALA after the default auth_to_local rule.

6. Read the KDC's log

   docker logs lab12k-kdc-1 | grep -E "AS_REQ|TGS_REQ"

   1 AS_REQ 172.23.0.3: ISSUE: riya@LAB.MYPHATSHALA for krbtgt/LAB.MYPHATSHALA@LAB.MYPHATSHALA 4 TGS_REQ 172.23.0.3: ISSUE: riya@LAB.MYPHATSHALA for nn/nn-secure@LAB.MYPHATSHALA 1 AS_REQ 172.23.0.3: ISSUE: nn/nn-secure@LAB.MYPHATSHALA for krbtgt/LAB.MYPHATSHALA@LAB.MYPHATSHALA 4 TGS_REQ 172.23.0.3: ISSUE: nn/nn-secure@LAB.MYPHATSHALA for nn/nn-secure@LAB.MYPHATSHALA
   Exactly the two exchanges from the definition: one AS request for a TGT, then TGS requests for the NameNode's service ticket (one per hdfs command, since each is a new JVM). The KDC never sees a password; it only encrypts with the key derived from it.

7. Authorization beyond owner and group: an ACL

   hdfs dfs -chmod 750 /finance
   hdfs dfs -setfacl -m user:kabir:r-x /finance
   hdfs dfs -getfacl /finance

   user::rwx user:kabir:r-x group::r-x mask::r-x other::--- -- kabir (named in the ACL): 1 file visible -- riya (not in the ACL): Permission denied: user=riya drwxr-x---+ hadoop /finance
   The + after the permission bits means “has an ACL”. Ranger generalises this idea: policies by user or group, kept in one place, across services, with audit. (This step ran on the unsecured lab, so the names are only as trustworthy as step 1.)

A Kerberos login, as the KDC saw it

Six steps from our secure lab: the client's terminal on the left, the KDC and NameNode on the right.

Security at work

Simplified; organisations differ in tools and policies. India's Digital Personal Data Protection Act, 2023 adds legal duties for personal data; ask your legal team what applies, this lesson is not legal advice.

Simulate a Kerberos login

A toy Kerberos with real ideas and fake cryptography: each principal has a secret key, tickets are “sealed” with the key of whoever must read them, and every ticket has an expiry time. Watch how riya gets into the NameNode and why an impostor cannot.

import hashlib
def key(secret):                                  # a "key" derived from a password or keytab
    return hashlib.sha256(secret.encode()).hexdigest()[:8]
def seal(k, data):  return {"sealed_with": k, "data": data}
def open_(k, box):
    if box["sealed_with"] != k:
        raise PermissionError("cannot decrypt: wrong key")
    return box["data"]

KDC = {"riya@LAB": key("riya123"), "krbtgt@LAB": key("kdc-secret"), "nn/nn-secure@LAB": key("from-keytab")}

def as_req(user, now):                           # 1. AS: password -> TGT
    tgt = seal(KDC["krbtgt@LAB"], {"user": user, "expires": now + 10})
    return seal(KDC[user], {"tgt": tgt})          # only the user's key opens the reply

def tgs_req(tgt, service, now):                   # 2. TGS: TGT -> service ticket
    t = open_(KDC["krbtgt@LAB"], tgt)
    if now > t["expires"]:
        raise PermissionError("TGT expired")
    return seal(KDC[service], {"user": t["user"], "expires": t["expires"]})

def namenode(ticket, claimed_name=None):          # 3. service: reads the user from the ticket
    t = open_(KDC["nn/nn-secure@LAB"], ticket)
    return f"NameNode sees user={t['user'].split('@')[0]} (claimed {claimed_name})"

reply = as_req("riya@LAB", now=0)
tgt = open_(key("riya123"), reply)["tgt"]         # riya types the right password
ticket = tgs_req(tgt, "nn/nn-secure@LAB", now=1)
print(namenode(ticket, claimed_name="nn"))
try:
    open_(key("wrongpass"), reply)
except PermissionError as e:
    print("wrong password:", e)

• return seal(KDC[user], {"tgt": tgt}) # only the user's key opens the replyThe KDC answers anyone, but the answer is sealed with the user's key. That is why a wrong password gave "Password incorrect" without the password crossing the network.
• return seal(KDC[service], {"user": t["user"], "expires": t["expires"]})A service ticket is sealed with the service's key, which the NameNode has in its keytab.
• t = open_(KDC["nn/nn-secure@LAB"], ticket)The NameNode takes the user from the ticket. The claimed name is ignored, as HADOOP_USER_NAME was in our lab.

Run it on your PC: a KDC, a secured NameNode and an ACL

Part 1 and Part 3 use the Module 8 lab. Part 2 needs the three containers in the kerberos folder: an Ubuntu KDC (it installs MIT Kerberos with apt the first time, so it needs internet), a NameNode with Kerberos switched on, and a client. Put the five files below in a folder called kerberos, with an empty keytabs subfolder, then run docker compose -p lab12k up -d, wait about two minutes and run bash kerberos_lab.sh. Our secure NameNode has no DataNodes, so the demo only creates empty files.

# Lab 12.5: a Kerberos KDC, a Kerberos-secured NameNode, and a client machine.
services:
  kdc:
    image: ubuntu:24.04
    hostname: kdc
    command: ["bash", "/lab/kdc.sh"]
    volumes: [./:/lab:ro, ./keytabs:/keytabs]
  nn-secure:
    image: apache/hadoop:3.5.0
    hostname: nn-secure
    command: ["bash", "-c", "until kinit -kt /keytabs/nn.keytab nn/nn-secure@LAB.MYPHATSHALA 2>/dev/null; do sleep 3; done; kdestroy; [ -d /tmp/hadoop-hadoop/dfs/name/current ] || hdfs namenode -format -force -nonInteractive; hdfs namenode"]
    env_file: [./config]
    # no ENSURE_NAMENODE_DIR: the image would format before the keytab exists; we format after the KDC is up
    environment: { KRB5_CONFIG: /lab/krb5.conf }
    volumes: [./:/lab:ro, ./keytabs:/keytabs:ro, ./krb5.conf:/etc/krb5.conf:ro]
    depends_on: [kdc]
  client:
    image: apache/hadoop:3.5.0
    hostname: client
    command: ["sleep", "infinity"]
    env_file: [./config]
    volumes: [./krb5.conf:/etc/krb5.conf:ro, ./keytabs:/keytabs:ro]

# ---- krb5.conf (same folder) ----
# [libdefaults]
#   default_realm = LAB.MYPHATSHALA
#   dns_lookup_realm = false
#   dns_lookup_kdc = false
#   udp_preference_limit = 1
#   ticket_lifetime = 10h
#   forwardable = true
# [realms]
#   LAB.MYPHATSHALA = {
#     kdc = kdc
#     admin_server = kdc
#   }
# ---- config (same folder) ----
# CORE-SITE.XML_fs.defaultFS=hdfs://nn-secure:8020
# CORE-SITE.XML_hadoop.security.authentication=kerberos
# CORE-SITE.XML_hadoop.security.authorization=true
# HDFS-SITE.XML_dfs.namenode.rpc-address=nn-secure:8020
# HDFS-SITE.XML_dfs.namenode.kerberos.principal=nn/nn-secure@LAB.MYPHATSHALA
# HDFS-SITE.XML_dfs.namenode.keytab.file=/keytabs/nn.keytab
# HDFS-SITE.XML_dfs.web.authentication.kerberos.principal=HTTP/nn-secure@LAB.MYPHATSHALA
# HDFS-SITE.XML_dfs.web.authentication.kerberos.keytab=/keytabs/nn.keytab
# HDFS-SITE.XML_dfs.block.access.token.enable=true
# HDFS-SITE.XML_dfs.replication=1

# kdc.sh: a tiny MIT Kerberos KDC for the lab: realm LAB.MYPHATSHALA
set -e
apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq krb5-kdc krb5-admin-server > /dev/null
cp /lab/krb5.conf /etc/krb5.conf
mkdir -p /etc/krb5kdc /var/lib/krb5kdc
printf '[logging]\n kdc = STDERR\n[realms]\n LAB.MYPHATSHALA = {\n  database_name = /var/lib/krb5kdc/principal\n  acl_file = /etc/krb5kdc/kadm5.acl\n  key_stash_file = /etc/krb5kdc/stash\n }\n' > /etc/krb5kdc/kdc.conf
echo '*/admin@LAB.MYPHATSHALA *' > /etc/krb5kdc/kadm5.acl
kdb5_util create -s -r LAB.MYPHATSHALA -P masterkey123 > /dev/null
kadmin.local -q "addprinc -randkey nn/nn-secure@LAB.MYPHATSHALA" > /dev/null      # the NameNode service
kadmin.local -q "addprinc -randkey HTTP/nn-secure@LAB.MYPHATSHALA" > /dev/null    # its web UI (SPNEGO)
kadmin.local -q "addprinc -pw riya123 riya@LAB.MYPHATSHALA" > /dev/null           # two people
kadmin.local -q "addprinc -pw kabir123 kabir@LAB.MYPHATSHALA" > /dev/null
rm -f /keytabs/nn.keytab
kadmin.local -q "ktadd -k /keytabs/nn.keytab nn/nn-secure@LAB.MYPHATSHALA HTTP/nn-secure@LAB.MYPHATSHALA" > /dev/null
chmod 644 /keytabs/nn.keytab
kadmin.local -q "listprincs"
echo "KDC ready"
exec krb5kdc -n

# kerberos_lab.sh: the same commands on an unsecured cluster and on a Kerberos-secured NameNode.
# Run on your PC: part 1 uses the Module 8 lab, part 2 the lab12k containers.
echo "######## 1. UNSECURED (Module 8 lab): Hadoop believes whatever user name you claim"
docker exec lab8-namenode-1 bash -c '
hdfs dfs -mkdir -p /secure_demo && hdfs dfs -chmod 700 /secure_demo && hdfs dfs -put -f /etc/hostname /secure_demo/salaries.csv
echo "-- as riya (not the owner):"
HADOOP_USER_NAME=riya hdfs dfs -cat /secure_demo/salaries.csv 2>&1 | grep -o "Permission denied.*" | cut -c1-120
echo "-- as riya, claiming to be the owner \"hadoop\":"
HADOOP_USER_NAME=hadoop hdfs dfs -ls /secure_demo 2>&1 | grep salaries | awk "{print \$1, \$3, \$8}"
echo "-- as anyone, claiming to be the owner, deleting it:"
HADOOP_USER_NAME=hadoop hdfs dfs -rm -r -skipTrash /secure_demo 2>&1 | grep -v INFO'
echo
echo "######## 2. KERBEROS (lab12k): identity comes from a ticket, not from a variable"
docker exec lab12k-client-1 bash -c 'quiet() { grep -vE "^\s|^Usage|^$|^Generic|^command|^-[a-z]+ <|WARN"; }

echo "-- no ticket:"
hdfs dfs -ls / 2>&1 | grep -oE "Client cannot authenticate via:\[[A-Z, ]+\]|AccessControlException|GSS initiate failed" | sort -u | head -2
echo "-- claiming a name does nothing now:"
HADOOP_USER_NAME=nn hdfs dfs -ls / 2>&1 | grep -oE "Client cannot authenticate via:\[[A-Z, ]+\]" | head -1
echo "-- the service admin logs in with its keytab and makes a home folder for riya:"
kinit -kt /keytabs/nn.keytab nn/nn-secure@LAB.MYPHATSHALA
hdfs dfs -mkdir -p /user/riya /payroll 2>&1 | quiet
hdfs dfs -chown riya /user/riya; hdfs dfs -chmod 700 /payroll
hdfs dfs -touchz /payroll/salaries.csv      # our secure lab has no DataNode, so we create empty files (metadata only)
kdestroy
echo "-- riya logs in with her password:"
echo riya123 | kinit riya@LAB.MYPHATSHALA > /dev/null
klist | sed -n "1,6p"
hdfs dfs -touchz /user/riya/notes.txt
hdfs dfs -ls /user/riya | grep notes | awk "{print \$1, \$3, \$8}"
echo "-- riya tries the payroll folder, and tries to pretend:"
hdfs dfs -cat /payroll/salaries.csv 2>&1 | grep -o "Permission denied.*" | cut -c1-110
HADOOP_USER_NAME=nn hdfs dfs -cat /payroll/salaries.csv 2>&1 | grep -o "Permission denied.*" | cut -c1-110
echo "-- the cache holds only the TGT; Java asks the KDC for the NameNode service ticket in memory"
klist | grep -E "krbtgt|nn/"
echo "-- wrong password:"
kdestroy; echo wrongpass | kinit riya@LAB.MYPHATSHALA 2>&1 | tail -1'
echo "-- the KDC log saw all of it:"
docker logs lab12k-kdc-1 2>&1 | grep -E "AS_REQ|TGS_REQ" | sed -E 's/^.*krb5kdc\[[0-9]+\]\(info\): //; s/ \([0-9]+ etypes \{[^}]*\}\)//; s/authtime [0-9]+, etypes \{[^}]*\}, //' | uniq -c | tail -8
echo
echo "######## 3. AUTHORIZATION beyond owner/group: an HDFS ACL (Module 8 lab; the idea Ranger extends)"
docker exec lab8-namenode-1 bash -c '
hdfs dfs -mkdir -p /finance && hdfs dfs -chmod 750 /finance && hdfs dfs -touchz /finance/q3.csv
hdfs dfs -setfacl -m user:kabir:r-x /finance
hdfs dfs -getfacl /finance
echo "-- kabir (named in the ACL): $(HADOOP_USER_NAME=kabir hdfs dfs -ls /finance 2>&1 | grep -c q3.csv) file visible"
echo "-- riya (not in the ACL): $(HADOOP_USER_NAME=riya hdfs dfs -ls /finance 2>&1 | grep -o "Permission denied: user=riya" )"
hdfs dfs -ls -d /finance | awk "{print \$1, \$3, \$8}"
hdfs dfs -rm -r -skipTrash /finance > /dev/null'

# Run on your PC:  bash kerberos_lab.sh

Which key opens which ticket

Follow riya's login with three keys: hers, the KDC's and the NameNode's. Each step can only be opened by one of them.

Worked examples: real problems, solved and traced in memory

Four security design problems from different organisations: naming, mapping, policies and long jobs.

nn1, rm1, w1, w2 · realm TELCO.IN

6 principals
nn/nn1.blr.telco.in@TELCO.IN
rm/rm1.blr.telco.in@TELCO.IN
dn/w1.blr.telco.in@TELCO.IN
nm/w1.blr.telco.in@TELCO.IN
dn/w2.blr.telco.in@TELCO.IN
nm/w2.blr.telco.in@TELCO.IN

3 principals · 2 trusted realms

roll123@STUDENTS.UNI.AC.IN     -> roll123
meera@STAFF.UNI.AC.IN          -> meera
eve@EVIL.ORG                   -> nobody

6 roles · 25 folders each

ACL entries to manage   : 150
Ranger policies to manage: 6 (each with a list of paths, audited centrally)

job 30 h · TGT 10 h · token 24 h renew, max 7 d

TGT expires during job: True - harmless after submission
delegation token renewed 1 time(s) by the RM; max life OK: True

Check yourself

Quick revision

Interview and viva questions

Practice problems

Security exercises built on the real messages and settings from our lab.

one principal

nn nn-secure LAB.MYPHATSHALA

p = "nn/nn-secure@LAB.MYPHATSHALA"
name, realm = p.split("@"); service, host = name.split("/")
print(service, host, realm)

10:08:23 + 10 h

20:08:23

from datetime import datetime, timedelta
print((datetime(2026, 10, 22, 10, 8, 23) + timedelta(hours=10)).time())

drwx------

owner can enter
group cannot enter
others cannot enter

bits = "rwx------"
for who, part in zip(["owner", "group", "others"], [bits[0:3], bits[3:6], bits[6:9]]):
    print(who, "can enter" if part[2] == "x" else "cannot enter")

rwx AND r-x

kabir effective: r-x

def eff(entry, mask):
    return "".join(e if (e != "-" and m != "-") else "-" for e, m in zip(entry, mask))
print("kabir effective:", eff("rwx", "r-x"))

500 × 40 · 500 × 2

TGS: 20000 AS: 1000

print("TGS:", 500 * 40, "AS:", 500 * 2)

offsets in seconds

['w2', 'edge1']

offsets = {"nn1": 12, "w1": -40, "w2": 410, "edge1": -305}
print([h for h, s in offsets.items() if abs(s) > 300])

illustrative costs

one incident Rs 520,000 vs project Rs 160,000

incident = 3 * 5 * 8000 + 200 * 2000
project = 2 * 10 * 8000
print(f"one incident Rs {incident:,} vs project Rs {project:,}")

3 requests

claimed=hadoop ticket=None  simple->hadoop  kerberos->REJECTED
claimed=nn     ticket=riya  simple->nn      kerberos->riya
claimed=None   ticket=kabir simple->os-user kerberos->kabir

reqs = [{"claimed": "hadoop", "ticket": None}, {"claimed": "nn", "ticket": "riya"}, {"claimed": None, "ticket": "kabir"}]
for r in reqs:
    simple = r["claimed"] or "os-user"
    kerberos = r["ticket"] or "REJECTED"
    print(f"claimed={r['claimed']!s:6} ticket={r['ticket']!s:5} simple->{simple:7} kerberos->{kerberos}")

Real-world scenario questions (asked in company interviews)

Security questions come up in Hadoop admin, data engineering and platform interviews, usually with a story.