HDFS shell commands

1 The story

Linux commands, with a twist

Riya's first day on the data team. Her task: put two days of canteen orders into HDFS, check them, merge them into one local file for the accounts team, and clean up. She guesses hdfs dfs -ls, -mkdir, -cat, -rm, and they all work like Linux. Then two surprises. Copying a 403 MB file takes 5.6 seconds, but moving it takes 1.9 seconds, the same as deleting it. And when she deletes a file by mistake, it's simply gone: no trash.

Arjun Sir

hdfs dfs deliberately looks like Linux, so you'll feel at home in a day. But think about where each command works. mv only changes a name in the NameNode's memory, so moving a 403 MB file, or a 4 TB folder, takes the same time: most of those 1.9 seconds were just Java starting up. cp has to read every byte and write it again through a pipeline.

Riya

And the trash?

Arjun Sir

HDFS has a trash, but fs.trash.interval is 0 by default, so it's off unless the admins switch it on. On our lab it's off. With it on, -rm moves files to .Trash and you can get them back. Today: the commands you'll use every day, with real outputs, and which ones to be careful with.

2 Definition

The HDFS shell

Formal definition (write this in exams)

The HDFS shell is the command-line client for Hadoop file systems, invoked as hdfs dfs <command> (equivalent to hadoop fs for HDFS paths). Its commands mirror POSIX tools: listing and inspecting (-ls, -du, -df, -count, -stat, -test, -find, -checksum), reading (-cat, -head, -tail, -text), transferring between local and HDFS (-put/-copyFromLocal, -get/-copyToLocal, -getmerge, -moveFromLocal), changing the namespace (-mkdir, -mv, -cp, -rm, -appendToFile, -touchz) and managing attributes (-chmod, -chown, -chgrp, -setrep). Paths may be absolute (/user/…), relative to the user's home /user/<name>, or full URIs (hdfs://namenode/…, file:///…). Generic options such as -D property=value come before the command.

In simple words

hdfs dfs -something is Linux file handling for HDFS: list, copy in, copy out, read, move, delete, change permissions. Commands that only rename or list are instant; commands that move data take time.

Words you'll meet in this lesson

hdfs dfs vs hadoop fsSame shell; hadoop fs works with any Hadoop file system (S3, local…), hdfs dfs is the usual name for HDFS.

Home directory/user/<username>; relative paths start there.

-put / -getCopy local → HDFS / HDFS → local.

-getmergeConcatenate all files of an HDFS folder into one local file.

Trash.Trash/Current in your home; used by -rm only if fs.trash.interval > 0.

-skipTrashDelete immediately even if trash is on.

-du first columnLogical size; second column: raw space including replicas.

Exit code0 = success; -test returns 0/1, used in scripts.

Defaults that surprise people (Hadoop 3.5.0)

fs.trash.interval	0: trash is off; set minutes (e.g. 1440 = 1 day) in core-site.xml to enable
-put on an existing file	Fails with “File exists”; use `-f` to overwrite
-mkdir on an existing folder	Fails; `-p` creates parents and ignores existing folders
JVM start-up	About 1.8 s per command in the lab: script many operations in one call where possible
-du columns	4.5 M, 13.4 M: data size and space used with 3 replicas

The shell at three levels

Pick your level. New to this? Start at Beginner and climb.

Beginner Like a phone's file manager

Folders, files, copy, move, delete: the same ideas.
Upload (-put) is like sending a photo to the cloud; download (-get) brings it back.
Renaming is instant; copying a big video takes time. HDFS behaves the same way.

Intermediate Everyday patterns

Load data: -mkdir -p, then -put (or -put -f to replace).
Check: -ls, -du -s -h, -count, -head / -tail.
Publish results: jobs write to a temporary folder, then one -mv makes them visible atomically.
Export: -getmerge a result folder into one local file for business users.
Script checks: hdfs dfs -test -e path && ….

Advanced Things to know before you script

Each call starts a JVM. For thousands of small operations use the Java API, WebHDFS, or one job, not a loop of shell calls.
-cp copies through the client (it reads and writes every byte); for big copies use distcp, which runs as a parallel job.
-rm -r on a big folder is fast for you but creates lots of block deletions for DataNodes, done in the background.
Trash is per user and client-side policy plus server setting; never rely on it for important data.
Globs (*, ?) are expanded by HDFS: quote them so your local shell doesn't expand them first.

Exam tip: For “List any ten HDFS shell commands with syntax”, group them: list (-ls, -du, -count), read (-cat, -tail), transfer (-put, -get, -getmerge), change (-mkdir, -mv, -cp, -rm), attributes (-chmod, -chown, -setrep). Give the syntax hdfs dfs -command [options] path and one example each.

3 Types and ways to use it

Command reference by task

Task	Command	Example	Touches DataNodes?
List	`-ls [-R] [-h]`	`hdfs dfs -ls -h /user/hadoop/data`	No
Make folder	`-mkdir [-p]`	`hdfs dfs -mkdir -p raw/2026/10`	No
Upload	`-put [-f]`	`hdfs dfs -put day1.csv raw/`	Yes (write pipeline)
Download	`-get`, `-getmerge`	`hdfs dfs -getmerge out/ result.csv`	Yes (reads)
Read	`-cat`, `-head`, `-tail`, `-text`	`hdfs dfs -tail logs/app.log`	Yes
Size	`-du [-s] [-h]`, `-df -h`, `-count`	`hdfs dfs -du -s -h data`	No
Move / rename	`-mv`	`hdfs dfs -mv tmp/out final/`	No
Copy	`-cp` (small), `hadoop distcp` (big)	`hdfs dfs -cp a.csv b.csv`	Yes (read + write)
Delete	`-rm [-r] [-skipTrash]`	`hdfs dfs -rm -r old/`	Later (block deletion)
Permissions	`-chmod`, `-chown`, `-chgrp`	`hdfs dfs -chmod 640 raw/day1.csv`	No
Replication	`-setrep [-w] N`	`hdfs dfs -setrep -w 2 archive/`	Later (copies/deletes)
Inspect	`-stat`, `-test`, `-find`, `-checksum`	`hdfs dfs -stat "%r %o" f`	No (checksum: yes)

Same shell, other file systems

hdfs://

The default here (fs.defaultFS).

HDFS

file://

hadoop fs -ls file:///tmp lists the local disk.

Local

s3a://, abfs://, gs://

With the right connector and credentials, the same commands work on cloud storage.

Cloud

webhdfs://

HDFS over HTTP, e.g. through firewalls or from other clusters.

HTTP

Run hdfs dfs -help <command> (without the dash) for the exact options of your Hadoop version; we used it below for setrep.

4 Step by step

A morning of real commands

All run as user hadoop inside the namenode container of the lab cluster.

Create a folder and upload
```
printf 'order_id,item,amount\n2001,chai,15\n2002,dosa,45\n' > day1.csv
printf 'order_id,item,amount\n2003,poha,25\n' > day2.csv
hdfs dfs -mkdir -p /user/hadoop/canteen/raw
hdfs dfs -put day1.csv day2.csv /user/hadoop/canteen/raw/
hdfs dfs -ls /user/hadoop/canteen/raw
```
Found 2 items -rw-r--r-- 3 hadoop supergroup 47 2026-10-21 09:38 /user/hadoop/canteen/raw/day1.csv -rw-r--r-- 3 hadoop supergroup 34 2026-10-21 09:38 /user/hadoop/canteen/raw/day2.csv
Columns: permissions, replication, owner, group, size in bytes, modification time, path. Folders show - instead of a replication factor.
Read and check sizes
```
hdfs dfs -cat /user/hadoop/canteen/raw/day1.csv
hdfs dfs -du -h /user/hadoop/data
hdfs dfs -du -s -h /user/hadoop/data
hdfs dfs -df -h /
hdfs dfs -count -v -h /user/hadoop
```
order_id,item,amount 2001,chai,15 2002,dosa,45 4.5 M 13.4 M /user/hadoop/data/canteen_1mb.csv 4.5 M 13.4 M /user/hadoop/data/canteen_orders.csv 134.2 M 402.7 M /user/hadoop/data/orders_x30.csv 402.7 M 1.2 G /user/hadoop/data/orders_x90.csv 402.7 M 1.2 G /user/hadoop/data/orders_x90b.csv 948.6 M 2.8 G /user/hadoop/data Filesystem Size Used Available Use% hdfs://namenode 1007.9 G 2.8 G 66.8 G 0% DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME 13 11 958.1 M /user/hadoop
-du: the first column is the data size, the second the raw disk used with 3 replicas (4.5 M × 3 = 13.4 M). -df is the whole file system. -count -v adds a header.

Copy, rename, append, merge

hdfs dfs -cp /user/hadoop/canteen/raw/day1.csv /user/hadoop/canteen/day1_copy.csv
hdfs dfs -mv /user/hadoop/canteen/day1_copy.csv /user/hadoop/canteen/day1_backup.csv
hdfs dfs -appendToFile day2.csv /user/hadoop/canteen/day1_backup.csv
hdfs dfs -cat /user/hadoop/canteen/day1_backup.csv
hdfs dfs -getmerge /user/hadoop/canteen/raw merged.csv && cat merged.csv

order_id,item,amount 2001,chai,15 2002,dosa,45 order_id,item,amount 2003,poha,25 order_id,item,amount 2001,chai,15 2002,dosa,45 order_id,item,amount 2003,poha,25

Both append and getmerge simply concatenate bytes, so the header line appears twice. HDFS knows nothing about CSV headers; that is your program's job.

Permissions, tests and search
```
hdfs dfs -chmod 640 /user/hadoop/canteen/raw/day1.csv
hdfs dfs -ls /user/hadoop/canteen/raw
hdfs dfs -test -e /user/hadoop/canteen/raw/day1.csv; echo "exists: $?"
hdfs dfs -test -e /user/hadoop/canteen/raw/day9.csv; echo "exists: $?"
hdfs dfs -find /user/hadoop -name "*.csv"
hdfs dfs -stat "%n %b bytes, replication %r, block %o, modified %y" /user/hadoop/canteen/raw/day1.csv
```
-rw-r----- 3 hadoop supergroup 47 2026-10-21 09:38 /user/hadoop/canteen/raw/day1.csv -rw-r--r-- 3 hadoop supergroup 34 2026-10-21 09:38 /user/hadoop/canteen/raw/day2.csv exists: 0 exists: 1 /user/hadoop/canteen/day1_backup.csv /user/hadoop/canteen/raw/day1.csv /user/hadoop/canteen/raw/day2.csv /user/hadoop/data/canteen_1mb.csv /user/hadoop/data/canteen_orders.csv /user/hadoop/data/orders_x30.csv /user/hadoop/data/orders_x90.csv /user/hadoop/data/orders_x90b.csv day1.csv 47 bytes, replication 3, block 134217728, modified 2026-10-21 09:38:04
-test prints nothing; its exit code (0 = true) is what scripts use. Quote "*.csv" so the pattern reaches HDFS.
Delete: trash off, then on
```
hdfs getconf -confKey fs.trash.interval
hdfs dfs -rm /user/hadoop/canteen/day1_backup.csv
hdfs dfs -ls /user/hadoop/.Trash/Current/user/hadoop/canteen
hdfs dfs -D fs.trash.interval=1440 -rm /user/hadoop/canteen/raw/day2.csv
hdfs dfs -mv /user/hadoop/.Trash/Current/user/hadoop/canteen/raw/day2.csv /user/hadoop/canteen/raw/
```
0 Deleted /user/hadoop/canteen/day1_backup.csv ls: `/user/hadoop/.Trash/Current/user/hadoop/canteen': No such file or directory INFO TrashPolicyDefault:196 - Moved: 'hdfs://namenode/user/hadoop/canteen/raw/day2.csv' to trash at: hdfs://namenode/user/hadoop/.Trash/Current/user/hadoop/canteen/raw/day2.csv
With trash off (0), “Deleted” means gone. With trash on, the file moves to .Trash/Current under the same path, and a -mv restores it. Real clusters usually set fs.trash.interval in core-site.xml.
cp vs mv on a 403 MB file
```
time hdfs dfs -cp data/orders_x90.csv data/orders_x90_copy.csv
time hdfs dfs -mv data/orders_x90_copy.csv data/orders_x90_moved.csv
time hdfs dfs -rm -skipTrash data/orders_x90_moved.csv
```
real 0m5.615s real 0m1.863s Deleted /user/hadoop/data/orders_x90_moved.csv real 0m1.830s
cp read and rewrote 403 MB (×3 replicas on disk). mv and rm are metadata changes; their time is mostly the JVM starting, and would be the same for 4 TB.
Errors you will meet
```
hdfs dfs -ls /user/hadoop/canteen/raw/day9.csv
hdfs dfs -mkdir /user/hadoop/canteen/raw
hdfs dfs -put day1.csv /user/hadoop/canteen/raw/
```
ls: `/user/hadoop/canteen/raw/day9.csv': No such file or directory mkdir: `/user/hadoop/canteen/raw': File exists put: `/user/hadoop/canteen/raw/day1.csv': File exists
Fixes: check the path; use -mkdir -p; use -put -f to overwrite on purpose.

hdfs dfs -rm -r -skipTrash /some/path cannot be undone. Many teams protect important folders with snapshots (Lesson 7.4) and require trash to be enabled on the cluster.

5 See it happen

Which daemons does each command wake up?

Six commands from Part 4, and the daemons each one involves.

Metadata commands vs data commands 6 steps

What you see

Behind the screen

6 Real-world use

The shell at work

1. Atomic publishing with mv

Pipelines write today's results into out/_tmp_2026-10-21 and only then -mv it to out/2026-10-21. Readers never see half-written data, because a rename is instant and all-or-nothing.

Write to tmp→-mv→Visible at once

2. Shell checks in schedulers

Airflow or cron jobs start the next step only if hdfs dfs -test -e input/_SUCCESS returns 0, and alert if -du -s shows an unusually small input.

-test→Exit code→Run or alert

3. Handing results to business users

A job writes part-00000 … part-00099; an analyst runs -getmerge to get one CSV for a spreadsheet (and removes repeated headers first, as we saw).

part files→getmerge→One CSV

4. Trash as a safety net

Most production clusters set fs.trash.interval to a day or more. Support teams regularly restore files from .Trash after accidental deletes, something impossible with the default of 0.

Wrong -rm→.Trash→-mv back

Simplified; organisations differ in tools and policies.

7 Run the code

A tiny HDFS shell in Python

This program keeps a namespace in a dictionary (path → size and replication) and implements ls, mv, du and rm, counting which ones would touch DataNodes. Run it, then try the tasks.

ns = {"/user/hadoop/raw/mon.csv": (47, 3), "/user/hadoop/raw/tue.csv": (34, 3)}
datanode_bytes = 0

def ls(folder):
    for path, (size, rep) in sorted(ns.items()):
        if path.startswith(folder + "/"):
            print(f"-rw-r--r--  {rep} hadoop {size:>6} {path}")

def mv(src, dst):                 # metadata only: no bytes move
    ns[dst] = ns.pop(src)

def cp(src, dst):                 # reads and writes every byte
    global datanode_bytes
    size, rep = ns[src]
    datanode_bytes += size + size * rep
    ns[dst] = (size, rep)

def du(folder):
    files = [v for p, v in ns.items() if p.startswith(folder + "/")]
    print(sum(s for s, r in files), sum(s * r for s, r in files), folder)

cp("/user/hadoop/raw/mon.csv", "/user/hadoop/mon_backup.csv")
mv("/user/hadoop/mon_backup.csv", "/user/hadoop/archive_mon.csv")
ls("/user/hadoop/raw")
du("/user/hadoop")
print("bytes through DataNodes:", datanode_bytes)

-rw-r--r-- 3 hadoop 47 /user/hadoop/raw/mon.csv -rw-r--r-- 3 hadoop 34 /user/hadoop/raw/tue.csv 128 384 /user/hadoop bytes through DataNodes: 188

def mv(src, dst): # metadata only: no bytes moveA rename only changes the key in the namespace. That is why -mv of 403 MB took 1.863 s, mostly Java start-up.
datanode_bytes += size + size * repcp reads the file once and writes it again with 3 replicas: 47 + 141 bytes here.
print(sum(s for s, r in files), sum(s * r for s, r in files), folder)The two du columns: data size and raw space with replicas. Real output: 81 243 for the shell_lab folder before the copy.

Run it on your PC: a shell workout

This script creates two small CSV files and runs 14 everyday commands on a practice folder, printing each command before its output. Save it as shell_lab.sh in the hdfs-lab folder, copy it into the namenode container and run it.

Fourteen commands, one folder

#!/usr/bin/env bash
# shell_lab.sh: practise the everyday hdfs dfs commands on a small folder.
# Run inside the namenode container:  bash shell_lab.sh
set -e
run() { echo "\$ $*"; eval "$@"; echo; }
cd /tmp
printf 'order_id,item,amount\n4001,idli,30\n4002,chai,15\n' > mon.csv
printf 'order_id,item,amount\n4003,dosa,45\n' > tue.csv
D=/user/hadoop/shell_lab

run "hdfs dfs -rm -r -f -skipTrash $D > /dev/null"
run "hdfs dfs -mkdir -p $D/raw"
run "hdfs dfs -put mon.csv tue.csv $D/raw/"
run "hdfs dfs -ls $D/raw"
run "hdfs dfs -cat $D/raw/mon.csv"
run "hdfs dfs -du -s -h $D"
run "hdfs dfs -cp $D/raw/mon.csv $D/mon_backup.csv"
run "hdfs dfs -mv $D/mon_backup.csv $D/archive_mon.csv"
run "hdfs dfs -getmerge $D/raw week.csv && wc -l week.csv"
run "hdfs dfs -setrep -w 2 $D/archive_mon.csv"
run "hdfs dfs -stat '%n %r %b' $D/archive_mon.csv"
run "hdfs dfs -count -v $D"
run "hdfs dfs -D fs.trash.interval=60 -rm $D/archive_mon.csv"
run "hdfs dfs -ls -R $D"

# In the hdfs-lab folder:
#   docker cp shell_lab.sh lab5-namenode-1:/tmp/
#   docker exec lab5-namenode-1 bash /tmp/shell_lab.sh

$ hdfs dfs -rm -r -f -skipTrash /user/hadoop/shell_lab > /dev/null $ hdfs dfs -mkdir -p /user/hadoop/shell_lab/raw $ hdfs dfs -put mon.csv tue.csv /user/hadoop/shell_lab/raw/ $ hdfs dfs -ls /user/hadoop/shell_lab/raw Found 2 items -rw-r--r-- 3 hadoop supergroup 47 2026-10-21 09:54 /user/hadoop/shell_lab/raw/mon.csv -rw-r--r-- 3 hadoop supergroup 34 2026-10-21 09:54 /user/hadoop/shell_lab/raw/tue.csv $ hdfs dfs -cat /user/hadoop/shell_lab/raw/mon.csv order_id,item,amount 4001,idli,30 4002,chai,15 $ hdfs dfs -du -s -h /user/hadoop/shell_lab 81 243 /user/hadoop/shell_lab $ hdfs dfs -cp /user/hadoop/shell_lab/raw/mon.csv /user/hadoop/shell_lab/mon_backup.csv $ hdfs dfs -mv /user/hadoop/shell_lab/mon_backup.csv /user/hadoop/shell_lab/archive_mon.csv $ hdfs dfs -getmerge /user/hadoop/shell_lab/raw week.csv && wc -l week.csv 5 week.csv $ hdfs dfs -setrep -w 2 /user/hadoop/shell_lab/archive_mon.csv Replication 2 set: /user/hadoop/shell_lab/archive_mon.csv Waiting for /user/hadoop/shell_lab/archive_mon.csv ... WARNING: the waiting time may be long for DECREASING the number of replications. . done $ hdfs dfs -stat '%n %r %b' /user/hadoop/shell_lab/archive_mon.csv archive_mon.csv 2 47 $ hdfs dfs -count -v /user/hadoop/shell_lab DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME 2 3 128 /user/hadoop/shell_lab $ hdfs dfs -D fs.trash.interval=60 -rm /user/hadoop/shell_lab/archive_mon.csv 2026-10-21 09:54:53 INFO TrashPolicyDefault:196 - Moved: 'hdfs://namenode/user/hadoop/shell_lab/archive_mon.csv' to trash at: hdfs://namenode/user/hadoop/.Trash/Current/user/hadoop/shell_lab/archive_mon.csv $ hdfs dfs -ls -R /user/hadoop/shell_lab drwxr-xr-x - hadoop supergroup 0 2026-10-21 09:54 /user/hadoop/shell_lab/raw -rw-r--r-- 3 hadoop supergroup 47 2026-10-21 09:54 /user/hadoop/shell_lab/raw/mon.csv -rw-r--r-- 3 hadoop supergroup 34 2026-10-21 09:54 /user/hadoop/shell_lab/raw/tue.csv

Real output from the lab cluster. Dates differ on your run. Note: CONTENT_SIZE 128 = 47 + 34 + 47, DIR_COUNT 2 counts shell_lab itself and raw, and the trash message proves the file can still be restored.

8 Line by line

Publishing results with one rename

Follow a job that writes part files into a temporary folder and publishes them with a single move: readers see either nothing or everything.

Worked examples: real problems, solved and traced in memory

Three shell questions with numbers: how much JVM start-up costs in a loop, how big a folder really is on disk, and how much setrep can free. Each runs live and is traced in memory.

Example 1BeginnerScripting · cost

Is a loop of hdfs dfs calls slow?

Problem statement

A script checks 600 files with one hdfs dfs -test -e each. Each call costs about 1.8 s of JVM start-up (measured in the lab). How long does the loop take, versus one -ls -R of the folder (one call) and checking the names in the output?

Input

600 calls · 1.8 s each

Expected output

loop: 18 minutes, one listing: 1.8 s

Approach

Loop time = calls × start-up time.
One listing = one start-up.
Compare.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 2IntermediateCapacity · du

How much disk does the data folder really use?

Problem statement

The lab printed 948.6 M 2.8 G /user/hadoop/data. The files are 4.5 M, 4.5 M, 134.2 M, 402.7 M and 402.7 M with replication 3. Recompute both columns.

Input

five files · replication 3

Expected output

948.6 M   2.8 G

Approach

First column = sum of file sizes.
Second = sum × replication.
Convert MB to GB by dividing by 1024.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

Example 3AdvancedCost · setrep

How much space does setrep 2 on old data free?

Problem statement

A cluster has 600 TB of data at replication 3. Data older than 90 days is 70% of it and rarely read. If we -setrep 2 the old data, how much raw disk is freed, and what is the risk?

Input

600 TB · 70% old · 3 → 2

Expected output

raw before 1800 TB, after 1380 TB, freed 420 TB

Approach

Raw before = 600 × 3.
Raw after = old × 2 + new × 3.
Difference is the saving; risk: only one spare copy.

Solution: run it live

What happens in memory, step by step

Recorded from a real Python run: press Next to execute one line at a time. Watch variables change in each frame, and lists and dictionaries (numbered #1, #2 …) fill up as objects.

9 Quiz and revision

Check yourself

Quick revision

Lesson 6.3 in one screen

Syntax: hdfs dfs [-D prop=value] -command [options] paths; relative paths start at /user/<name>.
Metadata only (instant): -ls -mkdir -mv -rm -chmod -count -du -stat -test.
Data commands (time ∝ size): -put -get -cat -cp -getmerge -appendToFile.
Lab: cp 5.615 s vs mv 1.863 s vs rm 1.830 s on 403 MB.
-du: data size, raw size with replicas (81 / 243).
Trash is off by default (fs.trash.interval=0); with it on, files go to .Trash/Current.
Errors: File exists (use -p / -f), No such file or directory.
Scripts: use -test exit codes; quote globs; avoid thousands of separate calls.

Interview and viva questions

What is the difference between hadoop fs and hdfs dfs?

They run the same FsShell. hadoop fs works with any file system Hadoop supports (HDFS, local, S3, ADLS…), and hdfs dfs is the conventional name when working with HDFS. hadoop dfs is an old deprecated form.

How do I copy a file from local disk to HDFS?

hdfs dfs -put localfile /hdfs/path/ (or -copyFromLocal). Use -f to overwrite an existing file. To move instead of copy, -moveFromLocal deletes the local file after upload.

Why is hdfs dfs -mv so fast?

A move or rename within HDFS only changes the path in the NameNode namespace; no block data is copied. Its time is dominated by the client JVM start-up, so it is the same for a tiny or a huge file.

How do I recover a deleted file?

Only if trash was enabled (fs.trash.interval > 0) and the file was not deleted with -skipTrash: move it back from /user/<name>/.Trash/Current/<original path>. Otherwise restore from a snapshot or backup if one exists.

What do the columns of hdfs dfs -du mean?

First: the size of the data. Second: the disk space consumed including all replicas. Then the path. With -s you get one total line for the folder, and -h prints human-readable units.

How do I change the replication factor of existing files?

hdfs dfs -setrep N path (folders apply to all files inside). The NameNode updates the metadata immediately and schedules extra copies or deletions; -w waits until the actual replica count matches.

Practice problems

Shell exercises using the real outputs from the lab.

P1Easy

Parse an ls line

From a real -ls line, extract permissions, replication, size and file name.

Input

-rw-r--r--   3 hadoop supergroup         47 2026-10-21 09:54 /user/hadoop/shell_lab/raw/mon.csv

Expected output

-rw-r--r-- 3 47 mon.csv

Show solution and explanation

line = "-rw-r--r--   3 hadoop supergroup         47 2026-10-21 09:54 /user/hadoop/shell_lab/raw/mon.csv"
f = line.split()
print(f[0], int(f[1]), int(f[4]), f[7].rsplit("/", 1)[1])

-rw-r--r-- 3 47 mon.csv

How it works: -rw-r--r--, replication 3, 47 bytes, mon.csv. Split on whitespace; field 1 is replication (or - for folders).

P2Easy

Count lines after getmerge

mon.csv has 3 lines, tue.csv has 2. How many lines does getmerge produce, and how many are headers?

Input

3 + 2 lines

Expected output

lines: 5  headers: 2

Show solution and explanation

files = {"mon.csv": 3, "tue.csv": 2}
print("lines:", sum(files.values()), " headers:", len(files))

lines: 5 headers: 2

How it works: 5 lines (the real wc -l printed 5 week.csv), of which 2 are header lines.

P3Easy

Octal permissions

Convert chmod 640 into the rwx string that -ls shows.

Input

Expected output

-rw-r-----

Show solution and explanation

def rwx(mode):
    return "".join(("r" if d & 4 else "-") + ("w" if d & 2 else "-") + ("x" if d & 1 else "-") for d in map(int, mode))
print("-" + rwx("640"))

-rw-r-----

How it works: -rw-r-----, exactly what the real -ls showed after hdfs dfs -chmod 640 day1.csv.

P4Medium

Remove repeated headers

After getmerge, keep only the first header line.

Input

merged lines with header repeated

Expected output

order_id,item,amount
4001,idli,30
4002,chai,15
4003,dosa,45

Show solution and explanation

merged = ["order_id,item,amount", "4001,idli,30", "4002,chai,15", "order_id,item,amount", "4003,dosa,45"]
clean = merged[:1] + [l for l in merged[1:] if l != merged[0]]
print("\n".join(clean))

order_id,item,amount 4001,idli,30 4002,chai,15 4003,dosa,45

How it works: 4 lines remain: one header and three orders. Do this before handing a file to Excel.

P5Medium

Check count -v

The real -count -v printed 2 dirs, 3 files, 128 bytes for shell_lab. Rebuild those numbers from the paths.

Input

shell_lab, raw, 3 files (47, 34, 47)

Expected output

2 3 128

Show solution and explanation

paths = {"shell_lab/": None, "shell_lab/raw/": None, "shell_lab/raw/mon.csv": 47,
         "shell_lab/raw/tue.csv": 34, "shell_lab/archive_mon.csv": 47}
dirs = sum(v is None for v in paths.values())
files = [v for v in paths.values() if v is not None]
print(dirs, len(files), sum(files))

2 3 128

How it works: 2 3 128: the folder itself counts as a directory.

P6Medium

Time a cleanup script

A cleanup script runs 120 separate hdfs dfs -rm calls at 1.83 s each. Rewritten as one call with 120 paths, it takes about 3 s. How much time is saved per run and per month (one run a day)?

Input

120 × 1.83 s vs 3 s · 30 runs

Expected output

per run 217 s saved, per month 1.8 hours

Show solution and explanation

loop, single = 120 * 1.83, 3
print(f"per run {loop - single:.0f} s saved, per month {(loop - single) * 30 / 3600:.1f} hours")

per run 217 s saved, per month 1.8 hours

How it works: About 217 s per run and 1.8 hours a month: -rm accepts many paths (and globs) in one call.

P7Business problem

Cost of no trash

A team deletes data by mistake about 4 times a year. Without trash, each restore from backup takes 2 engineers × 6 hours at Rs 1,500/hour. With trash (1 day), each takes 15 minutes of one engineer. What does enabling trash save per year?

Input

4 incidents · Rs 1,500/hour

Expected output

Rs 70,500 per year

Show solution and explanation

rate = 1500
without = 4 * 2 * 6 * rate
with_trash = 4 * 0.25 * rate
print(f"Rs {without - with_trash:,.0f} per year")

Rs 70,500 per year

How it works: Rs 70,500 per year (illustrative), and some deletions cannot be restored at all without trash or snapshots.

P8Advanced

Mini FsShell

Implement mkdir -p, put (fail if the file exists unless -f), and rm with optional trash on a dictionary namespace. Print the messages like the real shell.

Input

commands list

Expected output

put: `/user/hadoop/lab/raw/a.csv': File exists
Moved: '/user/hadoop/lab/raw/a.csv' to trash at: /user/hadoop/.Trash/Current/user/hadoop/lab/raw/a.csv
['/user/hadoop/.Trash/Current/user/hadoop/lab/raw/a.csv']

Show solution and explanation

ns, trash_on = {"/user/hadoop": "dir"}, True

def mkdir_p(p):
    parts = p.strip("/").split("/")
    for i in range(1, len(parts) + 1):
        ns.setdefault("/" + "/".join(parts[:i]), "dir")

def put(p, size, force=False):
    if p in ns and not force:
        print(f"put: `{p}': File exists"); return
    ns[p] = size

def rm(p):
    if trash_on:
        t = "/user/hadoop/.Trash/Current" + p
        ns[t] = ns.pop(p); print(f"Moved: '{p}' to trash at: {t}")
    else:
        del ns[p]; print("Deleted", p)

mkdir_p("/user/hadoop/lab/raw"); put("/user/hadoop/lab/raw/a.csv", 47)
put("/user/hadoop/lab/raw/a.csv", 50); put("/user/hadoop/lab/raw/a.csv", 50, force=True)
rm("/user/hadoop/lab/raw/a.csv")
print(sorted(p for p in ns if ns[p] != "dir"))

put: `/user/hadoop/lab/raw/a.csv': File exists Moved: '/user/hadoop/lab/raw/a.csv' to trash at: /user/hadoop/.Trash/Current/user/hadoop/lab/raw/a.csv ['/user/hadoop/.Trash/Current/user/hadoop/lab/raw/a.csv']

How it works: The second put prints File exists, the -f put overwrites, and rm moves the file to .Trash/Current with its full path, which is all that is left as a file.

Real-world scenario questions (asked in company interviews)

Shell questions check practical habits: safe deletes, fast scripts and knowing what each command really costs.

HDFS · basicsFresher

“Name ten HDFS commands you use”

The most common first question on HDFS.

The interviewer is checking: real usage, not a memorised list.

Show a strong answer

Group them: list/size (-ls, -du -s -h, -count), transfer (-put, -get, -getmerge), read (-cat, -tail), change (-mkdir -p, -mv, -cp, -rm -r), attributes (-chmod, -setrep).
Add one detail that shows experience: du's second column includes replicas; mv is metadata-only; trash is off by default.

Operations · safetyExperienced

Someone ran hdfs dfs -rm -r on a production folder

A colleague deleted yesterday's processed data by mistake and asks you to bring it back.

The interviewer is checking: trash, snapshots and prevention.

Show a strong answer

Check .Trash/Current of the user who ran it; if it is there, -mv it back.
If trash was off or -skipTrash was used, restore from a snapshot (.snapshot) or reprocess from raw data.
Prevent: enable fs.trash.interval, snapshot important folders, restrict write permissions on production paths.

Scripting · performanceExperienced

A housekeeping script takes 40 minutes

It loops over 1,300 paths, calling hdfs dfs -test and -rm for each.

The interviewer is checking: JVM start-up cost.

Show a strong answer

Each call starts a JVM (about 1.8 s): 1,300 × 2 calls ≈ 40 minutes, almost all start-up.
List once (-ls -R or -find), decide in the script, then pass many paths or a glob to one -rm.
For complex logic, use the Java API or Python with WebHDFS in a single process.

Pipelines · correctnessFresher

Reports sometimes read half-written output

A dashboard reads a folder while the nightly job is still writing to it.

The interviewer is checking: atomic rename.

Show a strong answer

Write to a temporary folder first, then publish with one -mv; a rename is atomic in the NameNode.
Also have readers wait for a _SUCCESS marker (-test -e).

Linux commands, with a twist

The HDFS shell

Words you'll meet in this lesson

Defaults that surprise people (Hadoop 3.5.0)

The shell at three levels

Beginner Like a phone's file manager

Intermediate Everyday patterns

Advanced Things to know before you script

Command reference by task

Same shell, other file systems

hdfs://

file://

s3a://, abfs://, gs://

webhdfs://

A morning of real commands

Create a folder and upload

Read and check sizes

Copy, rename, append, merge

Permissions, tests and search

Delete: trash off, then on

cp vs mv on a 403 MB file

Errors you will meet

Which daemons does each command wake up?

The shell at work

1. Atomic publishing with mv

2. Shell checks in schedulers

3. Handing results to business users

4. Trash as a safety net

A tiny HDFS shell in Python

Run it on your PC: a shell workout

Fourteen commands, one folder

Publishing results with one rename

Worked examples: real problems, solved and traced in memory

Is a loop of hdfs dfs calls slow?

How much disk does the data folder really use?

How much space does setrep 2 on old data free?

Check yourself

Quick revision

Lesson 6.3 in one screen

Interview and viva questions

Practice problems

Parse an ls line

Count lines after getmerge

Octal permissions

Remove repeated headers

Check count -v

Time a cleanup script

Cost of no trash

Mini FsShell

Real-world scenario questions (asked in company interviews)

“Name ten HDFS commands you use”

Someone ran hdfs dfs -rm -r on a production folder

A housekeeping script takes 40 minutes

Reports sometimes read half-written output