Data quality, governance, security and the DPDP Act

Every job was green

Data quality, governance, security and privacy

Words you’ll meet in this lesson

DPDP Act and Rules: the key facts (as of October 2026)

This is a learning summary, not legal advice. Check the current Act, Rules and notifications on meity.gov.in before making compliance decisions.

Quality, governance and security at three levels

Six quality dimensions and four pillars

Four pillars of trustworthy data

Making a dataset trustworthy

1. Classify the data

   Kabir tags every column in the catalog: phone, email, address as PII; amount as financial; city as internal.

   You can’t protect what you haven’t labelled. Tags drive masking, access policies and retention automatically.

2. Write quality rules with the owner

   rows = [120, 450, -150, 300, 0, 99000]
   bad = [r for r in rows if not 1 <= r <= 50_000]
   print("bad:", bad, f"-> {len(bad) / len(rows):.0%} failed")

   bad: [-150, 0, 99000] -> 50% failed
   Rules come from business meaning, so the data owner signs off: “amount between ₹1 and ₹50,000”, “one row per order”, “ready by 6 am”.

3. Gate the pipeline

   Checks run after every load. Bad rows go to a quarantine table; if more than 2% fail, the pipeline stops and alerts the owner instead of publishing.

   A stopped pipeline with an alert is far better than a confident wrong dashboard.

4. Mask and restrict

   phone = "9876543210"
   print("masked:", "XXXXXX" + phone[-4:])

   masked: XXXXXX3210
   Analysts get masked phones and pseudonymous customer keys; only the support tool can see full numbers, and every lookup is logged.

5. Record lineage and ownership

   Each table in the catalog shows its owner, description, upstream sources and downstream dashboards, captured automatically from pipeline code where possible.

   When a number looks wrong, lineage turns hours of guessing into minutes of tracing.

6. Plan for rights and breaches

   A runbook says how to find and delete one customer everywhere, how long each table is kept, and who notifies the Data Protection Board and customers if a breach happens.

   DPDP turns these from good ideas into legal duties with deadlines.

A wrong number: what Kabir sees, and where it came from

Step through the six frames. Left: the dashboard Kabir is debugging. Right: the lineage graph that leads him to the cause.

Real-time scenarios: quality gate, access and privacy law

Three live scenarios: a quality gate on a real batch (numbers from the program below), one table seen through different roles, and an advanced look at erasure requests and breaches under the DPDP Act.

A quality gate catches 110 bad rows

These figures come from the program in “Run it on your PC” below. Watch a nightly batch pass through four checks before it is allowed into trusted tables.

One table, different views for different roles

Instead of making copies with columns removed, modern platforms apply masking and row filters when the query runs, based on who is asking. Watch the same query return different results.

An erasure request and a breach under the DPDP Act

Privacy law turns data engineering choices into deadlines. Watch how a well-governed lake handles a customer’s erasure request, and what must happen when a breach is found.

Where you’ll see this at work

Sources

• DPDP Rules timelines, breach reporting and penalties: Uniqus, November 2025; Government of India explainer on the DPDP Rules, PIB, November 2025.
• Proposal to shorten compliance timelines: S.S. Rana & Co., 2026.

Write your first data quality checks

A few order rows are checked against rules for validity, completeness and uniqueness. Rows that fail are quarantined with the reason. Then try the challenges: freshness, reconciliation, masking, pseudonymous keys and a publish threshold.

orders = [
    {"id": "O1", "city": "Pune",   "amount": 320},
    {"id": "O2", "city": "",       "amount": 150},
    {"id": "O3", "city": "Thane",  "amount": -40},
    {"id": "O1", "city": "Pune",   "amount": 320},
    {"id": "O4", "city": "Mumbai", "amount": 610},
]
seen, good, quarantine = set(), [], []
for o in orders:
    reasons = []
    if not o["city"]:
        reasons.append("missing city")
    if o["amount"] <= 0:
        reasons.append("amount not positive")
    if o["id"] in seen:
        reasons.append("duplicate id")
    seen.add(o["id"])
    (quarantine if reasons else good).append((o["id"], reasons))

print("good      :", [i for i, _ in good])
print("quarantine:", quarantine)

• if not o["city"]:Completeness: a required field is empty.
• if o["id"] in seen:Uniqueness: the same order arrived twice.

Run it on your PC: a quality gate with privacy masking

This program creates a batch of 10,030 orders with realistic problems (negative amounts, missing cities, invalid phones, duplicates), runs four quality checks, quarantines failing rows, decides whether to publish, and writes a clean CSV where emails become pseudonymous keys and phones are masked. Save it as quality_guard.py.

"""A small data-quality gate and privacy step for a batch of orders.
1) generate a realistic messy batch, 2) run quality checks,
3) quarantine bad rows, 4) mask personal data before it reaches analysts."""
import csv, hashlib, random, re
from collections import Counter

random.seed(11)
CITIES = ["Pune", "Mumbai", "Thane", "Nashik"]
rows = []
for i in range(1, 10_001):
    rows.append({"order_id": f"O{i:05d}", "phone": f"9{random.randint(100000000, 999999999)}",
                 "email": f"user{i}@example.com", "city": random.choice(CITIES),
                 "amount": str(random.randint(80, 900)), "order_date": "2026-10-19"})
# inject the kinds of problems real batches have
for i in random.sample(range(10_000), 40):
    rows[i]["amount"] = "-150"                    # negative amount
for i in random.sample(range(10_000), 25):
    rows[i]["city"] = ""                          # missing city
for i in random.sample(range(10_000), 15):
    rows[i]["phone"] = "12345"                    # invalid phone
rows += [dict(rows[i]) for i in random.sample(range(10_000), 30)]   # duplicates

CHECKS = {
    "amount is positive": lambda r: r["amount"].lstrip("-").isdigit() and int(r["amount"]) > 0,
    "city present": lambda r: r["city"] in CITIES,
    "phone is 10 digits": lambda r: re.fullmatch(r"[6-9]\d{9}", r["phone"]) is not None,
}
seen, good, bad, failures = set(), [], [], Counter()
for r in rows:
    problems = [name for name, ok in CHECKS.items() if not ok(r)]
    if r["order_id"] in seen:
        problems.append("order_id unique")
    seen.add(r["order_id"])
    for p in problems:
        failures[p] += 1
    (bad if problems else good).append(r)

print(f"batch rows      : {len(rows):,}")
for name, n in failures.most_common():
    print(f"  FAIL {name:20}: {n}")
print(f"passed          : {len(good):,}  quarantined: {len(bad)}")
fail_rate = len(bad) / len(rows) * 100
print(f"failure rate    : {fail_rate:.2f}%  ->", "PUBLISH" if fail_rate < 2 else "STOP the pipeline")

SALT = "keep-this-secret"       # in real life: from a secrets manager, never in code
def pseudonym(value):
    return hashlib.sha256((SALT + value).encode()).hexdigest()[:12]
def mask_phone(p):
    return "XXXXXX" + p[-4:]

with open("orders_clean.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=["order_id", "customer_key", "phone_masked", "city", "amount", "order_date"])
    w.writeheader()
    for r in good:
        w.writerow({"order_id": r["order_id"], "customer_key": pseudonym(r["email"]), "phone_masked": mask_phone(r["phone"]),
                    "city": r["city"], "amount": r["amount"], "order_date": r["order_date"]})
print("\nwrote orders_clean.csv (no raw email or phone). First row:")
print(open("orders_clean.csv").read().splitlines()[1])

# In a terminal:  python quality_guard.py

A quality check, one row at a time

Four amounts are checked against a valid range. Watch the pass and fail lists fill up.

Worked examples: real problems, solved and traced in memory

Four data-quality and privacy tasks from a real platform. Each runs live and is traced.

4 rows with name, city, phone

name : 25% missing
city : 50% missing
phone: 25% missing

7 daily medians and today’s median

baseline 322, today 32150, ratio 100x -> ALERT

catalog dictionary

customers: ['name', 'phone']
support_tickets: ['email', 'message']

tables keyed by customer_key, key "97c8"

deleted: {'orders': 2, 'features': 1, 'crm_export': 0} | remaining orders: [('a1b2', 150)]

Check yourself

Quick revision

Interview and viva questions

Practice problems

Data-quality, masking and privacy exercises. The last two are business tasks.

list

2

print([12, None, 7, None, 3].count(None))

list of IDs

['a', 'b']

from collections import Counter
print([k for k, v in Counter(["a", "b", "a", "c", "b", "a"]).items() if v > 1])

9876543210

XXXXXX3210

p = "9876543210"
print("X" * 6 + p[-4:])

4 strings

['9876543210', '6123456789']

import re
print([p for p in ["9876543210", "12345", "6123456789", "5123456789"] if re.fullmatch(r"[6-9]\d{9}", p)])

two totals

-125 MISMATCH

d = 15_43_164 - 15_43_289
print(d, "OK" if abs(d) <= 100 else "MISMATCH")

email

d17065027e

import hashlib
print(hashlib.sha256(("salt" + "asha@example.com").encode()).hexdigest()[:10])

ages in days

[400, 800]

print([a for a in [12, 400, 365, 800, 90] if a > 365])

1,850 × Rs 240

Rs 444,000 per month, Rs 5,328,000 per year

m = 1_850 * 240
print(f"Rs {m:,} per month, Rs {m * 12:,} per year")

46/50

92% below target

s = 46 / 50
print(f"{s:.0%}", "meets target" if s >= 0.95 else "below target")

Real-world scenario questions (asked in company interviews)

Quality, security and privacy questions from real projects and interviews. Decide your answer first, then open the model answer.