What is big data? The 5 Vs explained

The canteen file that Excel could not open

What is big data?

Words you'll meet in this lesson

How big is big? Units of data

Two ways of counting: computers count in powers of 2 (1 KB = 1,024 bytes; the official names are KiB, MiB, GiB), while disk makers count in powers of 10 (1 kB = 1,000 bytes). That is why a “1 TB” disk shows about 931 GB on your computer.

Quick memory trick: each step up is about a thousand times bigger. “Kids Make Great Tea, Papa Enjoys Zero Yelling” gives KB, MB, GB, TB, PB, EB, ZB, YB in order.

The 5 Vs at three levels

The 5 Vs, one by one

Three kinds of data (Variety in detail)

Most estimates say that the large majority of the world's new data is unstructured or semi-structured. That is one big reason traditional table-only databases are not enough. Lesson 2.1 goes deeper into each kind, and Lesson 2.2 into the file formats used to store them.

Is it big data? A quick test

Follow one ₹20 chai payment through the 5 Vs

1. One payment becomes one record

   {"txn_id": "T1001", "payer": "riya@okbank", "payee": "canteen@upi",
    "amount": 20, "time": "2026-10-19 08:01:07", "status": "SUCCESS"}

   This record is about 130 characters. With extra fields that real systems keep (device, location, bank codes), a payment record is a few hundred bytes. Small. On its own, this is not big data.

2. Volume: multiply by everyone

   per_record = 250                    # bytes
   per_month = 10_000_000_000          # 10 billion payments
   print(per_record * per_month / 1024**4, "TB per month")

   2.2737367544323206 TB per month
   About 2.3 TB a month just for the basic records, and banks keep them for years, with copies for safety. Add logs, fraud scores and app clicks, and you are into petabytes.

3. Velocity: they don't arrive evenly

   per_second_avg = 10_000_000_000 / (30 * 24 * 3600)
   print(round(per_second_avg), "payments per second on average")

   3858 payments per second on average
   Nearly 4,000 per second on average. At lunch, on salary day or during a sale, the peak is several times higher. Each one needs a fraud check and a reply in about a second.

4. Variety: the payment is not alone

   Around the same payment, the systems also receive the app's click logs (JSON), an SMS (text), the canteen's menu photo, and Riya's 5-star review with a comment.

   One table cannot hold all of these. Big data platforms store raw files of any format in a data lake, and process each kind with suitable tools.

5. Veracity: some records are wrong

   Riya's network dropped once, so the app retried and the same T1001 arrived twice. Another record has "amount": -20, and one has no payer at all.

   If you sum blindly, the canteen's income is wrong. Pipelines remove duplicates, reject invalid rows and log them for checking. Clean data first, numbers later.

6. Value: turn records into decisions

   After cleaning, patterns appear: chai sells most at 8 am and 4 pm; samosas run out by 1 pm; one account is paying from five cities in ten minutes.

   The canteen stocks more samosas before 1 pm, and the bank blocks the suspicious account. Storing data has a cost; value is what pays for it.

A ₹20 payment: what Riya sees, and what the data platform sees

Step through the six frames. Left: Riya's phone. Right: the same moment for the systems behind it.

Traditional data vs big data

Real-time scenarios: the 5 Vs under pressure

Three live scenarios: velocity breaking a single server on match night, all five Vs lighting up in the first seconds of a midnight sale, and an advanced one about how a record's value falls with age and where it should be stored at each stage.

Cricket final night at a food-delivery app

On a normal evening, orders arrive steadily. During a big cricket final, they come in waves: at the toss, in the innings break, and after the last ball. Watch velocity break a single database, and a big data setup absorb it.

Midnight sale: all five Vs in the first ten minutes

An online shop's big festive sale starts at 00:00. The data team sits in front of a live dashboard. Watch each V light up as crores of shoppers arrive at once.

The life of one payment record: value falls, so cost must fall too

Riya's ₹20 chai payment is stored for years. But it is not equally useful all that time. Follow the record as it ages and see where a sensible platform keeps it at each stage.

Where else the 5 Vs show up at work

Examples are simplified to show where each V matters. Real setups vary by company.

Measure the 5 Vs yourself

A tiny slice of the canteen's data: six payment records as JSON text, two photos and a review. The program measures each V. Read it first, then try the challenges, which include an Excel-limit calculator, a size converter and a record that breaks the program.

import json

# Six payment events, as they arrive from the app (JSON text)
events = [
    '{"id": "T1", "item": "chai", "amount": 20, "sec": 1}',
    '{"id": "T2", "item": "vada pav", "amount": 25, "sec": 1}',
    '{"id": "T3", "item": "chai", "amount": 20, "sec": 2}',
    '{"id": "T2", "item": "vada pav", "amount": 25, "sec": 1}',
    '{"id": "T4", "item": "samosa", "amount": -15, "sec": 2}',
    '{"id": "T5", "item": "chai", "amount": 20, "sec": 3}',
]
data = {"json": events, "image": ["chai.jpg", "menu.png"], "text": ["Best chai!"]}

# VOLUME: how much data?
size = sum(len(e) for e in events)
print("Volume  :", len(events), "records,", size, "bytes")

# VELOCITY: how fast does it arrive?
records = [json.loads(e) for e in events]
seconds = len({r["sec"] for r in records})
print("Velocity:", len(records) / seconds, "records per second")

# VARIETY: how many kinds of data?
print("Variety :", len(data), "kinds ->", ", ".join(data))

# VERACITY: how many records can we trust?
seen, clean, bad = set(), [], 0
for r in records:
    if r["id"] in seen or r["amount"] <= 0:
        bad += 1
        continue
    seen.add(r["id"])
    clean.append(r)
print("Veracity:", bad, "bad records out of", len(records))

# VALUE: what does the clean data tell us?
sales = {}
for r in clean:
    sales[r["item"]] = sales.get(r["item"], 0) + r["amount"]
print("Value   : income Rs", sum(sales.values()), "| best seller:", max(sales, key=sales.get))

• json.loads(e)Turns JSON text into a Python dictionary. Real pipelines do this millions of times per second, which is part of why velocity is hard.
• len({r["sec"] for r in records})A set keeps only unique values, so this counts how many different seconds the records arrived in.
• r["id"] in seenThe retry sent T2 twice. Without this check the income would be counted twice: a veracity problem.
• r["amount"] <= 0A payment of −15 is impossible, so it is rejected instead of silently reducing the total.
• max(sales, key=sales.get)Finds the item with the highest sales: a small example of turning data into a decision (value).

Run it on your PC: make a file Excel can't open

This program writes 15 lakh fake canteen orders to orders.csv (about 39 MB). Save it as gen_orders.py and run it with Python on your own computer. Then try opening the CSV in Excel. The second program reads the same file with Python in about two seconds. That is the point: too big for Excel is not the same as big data.

import csv, os, random, time

random.seed(7)
items = [("chai", 20), ("coffee", 25), ("vada pav", 25), ("samosa", 15), ("poha", 30)]
start = time.time()
with open("orders.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["txn_id", "item", "amount", "mode", "day"])
    for i in range(1, 1_500_001):
        item, price = random.choice(items)
        w.writerow([f"T{i}", item, price, random.choice(["UPI", "cash"]), i // 3700 + 1])
secs = time.time() - start
print("Rows written :", 1_500_000)
print("File size    :", round(os.path.getsize("orders.csv") / 1024**2, 1), "MB")
print("Time taken   :", round(secs, 1), "seconds")
print("Excel limit  :", 1_048_576, "rows -> this file is too big for one sheet")

# In a terminal:  python gen_orders.py

import csv, time
from collections import Counter

start = time.time()
sales = Counter()
with open("orders.csv") as f:
    for row in csv.DictReader(f):
        sales[row["item"]] += int(row["amount"])
print("Read 15 lakh rows in", round(time.time() - start, 1), "seconds")
for item, rs in sales.most_common():
    print(f"{item:10} Rs {rs:,}")

# In a terminal:  python read_orders.py

Cleaning payments, one line at a time

Veracity in action. Four records arrive: one is a duplicate caused by a retry, and one has an impossible amount. Step through and watch seen, income and the console. Objects #1 (the list) and #2–#5 (each record is a tuple) live on the heap.

Worked examples: real problems, solved and traced in memory

Four problems, from beginner to advanced. Each program runs live, and each is traced so you can see how the data builds up in memory.

600 bytes × 400 bills × 365 days × 2000 stores

One store per year : 83.5 MB
Whole chain per year: 163.2 GB

sales.csv, cctv.mp4, app_log.json, invoice.pdf, stock.csv, server.log, review.txt

{'structured': 2, 'unstructured': 3, 'semi-structured': 2}

13:00, 13:00, 13:01, 13:01, 13:01, 13:02

Orders per minute: {'13:00': 2, '13:01': 3, '13:02': 1}
Peak: 13:01 with 3 orders
Average: 2.0 orders per minute

78 GB/day, 40% growth, 3 copies, 4× compression, 5 years

Year 1: raw   27.8 TB -> disk   20.9 TB
Year 2: raw   66.7 TB -> disk   50.0 TB
Year 3: raw  121.2 TB -> disk   90.9 TB
Year 4: raw  197.5 TB -> disk  148.1 TB
Year 5: raw  304.3 TB -> disk  228.2 TB

Check yourself

Quick revision

Interview and viva questions

Practice problems

Small programs about measuring and cleaning data. The last three are business tasks you could meet in a data engineering interview.

5 TB

5120 GB

tb = 5
print(tb * 1024, "GB")

3 log lines

Records: 3
Bytes: 54

logs = ["08:01 INFO login riya", "08:02 INFO pay 20", "08:02 WARN retry"]
print("Records:", len(logs))
print("Bytes:", sum(len(x) for x in logs))

250000

Rows left: 798576

used = 250_000
print("Rows left:", 1_048_576 - used)

T1, T2, T1, T3, T2

['T1', 'T2', 'T3']

orders = ["T1", "T2", "T1", "T3", "T2"]
seen, unique = set(), []
for o in orders:
    if o not in seen:
        seen.add(o)
        unique.append(o)
print(unique)

4 customer records

Missing city: 2 of 4

customers = [
    {"name": "Riya", "city": "Pune"},
    {"name": "Aman", "city": None},
    {"name": "Neha", "city": ""},
    {"name": "Kabir", "city": "Mumbai"},
]
missing = sum(1 for c in customers if not c["city"])
print("Missing city:", missing, "of", len(customers))

[120, 135, 980, 140, 110, 1250, 130]

Average: 409.3
Peak: 1250
Peak-to-average ratio: 3.1

per_minute = [120, 135, 980, 140, 110, 1250, 130]
average = sum(per_minute) / len(per_minute)
peak = max(per_minute)
print("Average:", round(average, 1))
print("Peak:", peak)
print("Peak-to-average ratio:", round(peak / average, 1))

6 records

{'missing id': 1, 'bad amount': 1, 'duplicate': 1}
Quality score: 50% good records

records = [
    {"id": "T1", "amount": 20}, {"id": "T2", "amount": -5},
    {"id": "T1", "amount": 20}, {"id": None, "amount": 30},
    {"id": "T3", "amount": 45}, {"id": "T4", "amount": 15},
]
seen = set()
problems = {"missing id": 0, "bad amount": 0, "duplicate": 0}
good = 0
for r in records:
    if r["id"] is None:
        problems["missing id"] += 1
    elif r["amount"] <= 0:
        problems["bad amount"] += 1
    elif r["id"] in seen:
        problems["duplicate"] += 1
    else:
        seen.add(r["id"])
        good += 1
print(problems)
print(f"Quality score: {good / len(records) * 100:.0f}% good records")

2.3 TB per month, Rs 1.80/GB-month

Month 1 : Rs 4239
Month 12 : Rs 50872

new_gb = 2.3 * 1024
rate = 1.80
for month in [1, 12]:
    stored = new_gb * month
    print("Month", month, ": Rs", round(stored * rate))

18,000,000,000 in 30 days

6944 per second on average

txns = 18_000_000_000
seconds = 30 * 24 * 3600
print(round(txns / seconds), "per second on average")

hours: 8, 9, 13, 13, 16, 13, 8, 16, 13

Busiest hour: 13 with 4 orders

hours = [8, 9, 13, 13, 16, 13, 8, 16, 13]
count = {}
for h in hours:
    count[h] = count.get(h, 0) + 1
busiest = max(count, key=count.get)
print("Busiest hour:", busiest, "with", count[busiest], "orders")

Real-world scenario questions (asked in company interviews)

Situations freshers and junior data engineers really face. Decide your answer first, then open the model answer.