Structured, semi-structured and unstructured data

Six files, and only one opens in Excel

Structured, semi-structured and unstructured data

Words you’ll meet in this lesson

Common formats of each shape

Three shapes of data at three levels

Three shapes, compared

Side by side

Where data comes from

One food order in every shape

1. Structured: one row per order

   import csv, io
   text = "order_id,city,amount\nO1043,Pune,420\nO1044,Mumbai,250\n"
   for row in csv.DictReader(io.StringIO(text)):
       print(row["order_id"], row["city"], int(row["amount"]))

   O1043 Pune 420 O1044 Mumbai 250
   Every row has the same three columns, and the types are known. This is the easiest shape: load it into a database and query it with SQL.

2. Semi-structured: an order with a list inside

   import json
   event = json.loads('{"order_id": "O1043", "city": "Pune", "items": [{"dish": "thali", "qty": 1, "price": 220}, {"dish": "lassi", "qty": 2, "price": 100}]}')
   for item in event["items"]:
       print(event["order_id"], item["dish"], item["qty"] * item["price"])

   O1043 thali 220 O1043 lassi 200
   The JSON carries its own field names and contains a nested items list. Looping over the list flattens it into one row per item, ready for a table.

3. Semi-structured: a log line with a pattern

   line = "2026-10-19 13:02:11 rider=R77 order=O1043 lat=18.5204 lon=73.8567 status=PICKED"
   parts = line.split()
   fields = dict(p.split("=") for p in parts[2:])
   print(parts[0], parts[1], fields["order"], fields["status"])

   2026-10-19 13:02:11 O1043 PICKED
   A log line is text, but it follows a pattern: date, time, then key=value pairs. Splitting by spaces and = turns it into fields.

4. Unstructured: a free-text review

   review = "Thali was great but the delivery was very late and the lassi was warm"
   words = review.lower().split()
   labels = [w for w in ["late", "cold", "warm", "great", "spilled"] if w in words]
   print("labels:", labels)

   labels: ['late', 'warm', 'great']
   The review has no fields. Here a simple keyword list extracts labels; real systems use language models to detect sentiment and topics. The result is structured: order_id, label.

5. Unstructured: a photo of the food

   A 3 MB photo arrives with the review. It is stored as a file in object storage, and a row of metadata is saved: file path, size, upload time, order id. An image model can add labels such as “thali, spilled”.

   The photo itself never goes into a table. Its path and labels do, so analysts can count “spilled food” complaints by kitchen.

6. Link everything by order id

   Now all shapes can be joined: the order row, its items from JSON, the rider’s pickup time from logs, the review labels and the photo metadata, all keyed by O1043.

   This is the daily work of a data engineer: bring every shape into a structured, joinable form without losing the raw originals.

One order: what Riya sees, and what the data looks like

Step through the six frames. Left: Riya ordering food. Right: the shape of the data each moment creates.

Real-time scenarios: working with every shape

Three live scenarios: flattening nested JSON into a table, turning a flood of support chats into numbers during an outage, and an advanced one about an app update that silently changes the data.

Flatten a nested order into a table

TiffinGo’s app sends every order as nested JSON. The finance dashboard needs a flat table. Watch the three steps every pipeline uses, and the classic mistake to avoid at the end.

Outage night: turning 1,000 chats a minute into numbers

At 20:14 support chats explode. Nobody can read a thousand messages a minute. Watch unstructured text become a structured count that tells engineers exactly what broke.

Schema drift: the app update that broke the dashboard

Semi-structured data is flexible, which is also its danger. Watch a small app change silently break a pipeline, and the three defences real teams use.

Where each shape shows up at work

Sort records by shape, and read each one

Four records arrive at a data platform. The program decides the shape of each and prints a short summary. Then try the challenges: flatten nested JSON, parse a log line, label free text and catch schema drift.

import json

records = [
    ("table_row", ("O1043", "Pune", 420)),
    ("json", '{"order_id": "O1043", "items": [{"dish": "thali"}, {"dish": "lassi"}]}'),
    ("log", "13:02:11 rider=R77 status=PICKED"),
    ("text", "Thali was great but the lassi was warm"),
]

for kind, value in records:
    if kind == "table_row":
        print("STRUCTURED     :", value, "-> 3 fixed columns")
    elif kind == "json":
        event = json.loads(value)
        print("SEMI-STRUCTURED:", list(event), "-> keys inside the record,", len(event["items"]), "nested items")
    elif kind == "log":
        fields = dict(p.split("=") for p in value.split()[1:])
        print("SEMI-STRUCTURED:", fields, "-> parsed from a pattern")
    else:
        print("UNSTRUCTURED   :", len(value.split()), "words, no fields until we extract some")

• event = json.loads(value)json.loads turns JSON text into Python dictionaries and lists. The field names come from the data itself.
• fields = dict(p.split("=") for p in value.split()[1:])Split the line by spaces, skip the time, then split each key=value pair. This is log parsing.
• print("UNSTRUCTURED :", len(value.split()), "words, no fields until we extract some")Plain text has no fields. All we can do without extra work is count words.

Run it on your PC: create and read all three shapes

This program creates a sample_data folder with a CSV (structured), a JSON-lines file and a log file (semi-structured) and a text file of reviews (unstructured). Then it reads each one the right way: csv, json, a regular expression and simple text analysis. Only the Python standard library is used. Save it as three_shapes.py in an empty folder and run it, then open the files it creates.

"""Create one sample file of each shape in a folder, then read each one the right way."""
import csv, json, os, re

os.makedirs("sample_data", exist_ok=True)

# 1. Structured: a CSV with a fixed header
with open("sample_data/orders.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["order_id", "city", "amount"])
    w.writerows([["O1", "Pune", 420], ["O2", "Mumbai", 250], ["O3", "Pune", 180]])

# 2. Semi-structured: JSON lines with nesting and a missing key
events = [
    {"order_id": "O1", "items": [{"dish": "thali", "qty": 1}, {"dish": "lassi", "qty": 2}]},
    {"order_id": "O2", "items": [{"dish": "dosa", "qty": 1}], "coupon": "LUNCH20"},
]
with open("sample_data/events.jsonl", "w") as f:
    for e in events:
        f.write(json.dumps(e) + "\n")

# 3. Semi-structured: log lines with a pattern
with open("sample_data/app.log", "w") as f:
    f.write("2026-10-19 13:02:11 INFO  order=O1 status=PLACED ms=120\n")
    f.write("2026-10-19 13:02:15 ERROR order=O2 status=PAY_FAILED ms=3020\n")
    f.write("2026-10-19 13:02:19 INFO  order=O3 status=PLACED ms=95\n")

# 4. Unstructured: free-text reviews
with open("sample_data/reviews.txt", "w") as f:
    f.write("Loved the thali, will order again!\n")
    f.write("Food was cold and delivery very late.\n")

print("Files created:", sorted(os.listdir("sample_data")))

# ---- read each one the right way
with open("sample_data/orders.csv") as f:
    rows = list(csv.DictReader(f))
total = sum(int(r["amount"]) for r in rows)
print(f"\n[structured] {len(rows)} rows, columns {list(rows[0])}, total Rs {total}")

flat = []
with open("sample_data/events.jsonl") as f:
    for line in f:
        e = json.loads(line)
        for item in e["items"]:
            flat.append((e["order_id"], item["dish"], item["qty"], e.get("coupon")))
print(f"[semi-structured JSON] flattened into {len(flat)} item rows:")
for r in flat:
    print("   ", r)

pattern = re.compile(r"(\S+ \S+) (\w+)\s+order=(\w+) status=(\w+) ms=(\d+)")
with open("sample_data/app.log") as f:
    parsed = [pattern.match(line).groups() for line in f]
errors = [p for p in parsed if p[1] == "ERROR"]
print(f"[semi-structured log] {len(parsed)} lines parsed, {len(errors)} error: {errors[0][2]} {errors[0][3]}")

negative = {"cold", "late", "spilled", "wrong"}
with open("sample_data/reviews.txt") as f:
    for line in f:
        words = set(re.findall(r"[a-z]+", line.lower()))
        label = "complaint" if words & negative else "praise"
        print(f"[unstructured] {label:9} <- {line.strip()}")

# In a terminal:  python three_shapes.py

Flattening a nested order, one line at a time

An order with two items becomes two flat rows. Watch the inner loop run once per item and the rows list grow, with the order id repeated on each row.

Worked examples: real problems, solved and traced in memory

Four tasks a data engineer does with mixed data, from sorting files by shape to catching schema drift. Each runs live and is traced.

sales.csv, events.json, app.log, cctv.mp4, invoice.pdf, stock.parquet, review.txt, config.xml

{'structured': 2, 'semi-structured': 3, 'unstructured': 3}

2 orders with 3 items in total

{'thali': 660, 'lassi': 200}

6 log lines over 2 minutes

13:02 3 requests, 33 % errors
13:03 3 requests, 67 % errors

expected: order_id (str), amount (int), city (str) · 4 events

Accepted   : ['O1', 'O4']
Quarantined: ['O2', 'O3']
Problems   : {'bad type amount': 1, 'missing city': 1}

Check yourself

Quick revision

Interview and viva questions

Practice problems

Exercises on reading, flattening and validating mixed data. The last two are business tasks.

{"id": "O1", "city": "Pune", "amount": 420}

['id', 'city', 'amount'] 3

import json
o = json.loads('{"id": "O1", "city": "Pune", "amount": 420}')
print(list(o), len(o))

(1, "Pune"), {"a": 1}, "nice food"

(1, 'Pune') -> structured row
{'a': 1} -> semi-structured
nice food -> unstructured

for v in [(1, "Pune"), {"a": 1}, "nice food"]:
    if isinstance(v, tuple):
        print(v, "-> structured row")
    elif isinstance(v, dict):
        print(v, "-> semi-structured")
    else:
        print(v, "-> unstructured")

id,amount / O1,120 / O2,80

Total: 200

import csv, io
text = "id,amount\nO1,120\nO2,80\n"
rows = list(csv.DictReader(io.StringIO(text)))
print("Total:", sum(int(r["amount"]) for r in rows))

{"id": "O5", "items": [{"dish": "dosa", "qty": 2}, {"dish": "chai", "qty": 1}]}

O5 dosa 2
O5 chai 1

order = {"id": "O5", "items": [{"dish": "dosa", "qty": 2}, {"dish": "chai", "qty": 1}]}
for it in order["items"]:
    print(order["id"], it["dish"], it["qty"])

3 lines

['R1', 'R3']

lines = ["rider=R1 status=PICKED", "rider=R2 status=WAITING", "rider=R3 status=PICKED"]
picked = []
for ln in lines:
    f = dict(p.split("=") for p in ln.split())
    if f["status"] == "PICKED":
        picked.append(f["rider"])
print(picked)

v1: id, amount, city · v2: id, amount, city_name, coupon

Common : ['amount', 'id']
Only v1: ['city']
Only v2: ['city_name', 'coupon']

v1 = {"id", "amount", "city"}
v2 = {"id", "amount", "city_name", "coupon"}
print("Common :", sorted(v1 & v2))
print("Only v1:", sorted(v1 - v2))
print("Only v2:", sorted(v2 - v1))

4 records

id      0% missing
city    50% missing
coupon  75% missing

records = [{"id": 1, "city": "Pune", "coupon": None}, {"id": 2, "city": None},
           {"id": 3, "city": "Thane", "coupon": "X"}, {"id": 4}]
fields = ["id", "city", "coupon"]
for f in fields:
    missing = sum(1 for r in records if r.get(f) is None)
    print(f"{f:7} {missing / len(records) * 100:.0f}% missing")

2, 30, 400 TB

structured         0.5%
semi-structured    6.9%
unstructured      92.6%

sizes = {"structured": 2, "semi-structured": 30, "unstructured": 400}
total = sum(sizes.values())
for k, v in sizes.items():
    print(f"{k:16} {v / total * 100:5.1f}%")

late 420, cold 180, wrong item 90, praise 1310

late       61%
cold       26%
wrong item 13%

labels = {"late": 420, "cold": 180, "wrong item": 90, "praise": 1310}
complaints = {k: v for k, v in labels.items() if k != "praise"}
total = sum(complaints.values())
for k, v in sorted(complaints.items(), key=lambda x: -x[1]):
    print(f"{k:10} {v / total * 100:.0f}%")

Real-world scenario questions (asked in company interviews)

Situations with mixed data that come up in real teams and interviews. Decide your answer first, then open the model answer.