MyPhatshala

Big Data Fundamentals

What 'big data' really means, how data is stored and spread across machines, and how the whole ecosystem fits together.

Beginner to Advanced7 modules25 lessonsFree · no sign-up
25 of 25 lessons published
Start the course

Syllabus

01 IntroductionThe big data problem and where it came from. 4 lessons
  1. 1.1What is big data? The 5 Vs explained28 minStart
  2. 1.2Why one computer is not enough for big data28 minStart
  3. 1.3Distributed computing basics for big data30 minStart
  4. 1.4A short history of big data: from one server to the lakehouse25 minStart
02 Data and StorageWhat big data looks like and where it lives. 4 lessons
  1. 2.1Structured, semi-structured and unstructured data28 minStart
  2. 2.2File formats for big data: CSV, JSON, Avro, Parquet and ORC30 minStart
  3. 2.3Storage for big data: block, file and object storage (and HDFS)28 minStart
  4. 2.4NoSQL databases: key-value, document, wide-column and graph30 minStart
03 Distributed Systems ConceptsThe ideas every big data tool is built on. 4 lessons
  1. 3.1Scaling up vs scaling out (and back-of-envelope estimation)30 minStart
  2. 3.2Partitioning and sharding: splitting data across machines30 minStart
  3. 3.3Replication: leaders, followers and copies30 minStart
  4. 3.4The CAP theorem and consistency models30 minStart
04 The EcosystemThe tools, the architectures and how they fit. 7 lessons
  1. 4.1The big data ecosystem map: every tool in its place25 minStart
  2. 4.2Batch vs stream processing: now or later?28 minStart
  3. 4.3Data lake vs data warehouse (and the lakehouse)28 minStart
  4. 4.4Lambda and Kappa architectures explained28 minStart
  5. 4.5Data pipelines and orchestration (Apache Airflow basics)30 minStart
  6. 4.6Big data on the cloud: AWS, Azure and GCP30 minStart
  7. 4.7Big data at Indian scale: UPI, IRCTC, live cricket and sale days30 minStart
05 Governance and SecurityTrustworthy, safe and legal data. 1 lessons
  1. 5.1Data quality, governance, security and the DPDP Act30 minStart
06 CareersRoles, roadmap and interviews. 3 lessons
  1. 6.1Data engineer vs data analyst vs data scientist25 minStart
  2. 6.2Data engineering roadmap, portfolio, certifications and resume30 minStart
  3. 6.3Big data and data engineering interview questions with answers40 minStart
07 Capstone ProjectPut it all together. 2 lessons
  1. 7.1Capstone: build a mini data pipeline on your PC45 minStart
  2. 7.2Final test, cheat sheet and exam questions: Big Data Fundamentals45 minStart