MyPhatshala

Apache Hadoop

Store and process huge data across many machines: clusters, HDFS, MapReduce and YARN, from zero to industry level.

Beginner to Advanced14 modules68 lessonsFree · no sign-up
68 of 68 lessons published
Start the course

Syllabus

01 Foundations: before HadoopServers, racks, clusters and why big data needs many machines. 5 lessons
  1. 1.1Server, node, rack and data centre22 minStart
  2. 1.2What is a cluster? Masters and workers25 minStart
  3. 1.3Scale out in practice: how big should a cluster be?26 minStart
  4. 1.4Failures are normal26 minStart
  5. 1.5Move the code, not the data24 minStart
02 Why Hadoop was bornThe web-scale problem and how Hadoop solved it. 4 lessons
  1. 2.1The problem Hadoop solved (and is Hadoop still relevant?)24 minStart
  2. 2.2From Google's papers to Apache Hadoop22 minStart
  3. 2.3How Hadoop solved each problem25 minStart
  4. 2.4Hadoop 1, 2 and 3: what changed25 minStart
03 How one Hadoop cluster worksThe big picture before the details. 4 lessons
  1. 3.1Hadoop architecture: HDFS, YARN, MapReduce and Common26 minStart
  2. 3.2The life of a file and a job, end to end28 minStart
  3. 3.3Node roles: master, worker, edge and gateway24 minStart
  4. 3.4When not to use Hadoop22 minStart
04 SetupInstall and run your own Hadoop cluster. 4 lessons
  1. 4.1Prerequisites: Linux, Java and SSH24 minStart
  2. 4.2Standalone, pseudo-distributed and fully distributed modes24 minStart
  3. 4.3Installing Hadoop: Ubuntu/WSL and a Docker cluster40 minStart
  4. 4.4Configuration files and web UIs28 minStart
05 HDFS architectureHow HDFS stores huge files safely. 6 lessons
  1. 5.1NameNode and DataNode28 minStart
  2. 5.2Blocks (and how they differ from splits)26 minStart
  3. 5.3Replication and rack awareness28 minStart
  4. 5.4fsimage, edit log and checkpointing26 minStart
  5. 5.5Heartbeats and block reports24 minStart
  6. 5.6Safe mode22 minStart
06 HDFS operationsReading, writing and managing files. 5 lessons
  1. 6.1How a file is written: the write pipeline28 minStart
  2. 6.2How a file is read24 minStart
  3. 6.3HDFS shell commands26 minStart
  4. 6.4Admin commands: dfsadmin, fsck and balancer28 minStart
  5. 6.5The HDFS Java API26 minStart
07 HDFS advancedAvailability, scale and efficiency. 5 lessons
  1. 7.1NameNode high availability (QJM and ZooKeeper)30 minStart
  2. 7.2HDFS federation26 minStart
  3. 7.3Erasure coding28 minStart
  4. 7.4Snapshots, quotas and trash26 minStart
  5. 7.5The small files problem26 minStart
08 MapReduce fundamentalsDistributed processing, from first principles. 6 lessons
  1. 8.1Thinking in MapReduce28 minStart
  2. 8.2Input splits, InputFormat and RecordReader26 minStart
  3. 8.3The Mapper26 minStart
  4. 8.4Shuffle and sort30 minStart
  5. 8.5The Reducer26 minStart
  6. 8.6Word count end to end30 minStart
09 MapReduce in depthPatterns used in real jobs. 8 lessons
  1. 9.1Combiner26 minStart
  2. 9.2Partitioner24 minStart
  3. 9.3Custom Writable types26 minStart
  4. 9.4Counters24 minStart
  5. 9.5Reduce-side join26 minStart
  6. 9.6Map-side join and the distributed cache26 minStart
  7. 9.7Secondary sort28 minStart
  8. 9.8Hadoop Streaming with Python26 minStart
10 Data formats and compressionStore data so jobs run fast. 3 lessons
  1. 10.1Text, SequenceFile and Avro28 minStart
  2. 10.2Parquet and ORC30 minStart
  3. 10.3Compression codecs and splittability26 minStart
11 YARNResource management for the whole cluster. 5 lessons
  1. 11.1YARN architecture: ResourceManager, NodeManager, ApplicationMaster30 minStart
  2. 11.2The life of a YARN application28 minStart
  3. 11.3Schedulers: FIFO, Capacity and Fair30 minStart
  4. 11.4Memory and CPU tuning30 minStart
  5. 11.5Logs and debugging28 minStart
12 Multiple clusters and operationsRunning Hadoop like a company does. 6 lessons
  1. 12.1Why companies run many clusters26 minStart
  2. 12.2DistCp and disaster recovery32 minStart
  3. 12.3Cluster sizing with real numbers32 minStart
  4. 12.4Monitoring and alerts28 minStart
  5. 12.5Security: Kerberos and Ranger30 minStart
  6. 12.6Troubleshooting playbook30 minStart
13 Hadoop in industry todayWhere Hadoop fits in 2026. 4 lessons
  1. 13.1The Hadoop ecosystem: Hive, HBase, ZooKeeper and more28 minStart
  2. 13.2Hadoop vs Spark30 minStart
  3. 13.3Hadoop in the cloud, and Cloudera26 minStart
  4. 13.4A data engineer's day: pipelines, migrations and the lakehouse32 minStart
14 Capstone projectBuild a complete pipeline. 3 lessons
  1. 14.1The problem and the dataset30 minStart
  2. 14.2Build the pipeline34 minStart
  3. 14.3Review and interview preparation34 minStart