Distributed File Systems: HDFS & Google File System (GFS) Architecture

In 2003, Google published "The Google File System" (GFS) (Ghemawat, Gobioff, Leung), which formed the open-source blueprint for the Apache Hadoop Distributed File System (HDFS).

The foundational design assumptions of GFS/HDFS departed radically from traditional enterprise SAN/NAS storage:

1. Component Failures are the Norm, Not the Exception: Across a cluster of 10,000 commodity Linux servers with spinning mechanical hard disks, multiple hard drives, network switches, and motherboards fail daily. The filesystem must detect, tolerate, and heal failures automatically without human intervention.
2. Files are Enormous (Gigabytes to Terabytes): Files are composed of massive multi-gigabyte append-only event streams and web crawls rather than millions of tiny 4KB documents.
3. Write-Once, Read-Many (WORM): Data is written sequentially once and read sequentially millions of times. Random in-place modifications in the middle of a file are disallowed.
4. Decouple Control Flow from Data Flow: The Master/NameNode participates only in metadata coordination; all high-bandwidth data transfers occur directly between the client and worker nodes over TCP streams.