Memory Hierarchy: CPU, RAM, Disk, Cache

Modern CPU cores run at clock frequencies between 3.5 GHz and 5.5 GHz, completing single-cycle arithmetic operations in approximately 0.2 to 0.3 nanoseconds. However, electrical signals traveling across motherboard copper traces to main memory (DRAM) face significant physical and protocol delays, requiring approximately 100 nanoseconds to fetch data.

If a modern processor had to communicate directly with DRAM on every instruction, the execution pipeline would stall for 300 to 500 clock cycles on every single memory reference. This massive performance gap is known in computer architecture as the "Memory Wall".

To prevent CPU execution units from idling, hardware engineers stack a hierarchy of progressively smaller, faster, and more expensive SRAM (Static RAM) caches directly on the processor die. Each tier acts as a high-speed staging buffer for the tier below it:

• Registers: Zero-cycle to half-nanosecond direct operand access for CPU Arithmetic Logic Units (ALUs).
• L1 Cache (Level 1): Split into L1 Instruction Cache (L1i) and L1 Data Cache (L1d). Operates at ~1ns.
• L2 Cache (Level 2): Dedicated per-core SRAM buffer (~3-4ns).
• L3 Cache (Level 3 / LLC): Last Level Cache shared across all processor cores on the socket (~15-40ns).
• Main Memory (DRAM): High-density volatile capacitive memory connected via DDR channels (~100ns).
• Non-Volatile Storage (NVMe SSD / HDD): Block devices communicating over PCIe or SATA buses (microseconds to milliseconds).

Data is never transferred between DRAM and CPU caches as individual bytes. Instead, memory controllers fetch fixed-size chunks called Cache Lines, almost universally 64 bytes in size on modern x86 and ARM64 architectures.

Software performance in high-scale distributed systems is fundamentally governed by the Principle of Locality:

1. Temporal Locality: If a particular memory location is accessed, it is highly likely to be accessed again in the near future (e.g., loop counter variables, accumulator totals, frequently accessed hash map buckets).
2. Spatial Locality: If a particular memory location is accessed, memory locations with adjacent addresses are highly likely to be accessed soon (e.g., sequentially traversing an array or struct buffer).

When a program iterates over a contiguous array of 64-bit integers (8 bytes each), accessing the first element loads the entire 64-byte cache line from RAM into L1/L2/L3. The subsequent 7 integer accesses are guaranteed L1 cache hits (0.5-1ns), executing at hardware wire speed. In contrast, traversing a linked list scattered across random heap allocations incurs an LLC cache miss and DRAM latency (~100ns) on virtually every single pointer dereference.

Enterprise cloud servers (e.g., AWS c6i.32xlarge, bare-metal database nodes) use multi-socket motherboards with 2, 4, or 8 physical CPU processors. In these systems, memory is partitioned physically across sockets in a NUMA architecture:

• Local Memory Access: When Core 0 on Socket 0 accesses DRAM directly wired to Socket 0's memory controller, memory latency is ~80-90ns.
• Remote Memory Access: When Core 0 on Socket 0 must access DRAM wired to Socket 1, the request must traverse an inter-socket bus (Intel UPI / AMD Infinity Fabric), doubling latency to ~160-220ns.

High-throughput distributed systems (PostgreSQL, Redis, Kafka, Cassandra) must be NUMA-aware. Misconfigured thread placement can lead to random 2x-3x p99 latency spikes because worker threads continuously fetch remote NUMA pages.

# Check NUMA topology on a database server
numactl --hardware

# Output shows memory split across nodes:
# node 0 cpus: 0 1 2 3 4 5 6 7
# node 0 size: 65536 MB
# node 1 cpus: 8 9 10 11 12 13 14 15
# node 1 size: 65536 MB

# Run high-performance Redis instance pinned to single NUMA node
numactl --cpunodebind=0 --membind=0 redis-server /etc/redis/redis.conf

Understanding the memory hierarchy dictates major architectural decisions when designing databases, caches, and stream processors:

1. Columnar Storage Formats (Parquet, ClickHouse): Storing data column-by-column rather than row-by-row guarantees that scanning a single column (e.g., user_age) reads contiguous memory addresses, achieving 100% spatial locality and enabling SIMD (Single Instruction, Multiple Data) vectorized execution.
2. In-Memory Caches (Redis, Memcached): Keeping active datasets in DRAM bypasses OS storage stacks, eliminating context switches and disk block transfers to achieve sub-millisecond latencies.
3. Log-Structured Merge Trees (LSM Trees in RocksDB/Cassandra): Converting random disk writes into sequential append-only writes maximizes drive controller buffer utilization and sequential write bandwidth.