Latency Numbers Every Programmer Should Know

Originally popularized by Google Senior Fellow Jeff Dean and Peter Norvig, these foundational latency benchmarks are the mathematical backbone of all system design estimations. Memorizing these orders of magnitude allows engineers to immediately evaluate whether an architectural proposal is physically feasible before writing code:

Every distributed architectural pattern exists to bypass one of the massive latency cliffs in the hierarchy:

1. Rule 1: RAM is 1,000x Faster than Storage.

   • Fetching a session from Redis (in-memory DRAM: 100ns) is 500x faster than fetching it from an NVMe SSD (50μs) and 100,000x faster than mechanical disk (10ms).
   • This single physical fact explains why multi-tier caching (Redis, Memcached, Application In-Memory Caches) is universal in modern high-throughput architectures.

2. Rule 2: Sequential I/O is 10x to 100x Faster than Random I/O.

   • Reading 1MB sequentially from an NVMe SSD takes ~250μs, whereas performing 250 random 4KB reads takes ~5,000μs (20x slower).
   • On mechanical hard drives, sequential reads run at ~150-200 MB/s, while random seeks achieve less than 1-2 MB/s.
   • Architectural consequence: Databases use Write-Ahead Logs (WAL) and Log-Structured Merge (LSM) Trees (e.g., RocksDB, Cassandra, Kafka) to turn all incoming writes into sequential appends.

3. Rule 3: Network Latency Dominates Everything.

   • Even within the same availability zone, an inter-service RPC (0.5ms) takes 5,000 times longer than reading from local memory (100ns).
   • Across regions (SF to Frankfurt: ~120ms), network propagation speed is bound by the speed of light in fiber optic glass (~200,000 km/s). No amount of CPU optimization can reduce this physical floor.

During architecture design and interview sessions, apply these numbers to establish system feasibility:

Example: Can we design a system handling 50,000 writes/sec to a single MySQL instance?

• If each write performs a random synchronous disk I/O without batching:

Throughput Limit = \frac{1}{Disk Latency} = \frac{1}{0.001 s} = 1,000 IOPS

• Single-instance synchronous random writes will fail catastrophically at 50,000 writes/sec.
• Required Architectural Solution: Introduce a message queue (Kafka) to buffer writes, use in-memory write-back caching (Redis), or employ a distributed LSM-tree database (Cassandra) that writes sequentially to disk.

// ANTI-PATTERN: N+1 Network RPC calls (100 items * 0.5ms RTT = 50ms latency)
for (const id of userIds) {
  const user = await userClient.getUser({ id }); // 100 individual network roundtrips
}

// LATENCY-OPTIMIZED: Single Batch RPC (1 network roundtrip = 0.6ms latency)
const users = await userClient.getUsersBatch({ ids: userIds }); // 80x faster!