Local (In-Memory) vs Distributed Cache

When architecting a caching layer, system designers must choose between two distinct physical tiers:

1. Local (In-Process / Heap) Cache (e.g., Caffeine in Java, Go sync.Map, Node.js lru-cache):

• Mechanism: Data is stored directly inside the operating system process memory space (JVM heap or Go process RAM) on the application server.
• Latency: ~ 50 - 100 nanoseconds. There is zero network serialization, zero TCP socket overhead, and zero context switching.
• Limitations: Data is strictly isolated to that single server instance. If you run 100 auto-scaled Kubernetes pods, you have 100 isolated local caches. Scaling pods does not share cache state, and memory consumption directly competes with application business logic heap.

2. Distributed (Remote) Cache (e.g., Redis Cluster, Memcached, Dragonfly):

• Mechanism: A dedicated, standalone cluster of caching servers accessed over the internal datacenter network.
• Latency: ~ 0.5 - 2 milliseconds (dominated by TCP/IP network hops and serialization).
• Advantages: A single unified state shared seamlessly across thousands of ephemeral microservice containers. Adding or killing application pods has zero impact on cache hit rates.