Failure Detection: Heartbeats & Phi Accrual

In distributed systems, detecting when a node has failed is a delicate balancing act between two competing metrics:

1. Completeness: Every truly crashed node must eventually be identified by the cluster so that standby nodes can be promoted and data rebalanced.
2. Accuracy (Speed vs False Positives): The system must avoid falsely declaring a healthy node "dead" due to a transient network blip, router queue spike, or JVM Stop-the-World garbage collection pause.

The Pitfall of Fixed Heartbeat Timeouts:

Traditional systems use a fixed binary timeout: "If no heartbeat arrives within 5 seconds, declare the node dead."

• If the threshold is too aggressive (e.g., 1s), a brief 1.1s GC pause causes false-positive failovers, triggering massive, expensive data resharding storms across the cluster.
• If the threshold is too conservative (e.g., 30s), the cluster wastes half a minute routing traffic to a black-holed server before failing over.