Don't Let Your System Decide It's Dead

Distributed systems fail in two ways: the failure itself, and the system's automatic response to the failure. The second is often worse. The root cause is systems making semantic decisions with insufficient information—promoting "slow" to "dead" based on wall-clock constants rather than measured resource models. This paper identifies a precise invariant—the recovery invariant—that separates safe automatic recovery from cascading failure, and a design principle—the compensation boundary—that determines who should make which decisions. The recovery invariant has three levels: individual cost boundedness, aggregate rate boundedness, and feedback stability (spectral radius of the recovery gain matrix less than one). The compensation boundary is the architectural point where a system stops surfacing failure to a higher authority and starts acting on it automatically. We analyze TCP's retransmission timeout as a reference design that satisfies all three levels, and show how Cassandra's hinted handoff, UNIX swap and OOM killer, and wall-clock-based tombstone garbage collection violate them. We connect this framework to the metastable failures literature, resilience engineering, and control theory, and apply the principles to CRDT tombstone garbage collection as a case study.