Evaluates GPU failure patterns in supercomputers, indicating cooling method impact.
On the Titan supercomputer, where the coolant touching each GPU is air that climbs the cabinet, a card's physical position predicts when it dies. Across 18,837 archived lifetimes, cards in the top cage failed at 10.75 times the hazard of cards in the bottom cage, and combining cabinet height with floor position widens the gap to a factor of 27. This is a replication: Ostrouchov et al. reported the same dependence on heat dissipation and cooling architecture, and the direction and mechanism they describe are the ones we recover. The contribution here is what comes next. If the mechanism is absorption capacity, then the law it instantiates, stability as load over absorption capacity, forbids something specific: make absorption capacity uniform across the cabinet and the spatial signal must disappear. On Summit, every GPU has its own cold plate fed with water from a rack manifold. Height does not change what the chip is cooled by. The spatial signal disappeared. Across 23,520 GPUs, cabinet height, GPU slot within the node, floor column, floor row, and cabinet identity all return nothing (p = 0.43, 0.53, 0.95, 0.52, 0.078). No lifetime power or temperature aggregate survives correction for multiple testing. We also report, in full, four of our own findings that did not survive our own controls. Three were spatial or thermal artifacts of counting the same defective cards repeatedly, or of choosing which cabinet levels to test after seeing the data. The fourth is the one that hurt: a strong apparent thermal-variance precursor before memory failure, up to five times the normal one-minute temperature fluctuation, surviving Bonferroni correction over 75 metrics. It matched a theory we already held. Matching both groups on GPU activity shrank it from 5x to 1.6x and it no longer survives. With only 38 actively-loaded failing cards, that control cannot separate "no effect" from "a modest effect we cannot resolve," so we report the precursor as unproven rather than absent, and flag it as the strongest open question in this data.
No takes yet. Share an insight, caveat, or question.
Oren Speiser (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: