Answers are grounded in this paper and cite their sources.
Quantifies HBM traffic for LLaMA-3 inference, indicating potential traffic reductions up to 95%.
Version 2 note: This version supersedes v1 ("Non-Lingering Activations: A Temporal Decay Compute Fabric for Transformer Inference"). The central claim has been narrowed from a proposed physical mechanism to a quantitative analysis of an existing, already-deployed architectural principle. The decay-based circuit mechanism of v1 is retained only as an explicitly unverified future direction (Appendix A); nothing in the main results depends on it. Intermediate activations in Transformer inference are produced by one sublayer, consumed once by the immediately following sublayer, and never read again. This single-producer, single-consumer property follows directly from the operational semantics of the forward pass — it requires no additional physical postulate. Despite this, current accelerator memory hierarchies default to routing these tensors through HBM, incurring a round-trip write and read for data that has no functional need to leave the chip. Dataflow accelerators — the Google TPU, Groq LPU, and Cerebras WSE — already forward such single-consumer intermediates through on-chip interconnect without HBM mediation, in production, today. What has not been established is how much HBM traffic this class of tensor accounts for at the scale of a modern large langmodel, and how that quantity scales with batch size and sequence le conditions. We answer this question for LLaMA-3 70B. We derive a closed-form byte-traffic model for the five cross-sublayer tensors that remain HBM-resident even after FlashAttention, validate the model's tensor-counting methodology agrward pass (relative error 0), and apply the validated counter tothe full LLaMA-3 70B configuration at H100-class HBM3 bandwidth (3.35 TB/s). At batch size 32, sequence length 512, cross-sublayer activation traffic is 322 GB against an irreducible weight-traffic floor of 137 GB; eliminating it via direct forwarding reduces total HBM traffic by 70%. At batch size 1, sequence length 4,096, the reduction is 37%. The reduction scales with the product e length, reaching 90% at B=32, S=2,048 and 95% at B=64, S=2,048 —the regime that characterizes current production serving. The result is robust across baseline kernel-fusion assumptions, remaining above 48% even under aggressive fusion. We report an upper bound on recoverable latency (96 ms per forward pass at peak bandwidth for the primary scenario) rather than a measured speedup, since the fraction of this time that overlaps with compute depends on implementation details outside the scope of this analysis. We do not propose a new physical mechanism; the traffic elimination follows from a topological design choice — ensors through HBM — that is already realized in existingaccelerator architectures. A decay-based physical realization that could extend this principle to reconfigurable interconnects where full topological hard-wiring is unavailable (cf. the Temporal Decay Inequality, doi:10.5281/zenodo.20471664) is discussed in Appendix A, and its circuit-level robustness is explicitly noted as unverified.
No takes yet. Share an insight, caveat, or question.
JULGI KANG (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: