Answers are grounded in this paper and cite their sources.
A language model whose attention keeps a fixed-size state can read an input of any length in one pass, with memory that does not grow with the input. That property is often reported as a long context. We separate it into three claims and test each one on byte-level PG-19. The models are fully complex-valued, built in the CNet framework, and use linear Born attention, | |^2 evaluated through a density-matrix state, with a fixed per-head decay and rotary positions. A model with 12M complex parameters (23.9M real-scalar equivalents) trained in strict FP32 on half a pass of the full 11.7 GB corpus scores 1.312 bits per byte on the PG-19 test partition when each 5,242,880-byte window is read in one unbroken pass with 12.8 MB of state; the MBLM hierarchy reports 2.448 at that window length, an uncontrolled comparison, and 1.240 at 98k bytes. Streaming is stable, but we find little aggregate benefit from persistence beyond a few thousand bytes. A reset sweep on identical validation bytes shows that carried state helps by 0.012 bits per byte for the 12M model and 0.006 for a 4M model, and both curves follow a fixed-cost-per-reset model, ΔL(C) = ΔL(∞)(1 − 4096/C), with residuals below 4e-5. This is consistent with most of the gain coming from state recovery after a reset. Extending the reset interval beyond 256k bytes improves the loss by about 1.8e-4 bits per byte, as the fixed-cost model predicts; the residual is below 4e-5, but an aggregate-loss experiment does not isolate the contribution of long-range information, and retrieval was not measured. The fixed decay suppresses old contributions exponentially, with half-lives of 22 to 709 bytes per head. A real-valued Transformer with the same parameter count, bytes and schedule, scored on the same windows, reaches 1.245 bits per byte on test against 1.324 for the Born model with the same 4,096-byte resets. In a matched five-configuration experiment at 10,000 steps in PyTorch, the same complex Born design achieves the lowest test loss, 1.413 against 1.495 for the Transformer, while Born attention placed in the real Transformer costs 0.017: the relative performance of Born and softmax depends on the representation and read-out around them. CNet's default initialization, which we show to be poorly conditioned, accounts for 0.023 of the original gap: with a Gaussian start the same design reaches 1.301 after 60,000 steps, and the Transformer still leads at full length (1.245). The ranking depends on the training budget: the complex Born design is better with a 10,000-step schedule and worse with a 60,000-step one. Constant-state streaming is a property of the method; long effective memory is not; neither is quality at equal size at full training length. Preprint, October 2026. Companion to "Born Attention is Linear" (doi:10.5281/zenodo.23685041). Code: the open-source CNet framework.
No takes yet. Share an insight, caveat, or question.
Marcel Crâşmaru (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: