Transformer sequence models encode order through positional signals that are ultimatelytied to absolute indices: sinusoidal and learned embeddings address positions directly, and evenrelative schemes decay or rotate as a fixed function of the index gap. Such signals saturate overlong spans and generalize poorly beyond the training length. We propose Semantic ReferenceFrames (SemRF), an attention bias that represents context and time relative to a small set oflearned semantic anchors rather than to absolute positions. Each token is softly assigned to itsnearest anchor and encoded as a residual offset within that frame; the attention bias between twopositions combines an anchor–frame affinity, a within–frame residual alignment, and a frameconditioned temporal decay whose rate is a learned function of a token’s semantic frame. SemRFstrictly generalizes ALiBi Press et al., 2022 (recovered when the content terms are disabled anddecay is frame–independent), so it inherits ALiBi’s length–extrapolation behaviour while addingcontent–aware structure. The factorization is directly inspectable: trained on enwik8 without any supervision, the anchors organize bytes into recognizable character classes—lowercase,digits, uppercase, whitespace, punctuation, markup—and the frames acquire markedly different learned time constants, with structural tokens decaying slowest and word–internal lettersfastest. The model discovers, from data alone, that structure is long–range and orthographyis local—a concrete realization of decoupling “what” (semantic frame) from “when” (relativetime). This interpretable structure comes with strong benchmarks: across three controlled diagnostics (associative recall, temporal recency, selective copying) and character–level languagemodeling on enwik8, we compare SemRF against seven positional schemes (NoPE, sinusoidal,learned–absolute, RoPE, ALiBi, the T5 relative bias, and CABLE) under matched backbones,parameter budgets, and training compute. On enwik8, SemRF attains the best test bits-percharacter of all eight schemes at the training context (1.4246 vs. 1.4276 for RoPE, 1.4348 forALiBi, and 1.4409 for CABLE) and, unlike rotary and absolute schemes—whose loss degradesseverely beyond the training length—its bpc continues to improve with longer evaluation contexts, reaching 1.340 at 8× the training length, ahead of CABLE (1.352) and ALiBi (1.356)with a margin that widens as context grows. On the diagnostics, SemRF is the only schemethat solves all three tasks at the training length: RoPE and the absolute schemes fail associative recall, ALiBi forms its retrieval circuit in only two of three seeds, and CABLE—whileexcellent on recall—fails selective copying outright. Ablations attribute SemRF’s recency andrecall behaviour causally to the frame-conditioned temporal decay, while the comparison withCABLE delineates an honest trade-off between structured and unstructured content-conditionedbiases on tasks requiring scale-free retrieval across arbitrary gaps. Code and configurations toreproduce every figure and table are released.
G W O Howe (Fri,) studied this question.