Attending to Memory Writes Reduces Seed Variability in Recurrent Length Generalization

Recurrent memory requires learning useful updates, but attention added to assist learning can also compromise length generalization. We study bounded delta attention: causal attention over uncorrected update proposals whose bounded correction enters the recurrent update. Its implementation in TraceRelay combines additive phase recurrence with attention computed before a parallel scan. On prefix-supervised modulo-5, trained at lengths 1--40, three findings characterize its behavior at length 80. First, upper delta attention attains 70.81±8.26% accuracy across seven seeds, versus 69.21±22.42% for the local-attention recurrent baseline (mean ± sample SD): a 63.2% reduction in SD with a similar mean. Upper state attention also reduces variation, reaching 74.75±6.24%, so this benefit is not unique to update-level attention. Disabling the upper delta correction retains 70.62±8.60%. Second, blocking gradients at the delta-attention inputs changes accuracy to 65.25±12.40% in the matched seven-seed comparison, increasing SD by 50.2%. It also delays a 99% validation threshold on average among the same six seeds that reach it; the 95% threshold is reached earlier on average with input-gradient blocking across all seven seeds. Third, lower delta attention exhibits substantially less extrapolation degradation than lower state attention: 62.60% versus 19.40%, against a 71.06% baseline. Removing its correction reduces accuracy to 22.56%; its advantage over lower state attention also persists under input-gradient blocking. These observations motivate bounded update-level attention as a recurrent learning scaffold, combining reduced seed variation with less severe degradation in a placement where state attention fails.