Which Characters Need Context? Measuring Character-Specific Context Gain in Natural Language and Source Code

A character bigram model conditions on only one preceding character, making its contextual limitation unusually transparent: any predictive information available from earlier characters is necessarily unavailable to the bigram. This paper asks not whether longer context helps in aggregate — that is already established — but which individual characters benefit, by how much, and at what context length? We define Character Context Gain CG_x(k; D) = S_x(1; D) − S_x(k; D), where S_x(k; D) is the target-character mean surprisal — the expected negative log-probability of character x given k characters of preceding context in corpus D. We measure it per individual character, then classify characters afterward, avoiding the apples-to-oranges comparison that arises from mixing singleton structural symbols with broad alphabetic aggregates. Across three corpora — tinyshakespeare (NL1), Pride and Prejudice (NL2), Python 3.12 stdlib (Code1) — we fit a panel regression with a log₂(k) × Structural interaction term. The central quantity is β₃: a positive value means structural symbols (punctuation, delimiters) accumulate context gain faster per doubling of context than lexical symbols do, after controlling for character frequency. NL2 (Pride and Prejudice) provides the strongest inferential finding: β₃ = +1.131 (cluster-robust p = 0.002, permutation p = 0.001), stable across coverage thresholds and functional forms. NL1 (Shakespeare) is directional: β₃ = +0.551 (cluster-robust p = 0.033, permutation p = 0.093, does not survive the stricter 75% coverage threshold). Code1 (Python stdlib) is null for the slope (β₃ = −0.024, p = 0.930) but shows an exploratory peak-gain difference (Mann-Whitney p = 0.022, does not survive Bonferroni), motivating a distinction between the trajectory and magnitude of context dependence. Python tokenization reveals that 42% of Python stdlib characters fall inside string literals or comments; tokenizer stratification is required when interpreting punctuation specifically as executable syntax operators. The reliable n-gram range is corpus-specific: k ≤ 7 for NL1, k ≤ 8 for NL2, k ≤ 10 for Code1. Code and data: https://github.com/asbassan/char-context-gain