Answers are grounded in this paper and cite their sources.
Supervised fine-tuning (SFT) is the standard means of injecting new facts into a language model, but it does not control how strongly, or from which direction, each injected fact will be recalled. We propose Importance-Aligned Normalization (IAN), an objective that injects a per-instance importance scalar w in [0, 1] into the training signal to modulate per-fact recall strength. IAN normalizes each per-sequence gradient to unit norm, multiplies it by w, and adds a bidirectional auxiliary objective that re-reads the same sequence under a bidirectional mask and predicts the subject tokens, with a coefficient proportional to w^2. The data are left unchanged; only the gradient combination rule changes. Injecting 300 facts into Gemma 3 4B, 12B, and 27B and measuring at the fact level, we find that importance modulates reverse recall at all three sizes (Spearman 0.21-0.23 averaged over three importance assignments; uniform SFT is near 0), and that both reverse recall and preservation of pre-existing known facts exceed those of uniform SFT at equal compute. Same-data, equal-compute ablations show that interference suppression comes from per-sequence gradient normalization, the slope comes from w, and the reverse direction requires a fact-level signal rather than KL distillation. The same framework also applies to self-distillation teacher rows.
No takes yet. Share an insight, caveat, or question.
Kwangsu Kim (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: