What question did this study set out to answer?

To determine what controls the performance and stability of orthogonalised momentum methods when subjected to noisy gradients and inexact matrix orthogonalisation in transformer training.

August 19, 2026Open Access

The Spectrum, Not the Norm: Noise and Inexactness in Orthogonalised Steepest Descent

View Full Paper

Ask AI

Bookmark

Authors

SSShashvat Singham

Discussion

Member takes

Implication

Theoretical and empirical analysis demonstrates how gradient spectra govern noise resilience in orthogonalised steepest descent, indicating standard nuclear-norm bounds are overly pessimistic.

Key Points

To determine what controls the performance and stability of orthogonalised momentum methods when subjected to noisy gradients and inexact matrix orthogonalisation in transformer training.
Derived analytical guarantees linking nuclear-norm stationarity to Frobenius measures via gradient stable rank and curvature ratios.
Formulated an inexact-oracle theorem for Newton-Schulz iterations and refined operator-norm noise bounds for isotropic noise against singular-value spectra.
Evaluated theoretical bounds and empirical gradient alignments on a pretrained 0.5B-parameter transformer, synthetic matrices, controlled quadratics, and character-level models.
Demonstrated that efficiency under inexact Newton-Schulz iterations is non-monotone across steps and fundamentally depends on the gradient spectrum.
Observed that matrices with flat singular-value spectra retain 94% alignment at high noise (||N||_* = 4||G||_*), whereas a 1/i spectrum falls to 49%.
Showed that worst-case nuclear-norm bounds were negative and vacuous across 100% of tested 0.5B transformer gradient matrices, whereas a spectrum-aware noise-floor model predicted alignment with an RMSE of 0.080 versus 4.487.

Cite This Study

Shashvat Singham (2026) studied this question.

synapsesocial.com/papers/6a8563c403308d306e2d71ab https://doi.org/10.5281/zenodo.21976342

View Full Paper

Ask AI

Bookmark

Implication

Theoretical and empirical analysis demonstrates how gradient spectra govern noise resilience in orthogonalised steepest descent, indicating standard nuclear-norm bounds are overly pessimistic.

The Spectrum, Not the Norm: Noise and Inexactness in Orthogonalised Steepest Descent

Discussion

Member takes

Implication

Key Points

Cite This Study

The Spectrum, Not the Norm: Noise and Inexactness in Orthogonalised Steepest Descent

Discussion

Member takes

Implication

Key Points

Cite This Study

Also Consider

Also Consider