What question did this study set out to answer?

The research aims to develop AdamN, an optimizer that accelerates deep learning training by enhancing momentum handling and bias correction.

February 5, 2026Open Access

AdamN: Accelerating Deep Learning Training via Nested Momentum and Exact Bias Handling

Key Points

The research aims to develop AdamN, an optimizer that accelerates deep learning training by enhancing momentum handling and bias correction.
Developed a nested-momentum adaptive optimizer replacing single EMA with compounded EMA.
Implemented exact double-EMA bias correction for smoother gradient memory.
Compared performance with AdamW across various datasets and tasks.
AdamN achieves higher training accuracy milestones significantly earlier than AdamW.
On CIFAR-100, AdamN reaches 80% and 90% training accuracy ~127 s and ~165 s faster, respectively.
AdamN lowers rare-token perplexity while matching performance on head and mid-frequency tokens.

Abstract

This paper introduces AdamN, a nested-momentum adaptive optimizer that replaces the single Exponential Moving Average (EMA) numerator in Adam/AdamW with a compounded EMA of gradients plus an EMA of that EMA, paired with an exact double-EMA bias correction. This yields a smoother, curvature-aware search direction at essentially first-order cost, with longer, more faithful gradient-history memory and a stable, warmup-free start. Under comparable wall-clock time per epoch, AdamN matches AdamW’s final accuracy on ResNet-18/CIFAR-100, while reaching 80% and 90% training-accuracy milestones ~127 s and ~165 s earlier, respectively. On pre-benchmarking workloads (toy problems and CIFAR-10), AdamN shows the same pattern: faster early-phase convergence with similar or slightly better final accuracy. On language modeling with token-frequency imbalance—Wikitext-2-style data with training-only token corruption and a 10% low-resource variant—AdamN lowers rare-token perplexity versus AdamW without warmup while matching head and mid-frequency performance. In full fine-tuning of Llama 3.1–8B on a small dataset, AdamN reaches AdamW’s final perplexity in roughly half the steps (≈ 2.25 xfaster time-to-quality). Finally, on a ViT-Base/16 transferred to CIFAR-100 (batch size 256), AdamN achieves 88.8% test accuracy vs. 84.2% for AdamW and reaches40–80% validation-accuracy milestones in the first epoch (AdamW reaches 80% by epoch 59),reducing epochs, energy use, and cost.

Read Full Paperexternally

Ask AI

Helpful

Bookmark

View Full Paper