What does this research mean for the field?

Restricting backpropagation to a specific sliding window of early-middle layers while freezing the rest at random initialization achieves lower perplexity and significantly reduces memory usage compared to full backpropagation during LLM pretraining. Novelty: ClaimNovelty.NOVEL_FINDING. Consensus alignment: ConsensusAlignment.NEUTRAL.

What question did this study set out to answer?

To identify which layers of a transformer model require backpropagation during pretraining to enhance efficiency.

May 28, 2026Open Access

View Full Paper

Which Layers Need Backpropagation? Position-Dependent Partial Training for Memory-Efficient LLM Pretraining

KKim

Key Points

To identify which layers of a transformer model require backpropagation during pretraining to enhance efficiency.
Trained Llama-1B using backpropagation limited to a sliding window of layers.
Frozen layers were initialized randomly to test performance with varying training horizons.
Conducted fine and position sweeps to analyze the impact of backpropagation position on training effectiveness.
Frozen-partial training achieved a perplexity of 35.2, outperforming full backpropagation at 39.0 by 11%.
Utilized 37% less VRAM (10.9 vs 17.4 GB) at matched micro-batch sizes.
Optimal backpropagation position varied by architecture, with attention models preferring early-middle layers.

Abstract

We ask which layers of a transformer actually need backpropagation during pretraining from scratch. Training Llama-1B with backpropagation restricted to a sliding window of layers (plus embeddings and the LM head) and freezing the remainder at random initialization, we find that the best training method depends critically on the training horizon. budget and schedule, frozen-partial reaches the lowest perplexity (35.2), beating full backpropagation (39.0) by 11% and GaLore (52.4) by 33%, while using 37% less VRAM at a matched micro-batch (10.9 vs 17.4 GB). A fine-grained sweep locates the crossover between 25M and 50M tokens on a single dataset, ruling out a dataset confound. A position sweep reveals a U-shaped curve whose optimum lies in the early-middle layers, coinciding with the network's lowest gradient signal-to-noise region. The non-BP layers can be frozen entirely: zeroth-order perturbation contributes at most ~4 PPL beyond freezing. Frozen-partial is also the most robust to learning-rate choice. The optimal BP position is architecture-dependent: attention models prefer an early-middle window, while recurrent/SSM models require the input side. All results are at the 1B scale; cross-scale (3B/7B) study is left to future work. Companion code and experiment logs: https://github.com/2264K/hybrid-zo-pretrain License: paper CC BY 4.0; code Apache-2.0.

AI에게 질문

Bookmark

View Full Paper

AI에게 질문

Bookmark

View Full Paper

Which Layers Need Backpropagation? Position-Dependent Partial Training for Memory-Efficient LLM Pretraining

Key Points

Abstract

Cite This Study