November 13, 2025Open Access

LoRA Eliminates Catastrophic Forgetting: Comprehensive Empirical Study with 234 Experiments Validating Frozen-Weight Continual Learning Architecture

Key Points

Memory use decreases by 83% while achieving 72.2% accuracy, highlighting efficiency.
Neural networks demonstrate significant improvement in learning retention using LoRA.
Quantization levels of 4-bit and 8-bit were validated across 234 systematic experiments with benchmark accuracy.
Results suggest that adopting this approach could enable more efficient neural network training.

Abstract

Catastrophic forgetting prevents continual learning in neural networks. We discover that Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method, reduces catastrophic forgetting by 95.5% (from 20.8% to 0.9%, p=0.0028). Through 234 systematic experiments across 8 random seeds on BERT-base, we validate LoRA ranks (4, 8, 16, 32), quantization levels (4-bit, 8-bit), and module selections. Combined with 4-bit quantization, our best configuration achieves 72.2% accuracy, 0.46GB memory (83% reduction), and 1.4% forgetting (93.1% reduction). Freezing ablation studies validate the frozen-backbone mechanism. Code and data available on GitHub.

Read Full Paperexternally

KI fragen

Bookmark

View Full Paper