February 26, 2024Open Access

Training Neural Networks from Scratch with Parallel Low-Rank Adapters

Key Points

Model pre-training can be enhanced with low-rank adaptation techniques, particularly low-rank heads.
The bi-level optimization algorithm, LoRA-the-Explorer, enables efficient parallel training across computing nodes.
Experiments with vision transformers indicate that LTE competes well against standard pre-training methods with reduced synchronization needs. The findings reveal a promising avenue for pre-training deep learning models efficiently.

Abstract

The scalability of deep learning models is fundamentally limited by computing resources, memory, and communication. Although methods like low-rank adaptation (LoRA) have reduced the cost of model finetuning, its application in model pre-training remains largely unexplored. This paper explores extending LoRA to model pre-training, identifying the inherent constraints and limitations of standard LoRA in this context. We introduce LoRA-the-Explorer (LTE), a novel bi-level optimization algorithm designed to enable parallel training of multiple low-rank heads across computing nodes, thereby reducing the need for frequent synchronization. Our approach includes extensive experimentation on vision transformers using various vision datasets, demonstrating that LTE is competitive with standard pre-training.

Read Full Paperexternally

AI से पूछें

Bookmark

View Full Paper