What question did this study set out to answer?

This research aims to enhance the adaptability of large language models by addressing limitations in existing fine-tuning methods that utilize mixture-of-experts architectures.

August 6, 2026Complex & Intelligent Systems0 citationsOpen Access

MoFL: fine-tuning LLMs with mixture of fused LoRA experts

SLSong Lin DGDehong Gao YMYufei Ma

Key Points

This research aims to enhance the adaptability of large language models by addressing limitations in existing fine-tuning methods that utilize mixture-of-experts architectures.
Introduced MoFL, a hierarchical mixture-of-experts architecture.
Implemented a fully differentiable ensemble routing strategy for expert collaboration.
Conducted extensive experiments on the HuggingFace Open LLM benchmark.
Achieved an average score of 52.45 while updating less than 2% of model parameters.
Demonstrated superior performance over existing LoRA-based PEFT methods across different backbone LLMs.
Improved mean score from 33.53 to 38.68 on the GSM8K mathematical reasoning benchmark.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) methods have enabled practical adaptation of Large Language Models (LLMs), yet integrating Mixture-of-Experts (MoE) architectures for multi-task scenarios introduces two critical challenges. First, existing MoE-based PEFT approaches overlook the need to balance commonalities and distinctions among diverse tasks, thereby limiting cross-task transferability. Second, prevailing routing strategies rely on discrete expert selection and gradient estimation techniques, which intermittently undermine the ability to consistently identify optimal routing pathways. To address these issues, we propose MoFL (Mixture of Fused LoRA), a hierarchical MoE architecture that incorporates Convergent Experts and Divergent Experts with fused low-rank parameters to jointly capture shared and task-specific knowledge. Complementing this design, a fully differentiable ensemble routing strategy is introduced, eliminating reliance on gradient estimation while enabling collaborative contributions from all experts during generation. Extensive experiments on the HuggingFace Open LLM benchmark demonstrate that MoFL achieves an average score of 52.45 while updating less than 2% of model parameters and consistently outperforms existing LoRA-based PEFT methods across diverse backbone LLMs. Notably, on the GSM8K mathematical reasoning benchmark, the mean score rises from 33.53 to 38.68, the most substantial improvement among all evaluated tasks.

Ask AI

Helpful

Bookmark

View Full Paper