Unified Large Language Models (LLMs) have transformed diverse recommendation tasks into a shared “text-to-text” paradigm. However, existing methods primarily face two challenges: 1) joint fine-tuning across highly heterogeneous tasks (e.g., discriminative sequential recommendation versus generative explanation generation) often suffers from the “seesaw effect” due to severe gradient conflicts; and 2) parameter-efficient methods struggle to isolate these task-specific conflicts while preserving the collaborative sharing of underlying general recommendation knowledge. To address these limitations, we propose Rec-MoELoRA, a hybrid fine-tuning framework that seamlessly integrates the mixture-of-experts architecture with low-rank adaptation. Specifically, we devise a soft-decoupling strategy that maintains the updates of most backbone network layers to continuously absorb globally shared knowledge, while exclusively introducing task-motivated low-rank experts into the self-attention layers for physical task isolation. Furthermore, to effectively overcome the prevalent “expert collapse” dilemma, we incorporate a dual regularization strategy of diversity and entropy. Combined with an asymmetric structural prior (e.g., allocating three experts for two tasks), this forces the model to spontaneously evolve a “Shared-Specific” representation pattern. Extensive experiments on real-world e-commerce datasets (Amazon Sports and Beauty) demonstrate that Rec-MoELoRA significantly outperforms the full fine-tuning baseline across core ranking and text generation metrics while introducing only a marginal number of additional parameters (e.g., HR@10 and BLEU-4 improve by 4.3% and 3.7% on the Sports dataset, respectively).
Sun et al. (Mon,) studied this question.