SchemaBank: Sparse Routing as Training Curriculum for Parameter-Efficient Fine-Tuning

We introduce SchemaBank, a three-stage training curriculum that uses sparse routing toimprove parameter-efficient fine-tuning of language models. Our approach trains specializedlow-rank adapters through progressively supervised routing during training, then removes therouting mechanism at inference to enable flexible adapter composition. We demonstrate that thistraining-time routing curriculum achieves 3.1× improvement over standard LoRA on GSM8Kmathematical reasoning with Qwen2-0.5B (11.8% vs 3.75% accuracy) while incurring no inferenceoverhead. Experiments across 40 training runs with 4 random seeds show consistentimprovements and reduced variance compared to baseline approaches. This preliminary singletask,single-model study suggests that, in our experimental setting, routing mechanisms maybe more effective as training curricula than as inference-time architectures, contrasting withconventional mixture-of-experts approaches.