High-quality code representations are fundamental to code intelligence. Achieving such representations with parameter-efficient fine-tuning (PEFT) remains a key challenge. While code pre-trained models (CodePTMs) offer a robust foundation for general-purpose embeddings, current PEFT approaches face two main obstacles when adapting them: (i) they fail to adequately capture the deep structural characteristics of programs, and (ii) they are limited by the model’s finite internal parameters, restricting their ability to overcome inherent knowledge bottlenecks. To address these challenges, we introduce a parameter-efficient code representation learning framework that combines retrieval augmentation with structure-aware priors. Our framework features three complementary, lightweight modules: first, a structure–semantic dual-channel retrieval mechanism that infuses high-quality external code knowledge as non-parametric memory to alleviate the knowledge bottleneck; second, a graph relative bias module that strengthens the attention mechanism’s capacity to model structural relationships within programs; and third, a span-discriminative contrastive objective that sharpens the distinctiveness and boundary clarity of span-level representations. Extensive experiments on three benchmarks spanning six programming languages show that our method consistently outperforms state-of-the-art parameter-efficient baselines. Notably, on structure-sensitive tasks using the PLBART backbone, RS-Rep surpasses full fine-tuning, delivering a 22.1% improvement in Exact Match for code generation and a 4.4% increase in BLEU scores for code refinement, all while utilizing only about 5% of the trainable parameters.
Zheng et al. (Wed,) studied this question.