CARM: Cross-Modal Alignment Recovery for Lightweight Referring Expression Comprehension

Referring Expression Comprehension (REC) localizes a target object in an image given a natural-language referring expression and is a core benchmark for fine-grained vision–language alignment. Recent detection-style multimodal Transformers achieve strong REC performance but typically rely on high-capacity visual and textual backbones, incurring substantial storage and compute costs. Replacing these backbones with lightweight alternatives greatly reduces model size, yet often degrades cross-modal alignment and yields a persistent accuracy gap. We propose CARM, a minimally invasive Cross-modal Alignment Recovery Module inserted between lightweight backbones and the downstream multimodal Transformer, requiring no changes to either component. CARM injects complementary priors via bidirectional prompts and uses a Cross-Attention Gate (CAG) to adaptively filter and scale prompt-induced updates; it further integrates Tree-of-Attributes Prompts (TAPs) to strengthen fine-grained grounding of attributes such as color, size, and spatial location. On RefCOCO, switching to lightweight backbones drops P@1 (IoU ≥ 0.5) to 84.51, while CARM improves it to 86.23, recovering most of the loss. Meanwhile, the overall model storage (checkpoint) is reduced by about 80%, demonstrating that the cross-modal alignment degradation induced by compression can be effectively restored.