Learning-based modality selection for efficient multimodal retrieval

Existing multimodal retrieval-augmented systems (RAG) typically retrieve both text and images for every query, including queries that can be answered using only textual information. This wastes computational resources and increases latency. We frame modality selection as a supervised classification problem in which a lightweight classifier determines whether visual retrieval is necessary, and we evaluate two variants based on when this decision is made. In the post-retrieval case, the classifier relies on image-relevance features and therefore requires image retrieval for every query; it can only save the time spent processing the retrieved images. The pre-retrieval-only classifier uses four features available after text retrieval and before visual retrieval, allowing the visual branch to be skipped for text-only queries. On a collection of 265 labeled queries over a corpus of 5,000 images, the pre-retrieval-only variant achieves 94.34% accuracy (95% Wilson interval [84.6%, 98.1%]; n = 53), with 1 unnecessary visual retrieval and 2 missed visual retrievals, while reducing measured pipeline latency from 118.97 to 65.61 ms per query, a 44.85% reduction. We compare the proposed variants with ten baselines. A query-only classifier using no retrieval information achieves 98.11%, indicating that a substantial portion of the label signal in this dataset can be recovered from the query surface form. The contribution therefore lies in the pre-retrieval routing architecture and its measured efficiency rather than classifier performance. Labels were produced by two annotators external to the author group, with Cohen’s kappa of 0.947 and complete agreement on the held-out test split. The main limitations are the small evaluation set, the fact that the queries were written by the authors, and the caption-only text corpus.