Can Small Language Models Use What They Retrieve? An Empirical Study of Retrieval Utilization Across Model Scale

Retrieval-augmented generation (RAG) is widely used to improve factual accuracy in language models, but it remains unclear whether smaller models can effectively utilize retrieved context. We present a controlled empirical study across models from 360M to 8B parameters and multiple retrieval settings including BM25, dense retrieval, and oracle retrieval. Using a parametric knowledge split that separates questions models already know from those requiring external knowledge, we isolate failures of retrieval quality from failures of context utilization. We find that even with oracle retrieval, models ≤7B extract the correct answer only 10–15% of the time on questions they cannot answer independently. Additionally, adding retrieval context destroys 42–64% of answers the models previously answered correctly, indicating a strong distraction effect driven by the presence of context itself. These results suggest that the main bottleneck in small-model RAG systems is not retrieval quality but the model’s ability to utilize retrieved information. Code and experiments:https://github.com/sanchitpandey/rag-utilization-study