Answers are grounded in this paper and cite their sources.
Systematic zero-shot analysis shows chain-of-thought degrades reranking in various query complexities.
Chain-of-thought (CoT) prompting has demonstrated strong gains on multi-step reasoning tasks, yet its utility in document reranking remains contested. Supervised approaches report improvements from CoT-augmented rerankers, while concurrent negative results challenge whether reasoning is necessary at all. Yet the zero-shot regime, where no labeled data or fine-tuning is available, has not been systematically studied. We present the first zero-shot analysis of CoT in pointwise document reranking. Using Qwen2.5-3B-Instruct on five benchmarks (BEIR and BRIGHT), we show that direct logit scoring (DIRECT-POINT) consistently outperforms reasonthen-score (REASON-POINT) across all datasets (∆NDCG@10 from 0.001 to 0.124; mean ∆ = 0.048). Stratifying queries into complexity terciles reveals that CoT degrades performance uniformly across simple, medium, and complex queries (Mann– Whitney U, all p < 0.05). A complexity-aware router that directs complex queries to CoT likewise fails to recover baseline performance. The source code is available athttps://github.com/raheelism/cot_reranking.git
No takes yet. Share an insight, caveat, or question.
Anwar et al. (2025) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: