What type of study is this?

This is a Literature Review study (also classified as: Quantitative Study).

October 5, 2025Open Access

Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey

Key Points

Reward models significantly enhance reasoning performance in large language models, enabling better output generation.
Key applications include selecting optimal outputs, facilitating data synthesis, and providing training signals in reinforcement learning.
A systematic review of the architectures and training methodologies of reward models highlights their critical role in LLM inference.
Addressing open questions about reward model selection and evaluation aids in the effective deployment and advancement of reasoning capabilities.

Abstract

Reward models (RMs) play a critical role in enhancing the reasoning performance of LLMs. For example, they can provide training signals to finetune LLMs during reinforcement learning (RL) and help select the best answer from multiple candidates during inference. In this paper, we provide a systematic introduction to RMs, along with a comprehensive survey of their applications in LLM reasoning. We first review fundamental concepts of RMs, including their architectures, training methodologies, and evaluation techniques. Then, we explore their key applications: (1) guiding generation and selecting optimal outputs during LLM inference, (2) facilitating data synthesis and iterative self-improvement for LLMs, and (3) providing training signals in RL-based finetuning. Finally, we address critical open questions regarding the selection, generalization, evaluation, and enhancement of RMs, based on existing research and our own empirical findings. Our analysis aims to provide actionable insights for the effective deployment and advancement of RMs for LLM reasoning.

Read Full Paperexternally

Mark Helpful

Bookmark

Relay

View Full Paper