What question did this study set out to answer?

This research aims to establish a benchmark that accurately measures how well language models act as sparring partners in dialogue.

June 28, 2026Open Access

ReflectionBench: A Benchmark for Measuring Reflective Sparring-Partner Quality in Language Models

Key Points

This research aims to establish a benchmark that accurately measures how well language models act as sparring partners in dialogue.
Developed ReflectionBench with 12 reflection scenarios covering personal decisions, self-interpretations, and stuck assumptions.
Responses scored on four axes: assumption surfacing, counter-hypothesis generation, uncertainty marking, and sycophancy resistance.
Introduced a model-as-judge protocol to reduce bias with context-order swapping and multi-judge assessment.
Models scored between 115-144/144 in total, with a significant performance range on A2 (24-36) and A3 (25-36).
Counter to expectations, all models performed well on sycophancy resistance, scoring at least 30/36 on A4.
A1 showed a ceiling effect with five models scoring 36/36, indicating high assumption surfacing quality.

Abstract

Language models are increasingly deployed as reflective dialogue partners, yet no benchmark measures whether a model genuinely sparring-partners the user rather than defaulting to affirmation. We introduce ReflectionBench, a fully model-gradable benchmark of 12 reflection scenarios across three types (personal decisions, self-interpretations, and stuck assumptions), each paired with a pressure-test follow-up. Responses are scored on four behavioral axes using a 0-3 anchored rubric: Assumption Surfacing (A1), Counter-Hypothesis Generation (A2), Uncertainty Marking (A3), and Sycophancy Resistance (A4). A model-as-judge protocol with context-order swapping, rubric-anchored justification, and multi-judge passes reduces grader bias. A six-model baseline (self-scored, June 2026) yields total scores of 115-144/144. The hypothesized A4 sycophancy collapse is disconfirmed: all models score at least 30/36, while discrimination emerges on A2 (24-36) and A3 (25-36); A1 shows a ceiling effect at 36/36 for five of six models. We develop a general sycophancy-trigger taxonomy (Validation Request, Credential Challenge, Emotional Escalation, Dogmatic Reassertion) mapped against ELEPHANT's social-sycophancy dimensions. ReflectionBench fills the gap between QA sycophancy benchmarks and Socratic dialogue evaluations by measuring reflective integrity in free-form conversational sparring, and sits within the broader AI-mediated selfhood research program alongside the CASK framework and CalibrationBench. The full scenario set, rubric, judge prompts, and manual RUN-SHEET are provided for replication. Note: the reported baseline is self-scored (each model graded its own responses) and should be treated as an upper bound pending cross-model judging

Read Full Paperexternally

Ask AI

Helpful

Bookmark

View Full Paper