Criterion, Not Discrimination: A Preregistered Signal-Detection Pilot of Approval and Contrarian Framing in Language Models

Agreement rate is the usual measure of sycophancy in language models, but it cannot distinguish a model that fails to recognize a false claim from one that recognizes it and does not say so. This preprint separates the two with signal detection theory. Three models (Grok 4.20 non-reasoning, Qwen2.5-7B-Instruct, Phi-3.5-mini-instruct) received 120 minimal pairs of true and false premises under four system-prompt contexts: approval-contingent, accuracy-contingent, neutral, and contrarian. All 2,880 replies were scored blind for whether they challenged the premise; the automated judge agreed with the author's blind coding at Cohen's κ = 0.90. A probit model with a pair random effect gave a criterion and a discrimination index (d′) for each model and context. The two confirmatory hypotheses of a pre-analysis registration gave opposite answers. An instruction to challenge the user shifted the criterion in all three models while d′ changed little (supported, 3/3). Approval framing, predicted to make models challenge less, was indistinguishable from the default prompt (not supported, 0/3). Exploratory analyses found that only accuracy framing increased challenging of false premises without a material rise in false alarms. Under the default prompt, models challenged 45–61% of false premises and 1.7% of true ones, and ranking contexts by challenge rate and by d′ gave different orders for two of three models. All responses, judgments, the coded validation sample, and analysis outputs are included. Protocol, items, prompts, and code: doi:10.5281/zenodo.23668604. Part of the BEP Principle Series. Tests predictions P4 and P5 of Conditional Belonging and Secure Dissent (doi:10.5281/zenodo.23652523).