Answers are grounded in this paper and cite their sources.
Results of the pre-registered study CRLN-PRE-002 (registration: 10.5281/zenodo.22968475). The result is null. Question. On the CRLN clinical research instrument, 9,393 stored model responses from five models had never used confidence 1 or 2 on a 1 to 5 scale. The prompt labeled only the ends of the scale and gave one worked example, a 4. The registration asked whether that example was the reason. Design. Four models from two vendors (Anthropic and Google) answered 200 keyed items, drawn with a registered seed, in 3 option arrangements each, under two prompts that differ only in the confidence sentence: the original wording (Arm A) and a variant that gives every scale point a meaning and shows no example value (Arm B). 4,800 rows were retained, 3 of them vendor errors. Result. Confidence 1 or 2 was used in 1 of 2,397 responses under Arm A (0.04%) and 4 of 2,400 under Arm B (0.17%). Two-proportion z test, two-sided: z = 1.34, p = 0.18. The registered hypothesis is not supported, and Arm B stayed far below the registered 5% floor, so by the registration's own rule the low end of the scale is not reachable on this instrument whatever the wording says. Accuracy did not change between arms (73.8% in both), so the comparison is not confounded. What the labeled wording did change, in an analysis that was not registered, was the top of the scale: the share at 5 fell from 54.0% to 42.6%. Addendum, reported separately. gpt-4o could not be reached when the main runs were collected and was run later the same day under the identical command, after the main result was written. It is reported in a labeled addendum and is not pooled into the primary test. It gives the same null: 1 of 600 and 3 of 600 at confidence 1 or 2 (z = 1.00, p = 0.32). Also reported. Every departure from the registration, a void run and an unrelated earlier run with their ids, and an exploratory, not pre-registered comparison with 996 human decisions from 49 learners on the CRLN platform, where 4.7% of decisions were at confidence 1 or 2 and every reported cell has at least five people. Limits. One instrument, one domain, one language, a single digit format, and no temperature control. The instrument is not SME-validated, and accuracy is measured against an authored key. Files: the results report (PDF and Markdown), the analysis script that reproduces every figure from the retained rows, the aggregate results table, and the license. No item text or answer key is included.
No takes yet. Share an insight, caveat, or question.
Joshua Webber (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: