A decision-only model as a low-cost judge of medical AI answers

Background. Testing and deploying medical AI systems at scale already results in more outputs than clinicians can review, and the use of large language models (LLMs) as judges is growing rapidly. Decision-only models, cheap and fast models that return structured probabilities rather than text, offer a potentially scalable option for that goal. Methods. In an OSF pre-registered evaluation, we tested TypeSafe Jev 1.13 as a judge on tasks from five validated clinical benchmarks: rubric grading (HealthBench), planted clinical errors (When Rubrics Fail), risk in AI-written clinical documents (MedVAL), counseling ratings and failures (CounselBench) and physicians' preferences between answers (Real-POCQi). Comparators were expert raters' agreement with each other, OpenAI Decisions, GPT-6 Luna, a proposed hybrid team combining decision models and a cheap LLM and, as frontier comparators, GPT-6.1 Sol and Claude Opus 5.5. The main outcome was plain agreement with the expert label. Results. The test comprised 25,091 units and 213,913 model calls. Jev's agreement with expert labels averaged 64.7% over six tasks. On 3,614 matched units it averaged 66.1%, against 65.0% for Opus and 60.7% for Sol, and it was non-inferior to both on four of six tasks. Jev judged the whole pool in 17 minutes for US$1.46; per judgment, it cost 1/34 of Sol's price and 1/152 of Opus's. A preset cutoff held its false-alarm rate on fresh planted-error answers, and in simulation, reviewing Jev's least-sure answers first found 1.4 times the errors of a random audit. Still, omissions were hard for both: at the same false-alarm rate, Jev missed 55.6% of planted omissions and Opus 48.1%, and Opus caught 8.8 points more planted errors overall. Conclusions. Across more than 200,000 model calls, a decision-only model was non-inferior to two frontier judges on most tasks at a small fraction of their cost and time, while reporting how sure it was of every call. With local cutoffs and expert oversight of omissions and clinical documents, judging every output of a deployed medical AI system becomes more affordable.