Systematic Scoring Differences in Large Language Model–Based Clinical Benchmarking Reveal Evaluator-Dependent Divergence: Controlled Evaluation Study

Background Large language model (LLM) benchmarking in clinical medicine is increasingly reliant on LLM-based assessment frameworks; in these systems, one AI scores the outputs of another. This practice relies on the assumptions of evaluator neutrality and rubric consistency. These assumptions have not been empirically tested in clinical fields. Objective In this study, we aim to directly examine these assumptions by using the PARM (Planning-Action-Reflection-Memory) framework as a theoretical lens to characterize endovascular device selection and clinical reflective reasoning as a clinical field. Methods We developed a scoring rubric (C1-C6) with 40 clinical vignettes and 6 criteria across 5 endovascular fields (peripheral artery disease, abdominal aortic aneurysm, chronic venous insufficiency, coronary in-stent restenosis [ISR], and cerebrovascular/neuroendovascular; 8 vignettes per field). Four LLMs were evaluated as clinical decision support systems: GPT-4o (M1), Claude Sonnet 4.6 (M2), Gemini 1.5 Pro (M3), and Llama 3.3 70B (M4). Two LLMs served as independent raters with the same evaluation criteria: Claude Sonnet 4.6 (rater 1) and GPT-4o (rater 2). The rater identity was intentionally disclosed; the primary research question concerned rater scoring behavior rather than the absolute performance of the model. Human assessment was intentionally excluded because its inclusion would have shifted the study question from rater disagreement to rater accuracy, a distinct and secondary question. Interrater reliability was measured using intraclass correlation coefficient (ICC 2,1) two-way random effects modeling. Results The combined ICC(2,1) was 0.153 (95% CI 0.007‐0.295). In this exploratory study, the low ICC was not distributed randomly: the magnitude of score differences varied across models (M2 Δ=+5, M4 Δ=+31, M1 Δ=−46, and M3 Δ=−158 points). Scoring behavior analysis identified 2 intrinsically consistent but mutually inconsistent criterion-interpretation models. Rater 1’s scores were most consistent with the reasoning quality approach (partial credit to clinical competence independent of vignette context). Rater 2’s scores were most consistent with the context fidelity approach (near-zero scores for wrong vignette responses independent of reasoning quality). Contrary to the self-preference hypothesis, both raters placed the other higher-performing model above their own. Under rater 1, C6 (PARM reflection) was the lowest-scoring criterion across all 4 models, whereas rater 2 showed greater model-dependent variation in C6 scoring. Conclusions AI raters used in LLM-based clinical benchmarking do not behave as interchangeable scoring tools. Their scoring behavior is most consistent with model-specific criterion interpretation patterns that systematically produce different outcomes. The observed low interrater reliability requires interpretation rather than simply being ignored: it reflects structured, direction-specific disagreement concentrated in models with context-tracking failures, not random inconsistency. PARM reflection showed a notable yet evaluator-dependent scoring pattern that warranted further investigation, rather than being interpreted as an evaluator-independent deficit.