"No Errors Found'': Review Verdicts Shift the Threshold at Which Language Models Flag Clinical Errors

Full code repository for the No Errors Found project. Layout Path Contents Top level run_experiment.py Pipeline entry point (steps below) analysis_v3.py Cross-model analysis entry point dump_generation_config.py Effective decoding settings and software versions quick_check.py Per-header summary of one model's raw answers config/ (every setting of the study) paths.py Data and results folders steps.py Default steps datasets.py MEDEC-MS and MedErrBench-EN files, 739-text sample size conditions.py Free-text header conditions and wordings structured.py FHIR, flat JSON and HL7 v2 status blocks factorial.py Verdict x reviewer conditions prompts.py System prompt, user template and prompt variants studies.py Study components: conditions x variants x texts x models models.py Model registry, decoding and serving settings inference.py GPU, server, pre-check and request settings evaluation.py Bootstrap, non-responsive rule and report settings noerrorsfound/pipeline/ common.py Output paths, logging, command-line parsing selection.py Model and study selection, GPU assignment dataset.py Benchmark loading, sentence parsing, case linking prompts.py Prompt rendering, content-hashed prompt registry (build step) client.py OpenAI-compatible requests, token log-probabilities server.py vLLM server start, health checks, shutdown hardware.py GPU memory, GPU processes, disk checks precheck.py Pre-run check of every model (readable answers, truncation, speed) inference.py Resumable inference, parallel GPU workers (infer step) parsing.py JSON answer parsing, repetition-loop detection metrics.py Sensitivity, specificity, d', c, bootstrap deltas, McNemar, Holm correction.py Secondary correction-quality scoring predictions.py Predictions, fingerprints, GEE summary, paired subsets evaluate.py Per-model effects and tests (evaluate step) diagnose.py Answer-format diagnostics (diagnose step) meta.py Random-effects pooling and meta-regression results.py Loader for all per-model results tables.py Report tables style.py Figure style figures.py Report figures report.py Report, index and run manifest (report step) steps.py Step dispatcher noerrorsfound/analysis/ common.py Loading, model metadata, non-responsive rule effects.py Paired bootstrap effects parse_sensitivity.py Unreadable-answer scoring sensitivity discrimination.py AUROC of P(error_flag = 1) localization.py Sentence localization and matched-hit analysis zroc.py zROC slopes noise_floor.py Rerun flip rates and excess flips metaregression.py REML, HKSJ pooling, meta-regression ablation.py 739-text comparisons across all models structured.py Structured-metadata effects variants.py Prompt-variant effects factorial.py Verdict x reviewer effects compile.py Model table, groups, pooled and primary tests robustness.py Error-type restriction, reasoning length, model pairs glmm.py Pooled case-clustered logistic model export.py Slim prediction files and summary main.py Analysis driver Other noerrorsfound/environment.py Generation-config and environment dump chat_templates/ Llama 3 chat template for OpenBioLLM-8B Data Clone the two benchmarks into data/: git clone https://github.com/abachaa/MEDEC data/MEDEC git clone https://github.com/congboma/MedErrBench data/MedErrBench Environment Linux with NVIDIA GPUs (two 48 GB GPUs for the 27B and 32B models). pip install -r requirements.txt Contact Nahid Hossain, MSc (nfh5328@psu.edu) Md Faisal Kabir, PhD (mpk5904@psu.edu) School of Science, Engineering and Technology The Pennsylvania State University, Harrisburg, PA, USA License The source code in this repository is released under the MIT License. Result tables and model predictions are released under CC BY 4.0. The MEDEC and MedErrBench datasets are not included in this repository and remain subject to their original licenses and terms of use. Disclaimer This project is intended for research purposes only and is not a medical device. It should not be used for clinical decision-making.