Toward trustworthy diagnostic AI: debiasing and grounding large language models for seizure semiology interpretation

Large language models (LLMs) have been proposed to map free-text seizure semiology to seizure-onset-zone (SOZ) likelihoods, but it remains unclear whether inference-time interventions can improve on the original SemioLLM benchmark without additional training. We evaluated two such interventions, candidate-order permutation within self-consistency and TF-IDF retrieval of case-specific labelled examples, individually and in combination (retrieval-augmented order-invariant self-consistency, RAOI-SC), across seven locally served open-weight LLMs on 1,268 Semio2Brain rows. Performance was assessed with weighted- and macro-F1, paired bootstrap confidence intervals, and exact McNemar tests against the self-consistency baseline, and a multi-dimensional reliability profile covering citation coverage and conditional verifiability, retrieval agreement, strict schema adherence, and confidence concentration; a TF-IDF nearest-neighbor classifier and grouped-split sensitivity analyses bounded how much of any gain could reflect retrieval leakage rather than the intervention itself. RAOI-SC had a higher weighted-F1 point estimate than self-consistency for six of seven models, but the improvement was supported by paired confidence intervals for four models and by McNemar testing for only one, Mistral-7B-Instruct-v0.3; the TF-IDF nearest-neighbor baseline alone achieved 55.72 weighted-F1 under row-level splitting, falling to 50.43 and 48.97 once identical-text and publication-reference duplicates were grouped together. These results support a cautious, model-specific reading of retrieval and order permutation as testable prompting interventions rather than a universally superior or clinically validated method, and define a row-level benchmark that does not by itself establish patient- or source-level generalization.