WCN26-4700 Robustness Gap of Large Language Models in Nephrology

Large language models (LLMs) achieve high accuracy on medical benchmarks, raising interest in their clinical application. However, whether this performance reflects genuine reasoning or pattern recognition remains unclear. To evaluate reasoning robustness, we replaced the correct answer in nephrology multiple-choice questions with “None of the other answers” (NOTA) and assessed changes in accuracy. We hypothesized that causal and pathophysiological reasoning would preserve accuracy, whereas reliance on memorized patterns would cause a marked decline.