Overconfident AI? Benchmarking LLM Self-Assessment in Clinical Scenarios

Abstract Background and Aim The capabilities of large language models (LLMs) to self-assess their own confidence in answering questions in the biomedical realm remain underexplored. This study evaluates the confidence levels of 12 LLMs across five medical specialties to assess their ability to accurately judge their responses. Methods We used 1,965 multiple-choice questions assessing clinical knowledge from internal medicine, obstetrics and gynecology, psychiatry, pediatrics, and general surgery areas. Models were prompted to provide answers and to also provide their confidence for the correct answer (0–100). The confidence rates and the correlation between accuracy and confidence were analyzed. Results There was an inverse correlation (r=-0.40, p=0.001) between confidence and accuracy, where worse performing models showed paradoxically higher confidence. For instance, a top performing model, GPT4o had a mean accuracy of 74% with a mean confidence of 63%, compared to a least performant model, Qwen-2-7B, which showed mean accuracy 46% but mean confidence 76%. The mean difference in confidence between correct and incorrect responses was low for all models, ranging from 0.6% to 5.4%, with GPT4o having the highest differentiation of 5.4%. Conclusion Better performing LLMs show more aligned overall confidence levels. However, even the most accurate models still show minimal variation in confidence between right and wrong answers. This underscores an important limitation in current LLMs’ self-assessment mechanisms, highlighting the need for further research before integration into clinical settings.