Large Language Models Aligned With Medical Experts on Established Topics but Were Less Aligned on Emerging or Debated Topics in Posterolateral Corner Treatment: An Artificial Intelligence‐Augmented Delphi Study

Purpose To evaluate how Large Language Models (LLMs) align with a concurrently performed human medical expert‐modified Delphi study on the posterolateral corner and to qualitatively analyze LLM behavior. Methods Four commercially available and current LLMs (Gemini‐2.5‐Pro. GPT‐4o, o4‐mini, and o3) were tasked with rating posterolateral corner consensus statements on a 5‐point Likert scale and estimating the level of expert consensus for each statement, mirroring a human expert Delphi process. LLM performance was evaluated based on alignment with expert majority opinion, accuracy in predicting expert consensus, and self‐consistency across multiple LLM invocations. Deep research to improve LLM performance was also explored. Results LLM‐generated ratings showed moderate correlation with average expert ratings, with some models exhibiting tendencies toward overly positive ratings. Consensus derived from LLM self‐consistency did not reliably approximate expert consensus. LLM estimates of consensus were informative, with some models showing improved alignment with actual expert consensus. Qualitative analysis revealed distinct behavioral patterns across different LLMs regarding justification generation, certainty expression, and handling of conflicting evidence. Preliminary findings suggest that deep research capabilities can enhance rating accuracy as well as consensus prediction. Evidence of hallucination in LLM reasoning is observed and requires further verification. Conclusions Although LLMs show potential in reflecting expert medical knowledge and estimating consensus, their direct ratings and self‐consistency may not perfectly align with expert panels. Different LLMs display unique strengths and weaknesses, suggesting that model selection or a combination can be beneficial for applications like modified Delphi studies. Improved consensus estimation was seen by providing the models access to web search, enabling them to find recent, relevant research articles. Level of Evidence Level V, expert opinion.