Accuracy, reproducibility, and concordance of ChatGPT and Gemini on text-based questions from the Turkish Orthopaedics and Traumatology Board Examination: a multi-year comparative study

Background Data on the reproducibility, shared error patterns, and multi-year performance of large language models (LLMs) on national orthopaedic board examinations are limited. This study compared the accuracy, response reproducibility, explanation quality, and inter-model concordance of ChatGPT and Gemini on text-based questions from the Turkish Orthopaedics and Traumatology Board Examination (TOTBE), with exploratory benchmarking against published examination statistics. Methods Of 500 TOTBE questions administered between 2021 and 2025, 52 image-based and one annulled question were excluded, leaving 447 text-based questions. Each question, presented in its original Turkish language, was submitted twice to ChatGPT 5.2 (OpenAI) and Gemini 3 Flash (Google) via their public interfaces on January 20 and January 27, 2026, yielding 1,788 responses. Primary analyses used the first response at the question level; second responses served for reproducibility assessment and an additional sensitivity analysis, and a generalized estimating equation (GEE) model accounted for within-question correlation. Accuracy was compared using the exact McNemar test. Two orthopaedic surgeons independently rated explanation quality on a 5-point Likert scale (inter-rater reliability, ICC(2,1)). Results ChatGPT answered 377 of 447 questions correctly (84.34%; 95% CI, 80.68–87.41) and Gemini 396 of 447 (88.59%; 95% CI, 85.31–91.21); Gemini was significantly more accurate ( p = 0.011). Reproducibility was excellent for both models (ChatGPT 94.41%, κ = 0.930; Gemini 97.32%, κ = 0.966). Of 35 questions missed by both models, 25 (71.4%) involved the same incorrect option. The GEE analysis confirmed higher odds of a correct response for Gemini (OR, 1.44; 95% CI, 1.10–1.89; p = 0.008). Explanation quality did not differ significantly ( p = 0.065). Both models exceeded the published passing thresholds in every year. Conclusions Contemporary LLMs achieved high accuracy and excellent reproducibility on text-based TOTBE questions, with Gemini demonstrating significantly higher first-attempt accuracy than ChatGPT. To our knowledge, this is the first study to evaluate accuracy, reproducibility, and inter-model concordance together across multiple years of a national orthopaedic board examination. Because only the text-based component was evaluated under non-equivalent testing conditions, the findings should not be interpreted as evidence of overall orthopaedic competence or as direct comparisons with human examinees. Clinical trial number Not applicable. Level of evidence Level IV Diagnostic/Methodological Study (retrospective).