Large language model–assisted scoring of physician-administered parent interviews for autism screening in young children: a retrospective diagnostic case-control and agreement study

Large language models (LLMs) may assist clinicians in scoring narrative parent-report interviews, but diagnostic discrimination, score agreement, output repeatability, and clinical utility are distinct claims. We evaluated an LLM-assisted workflow for scoring a physician-administered, level-two autism screening interview conducted with parents of young children. Three retrospective analytical datasets were evaluated. In all cohorts, the child was the target of assessment, and a physician administered the C-BeDevel-I to the child’s parent or primary caregiver. Cohort A included 518 children aged 9–42 months and was used to evaluate the internal consistency and diagnostic validity of C-BeDevel-I as a second-level screening parent interview for autism spectrum disorder (ASD). Cohort B was a selected ASD-versus-TD case-control sample of 256 distinct children (130 ASD, 126 typical development [TD]) whose interview recordings were scored twice by Gemini-2.5, GPT-5-mini, and Qwen-Plus. The first complete output was primary and the second assessed repeated-call stability. Primary outcomes were receiver operating characteristic area under the curve (ROC AUC) against consensus DSM-5 diagnosis and concordance with expert total scores. Cohort C was an independent external center ASD–TD case-control evaluation of 60 children (30 ASD and 30 TD) enrolled at the Department of Child Health, Lianyungang First People’s Hospital. The clinical ASD/TD assessment established before AI scoring served as the diagnostic reference. In cohort B, first-output ROC AUCs were 0.9961 (95% stratified-bootstrap CI 0.9916–0.9991) for Gemini-2.5, 0.9861 (0.9738–0.9950) for GPT-5-mini, and 0.9770 (0.9610–0.9896) for Qwen-Plus. After Holm correction of paired DeLong tests, only Gemini-2.5 versus Qwen-Plus remained statistically significant (AUC difference 0.0191; unadjusted paired-bootstrap 95% CI 0.0076–0.0335; adjusted p = 0.012); adjusted p = 0.055 for each other pair. Gemini-2.5 showed a concordance correlation coefficient of 0.9631 (95% bootstrap CI 0.9506–0.9731), mean absolute error of 0.824 points (0.715–0.941), and mean AI-minus-expert difference of +0.184 points (95% CI 0.032–0.335; limits of agreement −2.235 to 2.602). Its repeated-output ICC(A,1) was 0.9875 (0.9839–0.9905). Categorical sensitivity was 0.9692 (exact 95% CI 0.9231–0.9916) and specificity was 0.9206 (0.8589–0.9613). In cohort C, Gemini-2.5 correctly classified 53 of 60 children, with sensitivity of 0.9333 (exact 95% CI 0.7793–0.9918), specificity of 0.8333 (0.6528–0.9436), and accuracy of 0.8833 (0.7743–0.9518). Its score concordance correlation coefficient was 0.9387 and mean absolute error was 0.875 points. LLM-assisted C-BeDevel-I scoring demonstrated high selected ASD-versus-TD case-control discrimination, strong agreement with expert scores, and output stability, with supportive findings from an independent external-center ASD-versus-TD case-control evaluation. These results highlight the feasibility of LLM-assisted workflows for further clinician-supervised evaluation in early ASD screening. To establish real-world clinical effectiveness and generalizability, future prospective studies utilizing larger, consecutive, and clinically heterogeneous cohorts—including neurodevelopmental comparators—are warranted. This study evaluated an artificial intelligence system that assists clinicians in scoring interviews conducted by physicians with parents about their young children’s behavior for level-two autism screening. The child is the subject of assessment, but the parent provides the interview information. Results were promising in a Chinese ASD-versus-typical-development sample, but the system is not an autonomous diagnostic tool. Clinicians must review the evidence and retain final responsibility; broader prospective testing, including children with developmental delay, is needed.