Generative AI Versus Primary Care Physicians from Plan Initiation to Long-Term Maintenance of Chronic Disease Self-Management: A Vignette-Based Validation Study

Background Generative artificial intelligence (GenAI) may provide personalised and dynamically adjustable support for chronic disease self-management. However, previous studies have largely focused on isolated questions and a limited range of evaluation dimensions. Objective To compare the quality of self-management advice generated by GenAI and primary care physicians (PCPs) across multiple stages of chronic disease management and evaluation dimensions. Methods We conducted a cross-sectional, source-blinded comparative evaluation using 15 standardised vignettes of diabetes and hypertension derived from authentic primary care follow-up records. Each vignette comprised three sequential management stages, with medical accuracy, readability, and behaviour-change potential assessed across the three simulated stages; individualisation and multidisciplinary integration at plan initiation; contextual adaptability during acute adjustment; and empathy during long-term maintenance. The GenAI system and 22 PCPs independently generated advice which was independently rated by three clinicians，one each from general practice, endocrinology, and cardiology. across seven dimensions. Linear mixed-effects models were used for the primary comparisons, and case-level paired and physician-distribution analyses were conducted as robustness analyses. Results The GenAI system achieved estimated marginal means above 4.0 in all dimensions except readability. It scored significantly higher than PCPs in six of the seven dimensions, with adjusted mean differences ranging from 1.16 to 1.56 points (P<.001). Readability was the only exception, with no significant difference between GenAI and PCPs (−0.22, 95% CI −0.54 to 0.11; P=.19). The GenAI mean exceeded the individual means of all recruited PCPs in the six dimensions but ranked at approximately the 23rd percentile for readability. Conclusions Our findings indicated the GenAI system may generate more highly rated self-management advice than PCPs across most evaluated dimensions but offered no advantage in readability. These findings provide pre-implementation evidence for GenAI as a potential source of structured and adaptive chronic disease self-management support. Patient-level pilot studies and subsequent randomised trials are needed to determine its acceptability and effects on behaviour and clinical outcomes.