Characterizing Patients Who May Benefit from Mature Medical AI Models

The rapid development of medical artificial intelligence (AI) in clinical settings has raised critical debates about their real-world generalizability, yet evidence remains sporadic. Here, we systematically investigate which patients benefit from mature medical AI models during real-world deployments. Using an end-to-end NLP model, we synthesized data from 296,499 scientific publications spanning from January 1998 to January 2024, focusing on 4,361 studies that reported characteristics of mature medical AI models. These models were primarily used in a few specialties (e.g., oncology). Analyzing patient demographics and model performance from 1,590,612 patients, we uncovered a striking imbalance: 97.7% of patients benefiting from AI diagnostics were either Asian or White, predominantly from China (40.8%) and the US (12.4%), showing statistically significant geospatial, economic and racial inequalities (Dagum-Gini coefficient = 0.81, 0.87, 0.97, p < .05), particularly to Black populations in less economic developing countries (LEDC). Notably, almost all these patients (98.6%) were diagnosed using AI models developed with demographic samples similar to their own, showing superior performance compared to human diagnoses (85.9% vs. 77.9% accuracy, p < .001). However, when these AI models were applied outside of the training demographic groups, their effectiveness did not significantly surpass that of human practitioners. These results suggest a failure of generalization in real-world clinical applications, and highlight a troubling exclusivity/inequality in the benefits of mature medical AI models, which are largely limited to underrepresented populations, especially in Black patients from third-nation countries.