Investigating the utility of large language models for image-based rare disease phenotyping

Purpose: Artificial intelligence has been applied across various fields of medicine, with large language models (LLMs) demonstrating potential to assist in clinical decision-making for rare diseases.This study assessed the performance of LLMs in aiding the phenotyping process and guiding the identification of correct genetic conditions.Materials and Methods: Clinical images from 10 Korean individuals with genetically confirmed rare diseases were collected through a literature review.Using identical prompts, the top 10 Human Phenotype Ontology (HPO) terms and suspected genetic conditions were queried across three LLMs: generative pre-trained transformers (GPTs) models GPT-4o and GPT o1, and Claude 3.5 Sonnet.Concordance between 5 manually curated key HPO terms and the top 10 predicted terms were assessed, and the accuracy of genetic diagnoses among the LLMs was analyzed.Results: Clinical images, ranging from 3 to 9 per case, were used as input for 10 Korean rare disease cases.The average number of key HPO terms correctly matched among the top 10 predictions was 3.0 (1.99-4.01)for GPT-4o, 2.8 (2.06-3.54)for GPT o1, and 1.7 (0.80-2.60) for Claude 3.5 (mean, [95% confidence interval]).GPT models provided more specific HPO terms than Claude in these cases.The accuracy of genetic diagnosis within the top 10 predictions was 2/10 for GPT-4o, 3/10 for GPT o1, and 0/10 for Claude 3.5, with frequent hallucination events observed. Conclusion:LLMs demonstrate potential as a supportive tool for image-based rare disease phenotyping, while the frequent hallucinations highlight the need for further investigation and caution in clinical application.