Evaluating the appropriateness and safety of generative AI in delivering lifestyle guidance for atrial fibrillation patients

Q: What does this research mean for the field?

GPT-4o provides lifestyle guidance for atrial fibrillation patients with comparable scientific consensus to physicians and has a lower error rate with greater helpfulness. Novelty: ClaimNovelty.NOVEL_FINDING. Consensus alignment: ConsensusAlignment.SUPPORTS_CONSENSUS.

Masahiro Makino; Wan Jou She; Panote Siriaraya; Satoaki Matoba; Keitaro Senoo

doi:10.1038/s41598-025-34079-z

December 29, 2025Scientific ReportsOpen Access

Evaluating the appropriateness and safety of generative AI in delivering lifestyle guidance for atrial fibrillation patients

View Full Paper

Ask AI

Bookmark

Share

Key result

GPT-4o matches physicians on scientific consensus for AF lifestyle guidance, yielding fewer errors and greater helpfulness.

Figures

Why the study?

Limited consultation time often restricts physician-provided lifestyle counseling for AF patients, leaving them to manage changes independently.

Does generative AI (GPT-4o, DB GPT, PubMed GPT) provide appropriate, safe, and empathetic lifestyle guidance compared to an electrophysiologist for patients with atrial fibrillation?

Key Points

To assess the clinical utility, appropriateness, and safety of three large language models in delivering personalized lifestyle guidance to patients with atrial fibrillation.
Sixty-six questions from 16 atrial fibrillation patients across exercise, diet, lifestyle, and other domains were evaluated.
Responses were generated by three models: GPT-4o, a retrieval-augmented model with a curated Q&A database (DB GPT), and a modular RAG model retrieving from PubMed (PubMed GPT).
Five experienced electrophysiologists independently evaluated AI- and physician-generated responses across 10 clinical dimensions using a Generalized Linear Mixed Model with Dunnett's post-hoc tests.
GPT-4o achieved 88.0% scientific consensus (OR 1.95, 95% CI 0.73–5.21 vs. 80.0% for electrophysiologists), whereas DB GPT (67.0%, OR 0.46, 95% CI 0.20–1.05) and PubMed GPT (62.0%, OR 0.35, 95% CI 0.16–0.81) showed lower odds of consensus.
GPT-4o had significantly lower error rates for incorrect comprehension (OR 3.41, 95% CI 1.40–8.30) and incorrect retrieval (OR 3.46, 95% CI 1.28–9.31) compared to electrophysiologists.
GPT-4o yielded significantly higher specialized content (OR 10.80, 95% CI 4.28–17.29), superior bedside manner empathy (OR 5.55, 95% CI 2.60–11.82), and greater helpfulness (OR 6.74, 95% CI 3.05–14.88) than electrophysiologists.

Structured PICO

Does generative AI (GPT-4o, DB GPT, PubMed GPT) provide appropriate, safe, and empathetic lifestyle guidance compared to an electrophysiologist for patients with atrial fibrillation?

P

Population

16 patients with atrial fibrillation (mean age 69.4, 81.3% male, 62.5% paroxysmal AF) who generated 66 lifestyle-related questions.

I

Intervention

Three Large Language Models (LLMs): GPT-4o, DB GPT (retrieval-augmented model using curated Q&A), and PubMed GPT (modular RAG model retrieving from PubMed).

C

Comparator

Responses provided by an experienced electrophysiologist.

O

Outcome

Evaluation of responses across 10 dimensions (Scientific Consensus, Extent of Possible Harm, Evidence of Incorrect Comprehension, Evidence of Incorrect Retrieval, Evidence of Incorrect Reasoning, Inappropriate Content, Incorrect Content, Specialized Content, Bedside Manner, and Helpfulness of the Answer) by 5 independent blinded electrophysiologists.

Generative AI, particularly GPT-4o, can provide highly empathetic, helpful, and scientifically accurate lifestyle guidance for atrial fibrillation patients, performing comparably or superiorly to electrophysiologists in several domains.

Limitations

PubMed GPT translates Japanese questions into English and extracts keywords, which may alter the intent of the original question
ChatGPT responses were obtained in Japanese, where LLM accuracy is reported to be lower than in English
Results may vary with other chatbots or future versions
66 questions may not cover all potential queries
Responses were obtained only once per question
Evaluators were electrophysiologists with limited specific expertise in exercise and nutrition

Implication

May augment time-constrained AF lifestyle counseling; leaves open prospective trials before clinical adoption.

Abstract

Lifestyle factors play a major role in atrial fibrillation (AF) incidence, but the effectiveness of lifestyle counseling varies among individuals. Due to limited consultation time, physicians often provide only brief guidance, leaving patients to manage changes on their own. This study assessed the clinical utility of three Large Language Models (LLMs) for delivering accurate and personalized lifestyle guidance: (1) GPT-4o, (2) a retrieval-augmented model using a curated Q&A database (DB GPT), and (3) a modular RAG model retrieving evidence from PubMed (PubMed GPT). Sixty-six questions from 16 AF patients were categorized into exercise, diet, lifestyle, and other domains. Five experienced electrophysiologists independently evaluated LLM-generated lifestyle guidance and physician-provided counseling responses using ten dimensions. GPT-4o demonstrated a comparable level of scientific consensus to electrophysiologists, while achieving a lower error rate and significantly higher levels of specialized content, empathy, and helpfulness. DB GPT and PubMed GPT showed similar error rates, proportions of specialized content, empathy, and helpfulness compared to electrophysiologists, but exhibited strengths in specialized content in exercise-related and accuracy in diet-related dimensions. These findings suggest that integrating complementary model strengths may help develop safer and more reliable medical AI systems.

Discussion

No takes yet. Share an insight, caveat, or question.

Cite This Study

Makino et al. (2025) studied this question. GPT-4o provided lifestyle guidance for atrial fibrillation patients with comparable scientific consensus to physicians and had a lower error rate with greater helpfulness.

synapsesocial.com/papers/6963221491e05aa366cb8897 https://doi.org/10.1038/s41598-025-34079-z

Topics

Persistent AF management

Atrial fibrillation

View Full Paper

Ask AI

Bookmark

Share

Evaluating the appropriateness and safety of generative AI in delivering lifestyle guidance for atrial fibrillation patients

Figures

Key Points

Structured PICO

Limitations

Implication

Abstract

Discussion

Cite This Study

Topics

Evaluating the appropriateness and safety of generative AI in delivering lifestyle guidance for atrial fibrillation patients

Figures

Key Points

Structured PICO

Limitations

Implication

Abstract

Discussion

Cite This Study

Keep reading

Topics

Keep reading