Accuracy of artificial intelligence in meta-analysis: A comparative study of ChatGPT 4.0 and traditional methods in data synthesis

Meta-analysis is a critical tool in evidence-based medicine, particularly in cardiology, where it synthesizes data from multiple studies to inform clinical decisions. This study explored the potential of using ChatGPT to streamline and enhance the meta-analysis process. To investigate the potential of ChatGPT to conduct meta-analyses in interventional cardiology by comparing the results of ChatGPT-generated analyses with those of randomly selected, human-conducted meta-analyses on the same topic. We systematically searched PubMed for meta-analyses on interventional cardiology published in 2024. Five meta-analyses were randomly chosen. ChatGPT 4.0 was used to perform meta-analyses on the extracted data. We compared the results from ChatGPT with the original meta-analyses, focusing on key effect sizes, such as risk ratios (RR), hazard ratios, and odds ratios, along with their confidence intervals (CI) and P values. The ChatGPT results showed high concordance with those of the original meta-analyses. For most outcomes, the effect measures and P values generated by ChatGPT closely matched those of the original studies, except for the RR of stent thrombosis in the Sreenivasan et al study, where ChatGPT reported a non-significant effect size, while the original study found it to be statistically significant. While minor discrepancies were observed in specific CI and P values, these differences did not alter the overall conclusions drawn from the analyses. Our findings suggest the potential of ChatGPT in conducting meta-analyses in interventional cardiology. However, further research is needed to address the limitations of transparency and potential data quality issues, ensuring that AI-generated analyses are robust and trustworthy for clinical decision-making.