Background: Triple negative breast cancer (TNBC) is an aggressive subtype of breast cancer characterized by the lack of estrogen receptor(ER), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER2). The absence of these receptors reduces the effectiveness of targeted treatment approaches. With the increasing use of artificial intelligence (AI) in medical research and clinical decision- making, there is growing interest in evaluating the accuracy and reliability of large language models (LLMS), such as chatGPT-4, in oncology related applications. Objective: The research aims to systematically assess the reliability of ChatGPT-4 in addressing frequently asked questions related to TNBC in four critical areas: diagnosis, treatment, prognosis and survival, and quality of life. Expert evaluations and statistical analyses are employed to measure the accuracy of the models responses. Methods: A set of 100 questions related to TNBC was gathered from credible medical sources, including peer-reviewed journals and clinical oncology specialists evaluated the response generated by ChatGPT-4 using a structured assessment framework, classifying each answer into one of four accuracy levels, completely inaccurate, partially accurate, accurate but lacking depth and highly accurate. Results: To evaluate the consistency among reviewers, Cohen’s kappa coefficient was calculated, and descriptive statistical analysis was conducted to identify overall accuracy patterns. The findings indicated that 73% of the responses were classified as either “Accurate” or “Highly Accurate”, suggesting the potential of ChatGPT-4 as a supplementary resource for obtaining information on TNBC. However, 27% of the responses were categorized as “partially accurate” or ‘Completely Inaccurate, “highlighting gaps in contextual understanding and instances of misinformation.. Cohen’s kappa coefficient was recorded at 0.007, reflecting a week level of agreement among evaluators and highlighting the impact of subjective interpretation. The model demonstrated strong performance in well-established areas such as chemotherapy protocols and diagnostic procedures but faced challenges with emerging research topics, personalized treatment recommendations, and fertility related concerns. Conclusion: ChatGPT-4 exhibits significant potential in summarizing information on TNBC; however, the accuracy of its responses varies depending on the complexity and specificity of the queries. Due to inconsistencies and low inter-rater reliability, AI-generated medical content requires verification by medical professionals before being applied to patient care or clinical decision-making. Future developments in large language models should focus on reducing inaccuracies, incorporating the latest medical data, and improving adaptability to better support personalized medicine.
Gummadi et al. (Wed,) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: