ChatGPT hallucinating: can it get any more humanlike?

In its own (paraphrased) words, ChatGPT is ‘an artificial intelligence (AI) chatbot created by OpenAI using a neural network architecture called generative pre-trained transformer to understand natural language and generate humanlike responses to a wide range of questions and prompts; its purpose is to provide users with conversational AI that can assist them with various tasks, provide information, or just chat for fun’. Since its release in November 2022, ChatGPT has become one of the fastest growing consumer-facing applications in history and is disrupting the dissemination of knowledge through its ease of use, massive computational power, and ability to rationally synthesize large amounts of data available on the internet. In healthcare, ChatGPT may be used as a rapid resource for medical information about simple and complex concepts that may take the place of conventional searches of online bibliographical sources executed by the user. However, ChatGPT has not been trained specifically for healthcare or medical research. The accuracy of ChatGPT in delivering accurate medical information is unknown. Herein, we demonstrate an example of ChatGPT providing completely non-factual responses, including references to non-existent research papers, when asked to address a clinical question. The clinical scenario pertained to a patient who had undergone cardiac surgery a few days prior and required a secondary prevention implantable cardioverter-defibrillator (ICD) before dismissal from the hospital due to recent history of ventricular tachycardia. Among potential ICD options, we considered the subcutaneous ICD. Specific concerns with implanting this system in the immediate post-operative setting include potential disruption of the recent surgical incision during subcutaneous ICD (S-ICD) lead tunnelling and potentially unreliable surface electrocardiogram (ECG) vector analysis due to depolarization and repolarization abnormalities related to the immediate post-operative status. To our knowledge, there are no published data regarding the feasibility, safety, and effectiveness of this type of ICD implanted shortly after cardiac surgery. ChatGPT (non-subscription version 3.5) was queried on 31 March 2023, and the responses are reproduced verbatim in Box 1. As shown, ChatGPT initially refers to two different studies published in 2016 and 2017 reporting on the immediate post-surgical implantation of the S-ICD. However, a quick literature search failed to identify such studies. Further, implantation of an ICD within 24 h after cardiac surgery, as indicated in ChatGPT’s response, is almost implausible as there would be no clinical reason for an ICD to be implanted so early post-operatively. Therefore, we then asked ChatGPT to provide the specific references. In the second response, three other references are provided along with their complete bibliographic details, including journal, volume, issue, and even digital object identifier (DOI) and PubMed IDs (PMIDs) (though not the authors’ names). These realistic references are actually non-existent, and the PMIDs correspond to completely unrelated research papers. Question #1: Have there been any reports of implantation of a subcutaneous defibrillator immediately after cardiac surgery? ChatGPT: Yes, there have been published reports of implantation of subcutaneous defibrillators immediately following cardiac surgery during the same hospitalization. For example, a study published in the Journal of Cardiovascular Electrophysiology in 2016 reported on the outcomes of 25 patients who underwent subcutaneous defibrillator implantation within 24 h of cardiac surgery. The study found that the procedure was safe and effective, with no major complications or device-related infections reported during a median follow-up of 12 months. Another study published in the European Heart Journal in 2017 reported on a larger series of 55 patients who underwent subcutaneous defibrillator implantation immediately following cardiac surgery. The study found that the procedure was feasible and safe, with no device-related infections or major complications reported during a median follow-up of 9 months. Overall, these studies suggest that subcutaneous defibrillator implantation immediately following cardiac surgery is a safe and effective option for patients who require implantable defibrillator therapy. However, as with any medical procedure, individual patient factors and clinical circumstances should be carefully considered when making treatment decisions. Question #2: Have there been any reports of implantation of a subcutaneous defibrillator immediately after cardiac surgery and please provide references. ChatGPT: Yes, there have been reports of implantation of subcutaneous defibrillators (S-ICDs) immediately after cardiac surgery. Here are some references: 1. ‘Immediate postoperative implantation of a subcutaneous defibrillator in a patient undergoing cardiac surgery.’ J Interv Card Electrophysiol. 2016 Feb;45(2):203–5. doi: 10.1007/s10840-015-0087-6. Epub 2015 Dec 17. PMID: 26676620. 2. ‘Immediate postoperative subcutaneous implantable cardioverter-defibrillator implantation in patients undergoing cardiac surgery: feasibility, safety and outcomes.’ Europace. 2017 Jul 1;19(7):1150–1156. doi: 10.1093/europace/euw235. PMID: 27614059. 3. ‘Immediate postoperative implantation of a subcutaneous defibrillator in a patient undergoing cardiac surgery.’ J Arrhythm. 2017 Dec;33(6):639–641. doi: 10.1016/j.joa.2017.06.008. Epub 2017 Jul 5. PMID: 29276432. These studies suggest that immediate post-operative implantation of S-ICDs in patients undergoing cardiac surgery is feasible and safe. However, as with any medical procedure, there may be individual patient factors that need to be considered before making a decision about whether to perform the implantation immediately after surgery. It is important to discuss these factors with the patient’s healthcare team. (Variations of these responses, all erroneous, were given when ChatGPT was asked the exact same question different times.) While concerns have already been raised regarding the impact of ChatGPT on the authenticity of academic writing,1 this powerful tool is still at its infancy and little is known about other potential drawbacks. Herein, we demonstrate a striking use case scenario where ChatGPT provided seemingly realistic, but highly misleading and factually incorrect information that should raise concern. This behaviour of generative AI language models, such as ChatGPT, fabricating text that is not based on any real-world knowledge or context but rather on patterns and associations it has learned from its training data, has been described as ‘hallucinating’2 (aka confabulating). The phenomenon has been reported in other areas,3–5 but to our knowledge, this is the first such report with relevance to cardiovascular medicine. The exact reasons for the generation of fabricated references in our particular example are unknown as the model’s underlying reasoning and granular representations of the input data are obscure and not accessible to the user. In general, several technical reasons contributing to hallucinations have been proposed, including heuristic data collection mismatching the input data source and the output target, deficient encoding of input data/text, deficient decoding to generate correct output from the encoded input data, exposure to biased or incorrect input data, and parametric knowledge bias.6 The extent and implications of such a flawed behaviour are yet unknown, but it should be emphasized that the developer of ChatGPT provides clear disclaimers that inaccuracies may be present. ChatGPT was not specifically developed to generate accurate medical knowledge. Rather, it was trained to produce sensible, humanlike responses based on general cognitive capabilities rather than expertise in any specific domain. This open access and easy-to-use tool is prone to ‘off-label’ use not unlike many other promising prognostic, diagnostic, and treatment tools that are accessible to clinicians (and patients). In our opinion, off-label use of interventions is often useful and in fact necessary when dealing with unusual clinical scenarios not covered under the regulated, approved indications for use or when the pace of technological advancements addressing important needs exceeds the pace of regulatory oversight. We believe that tools such as ChatGPT fall into the latter category. The constant strive to increase the efficiency of clinical practice and research, the need to rapidly screen through massive volumes of literature, and the desire to simplify clerical tasks make large language models attractive applications at the clinician’s fingertips. However, it is also evident that there are real risks with off-label use of any tool. In this viewpoint, our goal was to highlight and raise awareness in the cardiology community about one of these risks as it applies to a real clinical scenario. Users not aware of this aberrant behaviour of chatbots may be at risk of false adoption of realistic-appearing, but clearly inexistent information. It is becoming increasingly evident that current iterations of chatbots have limited ability to provide guidance that requires deep subject matter knowledge and nuanced decision-making as is the case with the complex scenarios that clinicians often face in contemporary practice. Despite these limitations, AI based on advanced neural network methodology has a transformative potential across the spectrum of cardiovascular medicine. Several applications developed using large data sets and raw data inputs from cardiac diagnostic studies (electrocardiography, imaging, and others) are already being used as diagnostic and predictive classifiers of cardiovascular disease. One important difference between the popular large language models, such as ChatGPT, and the diagnostic/predictive deep learning-based AI tools is that the former have been developed using relatively uncurated open source data (essentially the entire public web domain), while the latter have been typically developed from highly curated data sets from academic institutions. After appropriate measures are implemented to address issues such as hallucinations and to refine their exposure to high-quality input sources, generative AI models should help not only with knowledge dissemination, but they may also prove valuable in automating, simplifying, and optimizing routine tasks in clinical medicine that are currently manual and often inefficient (record abstraction, documentation, and patient education, among others). In the research domain, large language models may help to efficiently summarize large volumes of data from the electronic health record in order to address clinical questions but also to identify and recruit subjects for clinical trials, decentralize follow-up assessment of outcomes, and aid with decision support systems. While purpose-built generative and diagnostic/predictive AI tools will serve seemingly distinct roles in healthcare, research, and education, future AI platforms may integrate such tools to complement each other in these different roles. For example, large language models (chatbots) may incorporate diagnostic and predictive neural networks to provide end-to-end solutions, from ingesting source data and processing them with the diagnostic/predictive models to generating sensible results and recommendations that can be communicated to providers and patients. The standards to which chatbots will be held and the guardrails for their applications in medicine remain to be defined. Future iterations of ChatGPT may address the issue of hallucinations along with other performance upgrades. Very recently, OpenAI researchers proposed the method of ‘process supervision’ for training more reliable generative models. This method provides feedback to the model for each reasoning step ultimately leading to superior performance in solving complex problems compared with the more conventional ‘outcome supervision’ method which provides feedback to the model only for the end result of a potentially long chain of reasoning.7 The effectiveness of these improvement efforts may help define the role of chatbots as trustworthy resources in medicine. However, regardless of whether the hallucination problems are adequately addressed, users should be aware of the spectrum of chatbots’ capabilities and limitations. Cautious use and vigilant fact-checking are as important as ever. A number of AI ECG algorithms have been licenced by Mayo Clinic to Anumana, Eko Health, and AliveCor, and Mayo Clinic, the authors, and other co-inventors may benefit financially from their commercialization.