Preventing harm from non-conscious bias in medical generative AI

Large language models such as OpenAI's GPT-4 have the potential to transform medicine1Clusmann J Kolbinger FR Muti HS et al.The future landscape of large language models in medicine.Commun Med (Lond). 2023; 3: 141Crossref PubMed Google Scholar by enabling automation of a range of tasks, including writing discharge summaries,2Patel SB Lam K ChatGPT: the future of discharge summaries?.Lancet Digit Health. 2023; 5: e107-e108Summary Full Text Full Text PDF PubMed Scopus (159) Google Scholar answering patient questions,3Ayers JW Poliak A Dredze M et al.Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum.JAMA Intern Med. 2023; 183: 589-596Crossref PubMed Scopus (156) Google Scholar and supporting clinical treatment planning.4Dennstädt F Hastings J Putora PM et al.Exploring the capabilities of large language models such as ChatGPT in radiation oncology.Adv Radiat Oncol. 2023; (published online Nov 4.)https://doi.org/10.1016/j.adro.2023.101400Summary Full Text Full Text PDF Google Scholar These models are so useful that their adoption has been immediate, and efforts are already well underway to integrate them with ubiquitous clinical information systems. However, the unchecked use of the technology has the potential to cause harm. An ambitious study by Travis Zack and Eric Lehman and colleagues5Zack T Lehman E Suzgun M et al.Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study.Lancet Digit Health. 2024; 6: e12-e22Summary Full Text Full Text PDF Google Scholar in The Lancet Digital Health comprehensively shows that GPT-4 exhibits racial and gender bias across clinically relevant tasks, including the generation of cases for medical education, support for differential diagnostic reasoning, medical plan recommendation, and subjective assessments of patients. For each of these tasks, GPT-4 was found to exaggerate known disease prevalence differences between groups, over-represent stereotypes including problematic representations of minority groups, and amplify harmful societal biases. These findings are seriously concerning and are in line with previous research about bias in large-scale generative artificial intelligence models more broadly. However, the study falls short of providing actionable recommendations on how the technology can safely be incorporated into clinical workflows given these findings. Strategies to mitigate bias in language models are a very active research area, commonly divided into approaches that augment or alter the model training data, strategies that enhance or provide additional model training, strategies that alter the model execution, and strategies that apply corrections post-processing.6Gallegos IO Rossi RA Barrow J et al.Bias and fairness in large language models: a survey.arXiv. 2023; (published online Sept 2.) (preprint).https://doi.org/10.48550/arXiv.2309.00770Google Scholar Supplementing such models with additional training through data augmentation or additional reinforcement learning from human feedback might help to some extent, but are unlikely to be completely successful as it is largely outside of the potential of the underlying technology to avoid bias completely. In addition, as Zack and colleagues5Zack T Lehman E Suzgun M et al.Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study.Lancet Digit Health. 2024; 6: e12-e22Summary Full Text Full Text PDF Google Scholar note, hospital information systems adopting a closed model such as GPT-4 have limited potential to redevelop the model itself, and even though this is in principle possible with open-source models, it is unlikely that individual hospitals have sufficient resources to resolve all the underlying problems. Thus, it follows that biased models will be deployed. Attempts to debias the results through instruction are also likely to fail: language models have no possibility for self-awareness or self-reflection, and no consistent model of the world to which their outputs can be expected to correspond. Thus, as Zack and colleagues5Zack T Lehman E Suzgun M et al.Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study.Lancet Digit Health. 2024; 6: e12-e22Summary Full Text Full Text PDF Google Scholar observed in their study, explicitly instructing the model to perform equitably or to avoid bias will probably not have the desired outcome, and might result in the model overcorrecting to a worse bias. Worryingly, even deployment with a clinician in the loop will not fully address the problem—biased model behaviour might lead the clinician to become more biased, rather than the desired outcome of the clinician reducing model bias.7Vicente L Matute H Humans inherit artificial intelligence biases.Sci Rep. 2023; 1315737Crossref Scopus (0) Google Scholar How, then, can these models be safely deployed? It is likely that there is no one-size-fits-all solution, but rather, a range of complex preprocessing and postprocessing workarounds will have to be developed depending on the nature of the task. For example, to mitigate bias in the generation of example case vignettes for use in medical education, it might suffice if the prompts for the generation of the case vignettes already prespecified the desired demographics for the case examples according to a detailed and preverified estimate of prevalence. To mitigate bias in the generation of differential options and considerations in support of clinical diagnostic reasoning and treatment planning, it seems imperative that the model be coupled to an external and verifiable source of bias-free knowledge using retrieval-augmented8Shi W Zhuang Y Zhu Y Iwinski H Wattenbarger M Wang MD Retrieval-augmented large language models for adolescent idiopathic scoliosis patients in shared decision-making.https://dl.acm.org/doi/10.1145/3584371.3612956Date accessed: November 28, 2023Google Scholar generation, rather than using suggestions generated directly by the model. Finally, for tasks related to the subjective evaluation of patient characteristics such as level of honesty and pain severity, it might be altogether inappropriate to apply the technology at this stage of development to this type of task. Advances in language technology offer many exciting possibilities for medicine, including possibilities to reduce the time-consuming burden of documentation9Wosny M Strasser LM Hastings J Experience of health care professionals using digital tools in the hospital: qualitative systematic review.JMIR Hum Factors. 2023; 10e50357Crossref PubMed Scopus (0) Google Scholar and thereby free more clinical time for human-focused work. In such workflows, biases might still have an effect that should be considered, but the overall net benefit will be enormous, and the risk mitigation might be more straightforward than for other applications of the technology. However, for tasks such as clinical decision making and treatment planning, the risks are potentially severe and there is a substantial requirement for diligent implementation strategies to avoid repeating algorithmic harms such as those that have been made in the past.10Obermeyer Z Powers B Vogeli C Mullainathan S Dissecting racial bias in an algorithm used to manage the health of populations.Science. 2019; 366: 447-453Crossref PubMed Scopus (1746) Google Scholar I declare no competing interests. Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation studyOur findings highlight the urgent need for comprehensive and transparent bias assessments of LLM tools such as GPT-4 for intended use cases before they are integrated into clinical care. We discuss the potential sources of these biases and potential mitigation strategies before clinical implementation. Full-Text PDF Open Access