A Comparative assessment of ChatGPT vs. Google Translate for the translation of patient instructions

Background: An estimated 30 million people in the United States demonstrate a non-English language preference (NELP). Professional interpreter services and NELP patient advocates can mitigate some barriers to quality care but are not readily available at all institutions. Many studies have explored the use of Google Translate (GT) for medical communication with mixed results, most concluding that GT alone should not be used in the translation of medical documents for patients. Studies of medical translation with newer artificial intelligence (AI) language models, particularly ChatGPT, do not exist to date. This study's purpose is to compare ChatGPT vs. GT in its ability to accurately translate instructional and educational medical documents across multiple languages. Methods: ChatGPT-3.5 and GT were prompted to translate two English language documents: post-operative discharge instructions for care after circumcision and patient information for undescended testicles (UDT). The instructions were copied and pasted into ChatGPT and GT, and translated into Spanish, Vietnamese, and Russian. Reviewers were blinded as to which documents corresponded to ChatGPT and GT. Members of the Language Services department independently reviewed the translations and assessed them based on meaning, expression, and technical errors. Each translation error was recorded with the reviewer's comment and reviewer's preferred translation. The errors were categorized as meaning, flow, language, form, style guide, or terminology errors, and were labeled as minor, major, or critical. Primary outcome assessed was accuracy of the translation, largely based on retention of meaning. Results: There were 132 sentences between the two documents. In Spanish, ChatGPT incorrectly translated 3.8% of all sentences, while GT incorrectly translated 18.1% of sentences. In Russian, ChatGPT and GT incorrectly translated 35.6% and 41.6% of all sentences, respectively. In Vietnamese, ChatGPT and GT incorrectly translated 24.2% and 10.6% of sentences, respectively. Conclusions: ChatGPT excelled considerably compared to GT for Spanish translation. However, it was outperformed by GT for the Vietnamese translations, and both ChatGPT and GT produced low quality translations in Russian. ChatGPT has an unacceptably high rate of translation error in Vietnamese and Russian and should not be used alone to translate medical documents from English to these languages. While it shows promise in the translation of Spanish materials, its utility for additional languages requires further study and it remains unreliable for use without human oversight.