Key points are not available for this paper at this time.
1565 Background: Generative AI models such as OpenAI’s ChatGPT have been broadly utilized throughout the medical literature. Previous studies have found that AI can generate scientific abstracts which can be difficult to distinguish from the work of human authors. There is a pressing need to characterize utilization of AI in scientific writing to guide policy. Methods: In collaboration with ASCO's Center for Research and Analytics, we extracted text from all scientific abstracts submitted to ASCO 2021 – 2023 Annual Meetings. Likelihood of AI content was evaluated by four AI detectors: GPTZero, Originality.ai (OGAI), Sapling, and Kashyap's AI Content Detector (AICD). Each detector produces a numeric likelihood of AI content. Predictions were dichotomized for uniform analysis across detectors: those in the top 5% for a given detector were considered as having high likelihood of AI content. Logistic regression was used to compute odds ratio (OR) for AI-generated content with respect to submission year and abstract characteristics. Predictions were also assessed for 10 human-written abstracts as negative controls, and 10 produced by OpenAI’s GPT-3 and GPT-4 models as positive controls. Results: 15,553 abstracts met inclusion criteria. 5,179 (33%) were published online only, and 5,327 (34%) referenced registered clinical trials. Across all detectors, abstracts submitted in 2023 were significantly more likely to contain AI content than those in 2021 (OR range 1.3 - 1.7). In abstracts from 2023, AI content score was associated with online only publication, lack of clinical trial number, and abstract track (Table). None of the 10 negative control human written abstracts were identified as AI generated, whereas 100%, 95%, 90%, and 30% of the GPT-3/4 generated abstracts were classified as AI generated by Sapling, GPTZero, OGAI, and AICD respectively using the 5% threshold. Additional results will be presented. Conclusions: AI content detectors uniformly suggest a higher likelihood of AI content generation for abstracts submitted in 2023. Predicted AI content is associated with triage of abstracts to online only presentation, suggesting predicted AI content is associated with lower perceived abstract quality. Further work is needed to understand the accuracy of AI detectors and utility in the abstract review process. Table: see text
Howard et al. (Sat,) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: