What question did this study set out to answer?

The research aims to evaluate the effectiveness of generative models in automating the appraisal of randomized controlled trials compared to fine-tuned models.

April 14, 2026Open Access

Zero-shot interpretable biomedical literature appraisal with generative large language models

Key Points

The research aims to evaluate the effectiveness of generative models in automating the appraisal of randomized controlled trials compared to fine-tuned models.
Sampled 800 articles from McMaster Premium LiteratUre Service and Clinical Hedges databases.
Applied two prompting schemes: classifier for independent assessments and verifier for validation of BioLinkBERT.
Appraised performance against human assessments using Matthews correlation coefficient and bootstrapping for confidence intervals.
GPT-4o classifier with full text had an MCC of 0.429, comparable to BioLinkBERT's MCC of 0.466.
GPT-4o outperformed GPT-o3-mini classifier (MCC 0.272) significantly.
Using title and abstract alone resulted in greatly reduced performance for GPT models (MCC ≤0.100).

Abstract

Abstract Objective This study aims to apply 2 decoder-based Generative Pre-trained Transformer (GPT) models (GPT-4o and GPT-o3-mini) in automating the methodological appraisal of randomized controlled trials (RCTs), under a variety of prompt designs, and to compare their performance to a fine-tuned encoder-only BioLinkBERT model Materials and Methods A stratified random sample of 800 articles from the McMaster Premium LiteratUre Service and Clinical Hedges databases was appraised using 2 prompting schemes: (1) classifier (independent assessment) and (2) verifier (validation of BioLinkBERT) considering either the title and abstract (TIAB) or the full text of an article. Performance was primarily evaluated against human assessments using Matthews correlation coefficient (MCC). Bootstrapping over 1000 iterations was used to estimate 95% CIs. Results GPT-4o as a classifier with full text demonstrated comparable performance (MCC 0.429; 95% CI, 0.387-0.470) to BioLinkBERT (MCC, 0.466; 95% CI, 0.409-0.519), drastically outperforming the best GPT-o3-mini scheme (MCC, 0.272; 95% CI, 0.211-0.334). GPT-4o as a verifier with full text showed similar performance (MCC, 0.391; 95% CI, 0.335-0.444). GPT models provided transparent criterion-specific justifications. Performance using TIAB alone markedly decreased for GPT models (MCC, ≤0.100), highlighting dependency on detailed methodological information. Discussion GPT-4o effectively automates RCT critical appraisal with comparable performance to specialized fine-tuned models when provided full text, enhancing interpretability and transparency through explicit justifications. Limitations in abstract-level detail suggest complementary roles for fine-tuned models when full texts are unavailable. Future studies should optimize goal-specific prompting to further facilitate adoption in clinical knowledge translation workflows.

Read Full Paperexternally

Demander à l'IA

Bookmark

View Full Paper