What question did this study set out to answer?

To introduce PubMedQA, a biomedical question answering dataset designed to evaluate model reasoning over quantitative findings in biomedical research abstracts.

September 13, 2019Open Access

PubMedQA: A Dataset for Biomedical Research Question Answering

View Full Paper

Ask AI

Bookmark

Authors

QJQiao JinNorth Carolina State University BDBhuwan DhingraDuke University ZLZhengping LiuShandong Academy of Pharmaceutical Sciences

Discussion

Member takes

Implication

Machine learning evaluation demonstrates language models achieve 68.1% accuracy on biomedical abstracts, highlighting a significant performance gap compared to human experts.

Key Points

To introduce PubMedQA, a biomedical question answering dataset designed to evaluate model reasoning over quantitative findings in biomedical research abstracts.
Curated a dataset comprising 1k expert-annotated, 61.2k unlabeled, and 211.3k artificially generated QA instances extracted from PubMed abstracts.
Structured each instance with a question, context (abstract without conclusion), long answer (conclusion), and a categorical yes/no/maybe label.
Evaluated a multi-phase fine-tuned BioBERT model incorporating bag-of-words statistics from long answers as auxiliary supervision against baseline models and human performance.
The best-performing multi-phase fine-tuned BioBERT model achieved an accuracy of 68.1%.
Model accuracy surpassed the majority-class baseline of 55.2% but remained below single human expert accuracy of 78.0%.

Cite This Study

Jin et al. (2019) studied this question.

synapsesocial.com/papers/6a0da36168ddba849a09ced5 https://doi.org/10.48550/arxiv.1909.06146

View Full Paper

Ask AI

Bookmark

Implication

Machine learning evaluation demonstrates language models achieve 68.1% accuracy on biomedical abstracts, highlighting a significant performance gap compared to human experts.

PubMedQA: A Dataset for Biomedical Research Question Answering

Discussion

Member takes

Implication

Key Points

Cite This Study

PubMedQA: A Dataset for Biomedical Research Question Answering

Discussion

Member takes

Implication

Key Points

Cite This Study

Also Consider

Also Consider