What question did this study set out to answer?

This paper addresses the problem of data contamination in NLP evaluations, particularly regarding large language models.

January 1, 2023Open Access

NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark

View Full Paper

Ask AI

Bookmark

Authors

Position paper argues for improved methods to detect data contamination in NLP benchmarks, suggesting community engagement.

This paper addresses the problem of data contamination in NLP evaluations, particularly regarding large language models.
Defined various levels of data contamination in NLP benchmarks.
Proposed the need for automatic and semi-automatic measures to assess model exposure to benchmark data.
Recommended flagging compromised conclusions in published papers.
Identified that contaminated models show overestimated performance in benchmark evaluations.
Highlighted potential for harmful scientific conclusions resulting from data contamination.
Stressed the necessity for community awareness and action against this issue.

Sainz et al. (2023) studied this question.

View Full Paper

Ask AI

Bookmark