What type of study is this?

This is a Experimental Study study.

October 10, 2025Open Access

Scaling Laws Revisited: Modeling the Role of Data Quality in Language Model Pretraining

View Full Paper

Ask AI

Bookmark

Authors

This research introduces a quality-aware scaling law for language models, predicting loss based on data quality and model size.

The proposed quality-aware scaling law predicts loss as a function of model size, data volume, and data quality.
Synthetic experiments demonstrate that improved data quality can significantly reduce model size and computation needs.
Our findings indicate a sublinear decay of effective data with increasing quality, showing robustness to moderate data corruption.
This work establishes a generalizable law for data quality, aiding in balancing data curation efforts with model scale.

Subramanyam et al. (2025) studied this question.

View Full Paper

Ask AI

Bookmark