What question did this study set out to answer?

This work aims to understand and mitigate the effects of model collapse in large language models (LLMs) during training.

Learning by Surprise: Adaptive Mitigation of Model Collapse in Large Language Models

View Full Paper

Ask AI

Bookmark

Authors

Simulations reveal new insights on model collapse in generative AI, suggesting novel mitigation strategies.

This work aims to understand and mitigate the effects of model collapse in large language models (LLMs) during training.
Conducted simulations across multiple datasets and LLMs.
Introduced model-intrinsic measures based on next-token probability distributions.
Developed a perplexity-based filtering strategy for fine-tuning.
Model collapse associates with increased concentration of probability mass on a small set of tokens.
Decline in commonsense inference accuracy indicates loss of common sense.
Perplexity-based filtering consistently mitigates model collapse and improves performance compared to human-data baselines.

Gambetta et al. (2026) studied this question.

View Full Paper

Ask AI

Bookmark