What question did this study set out to answer?

To investigate the training and exact memorization dynamics of large language models across different model sizes, dataset sizes, and learning rates.

May 22, 2022Open Access

Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models

View Full Paper

Ask AI

Bookmark

Authors

KTKushal TirumalaCalifornia Institute of Technology AMAram MarkosyanMeta (United States)LZLuke ZettlemoyerUniversity of Washington

Discussion

Member takes

Implication

Empirical study demonstrates faster data memorization and delayed overfitting in large language models, indicating that scaling fundamentally alters retention dynamics.

Key Points

To investigate the training and exact memorization dynamics of large language models across different model sizes, dataset sizes, and learning rates.
Empirically tracked exact memorization throughout the training process in both causal and masked language modeling architectures.
Evaluated the impact of dataset size, learning rate, and model capacity on memorization and forgetting rates.
Analyzed token-level memorization patterns across different parts of speech, focusing on nouns and numbers.
Larger models memorized training examples significantly faster across all tested settings and forgot less data during training.
Larger architectures retained a higher proportion of training data before exhibiting overfitting compared to smaller models.
Models consistently memorized nouns and numbers first, which functioned as unique identifiers for recalling entire individual training examples.

Cite This Study

Tirumala et al. (2022) studied this question.

synapsesocial.com/papers/6a0ede302eca052da647dc69 https://doi.org/10.48550/arxiv.2205.10770

View Full Paper

Ask AI

Bookmark

Implication

Empirical study demonstrates faster data memorization and delayed overfitting in large language models, indicating that scaling fundamentally alters retention dynamics.

Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models

Discussion

Member takes

Implication

Key Points

Cite This Study

Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models

Discussion

Member takes

Implication

Key Points

Cite This Study

Also Consider

Also Consider