What question did this study set out to answer?

To evaluate how increasing prompt length impacts the reasoning capabilities of large language models across standardized tasks.

January 1, 2024Open Access

Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models

View Full Paper

Ask AI

Bookmark

Authors

MLMosh LevyBar-Ilan University AJAlon Jacoby YGYoav GoldbergBar-Ilan University

Discussion

Member takes

Implication

Experimental evaluation reveals reasoning performance degradation in large language models as input length increases, highlighting operational limits far below theoretical context windows.

Key Points

To evaluate how increasing prompt length impacts the reasoning capabilities of large language models across standardized tasks.
Designed a question-answering reasoning framework that controls for task difficulty across varying input lengths.
Isolated context length effects by modifying identical samples with padding of different lengths, types, and locations.
Analyzed the relationship between model reasoning performance and standard next-word prediction metrics across multiple dataset variants.
Reasoning performance degraded significantly across all tested models at input lengths far shorter than their technical maximum limits.
Performance degradation trends persisted across every variation of padding type and location tested.
Traditional next-word prediction metrics correlated negatively with reasoning accuracy as input lengths expanded.

Cite This Study

Levy et al. (2024) studied this question.

synapsesocial.com/papers/69dd5be580eea7d3f699bef4 https://doi.org/10.18653/v1/2024.acl-long.818

View Full Paper

Ask AI

Bookmark

Implication

Experimental evaluation reveals reasoning performance degradation in large language models as input length increases, highlighting operational limits far below theoretical context windows.

Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models

Discussion

Member takes

Implication

Key Points

Cite This Study

Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models

Discussion

Member takes

Implication

Key Points

Cite This Study

Also Consider

Also Consider