What question did this study set out to answer?

To identify the most effective model architectures and pretraining objectives when constrained by a substantial compute budget of approximately one million GPU hours.

January 1, 2022Open Access

What Language Model to Train if You Have One Million GPU Hours?

View Full Paper

Ask AI

Bookmark

To identify the most effective model architectures and pretraining objectives when constrained by a substantial compute budget of approximately one million GPU hours.
Benchmarked causal decoder-only, non-causal encoder-decoder, and masked language modeling architectures under strictly matched computational budgets.
Evaluated models across diverse downstream natural language processing tasks, multi-task benchmarks, and zero-shot prompted evaluation settings.
Tracked parameter scaling, token throughput, and sample efficiency during training runs scaled up to large-scale GPU clusters.
Autoregressive causal decoder-only architectures achieved the strongest zero-shot generalization and downstream task performance when compute was matched across training runs.
Encoder-decoder models demonstrated competitive parameter efficiency on specific fine-tuning benchmarks but required higher architectural complexity for equivalent generative capability.
Pretraining with multitask prompted targets or causal objectives provided superior downstream scaling returns relative to masked language modeling under fixed compute constraints.

Implication

Comparative empirical study reveals performance trade-offs across transformer architectures under fixed compute budgets, guiding efficient large-scale model pretraining.

Abstract

Teven Le Scao, Thomas Wang, Daniel Hesslow, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong, Julien Launay, Iz Beltagy. Findings of the Association for Computational Linguistics: EMNLP 2022. 2022.

Discussion

No takes yet. Share an insight, caveat, or question.

Cite This Study

Scao et al. (2022) studied this question.

synapsesocial.com/papers/6a0fd7d02badbc352afece65 https://doi.org/10.18653/v1/2022.findings-emnlp.54

View Full Paper

Ask AI

Bookmark

Implication

Comparative empirical study reveals performance trade-offs across transformer architectures under fixed compute budgets, guiding efficient large-scale model pretraining.

What Language Model to Train if You Have One Million GPU Hours?

Key Points

Implication

Abstract

Discussion

Cite This Study

What Language Model to Train if You Have One Million GPU Hours?

Key Points

Implication

Abstract

Discussion

Cite This Study

Keep reading

Keep reading