What question did this study set out to answer?

To establish a unified pretraining framework that achieves strong performance across natural language understanding, conditional text generation, and unconditional generation.

January 1, 2022Open Access

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

View Full Paper

Ask AI

Bookmark

To establish a unified pretraining framework that achieves strong performance across natural language understanding, conditional text generation, and unconditional generation.
Formulated an autoregressive blank-infilling objective that randomly masks continuous spans of tokens and trains the model to sequentially reconstruct them.
Integrated 2D positional encodings with bidirectional attention over unmasked tokens to combine bidirectional contextual learning with autoregressive generation.
Evaluated the pretrained general language model against standard autoencoding, autoregressive, and sequence-to-sequence architectures across NLU and generation benchmarks.
Outperformed standard BERT-style architectures on SuperGLUE natural language understanding benchmarks using identical model scale and pretraining data.
Surpassed baseline sequence-to-sequence and autoregressive language models on document summarization, machine translation, and language modeling tasks.

Discussion

No takes yet. Share an insight, caveat, or question.

Implication

Model evaluation demonstrates improved understanding and generation across diverse benchmarks, indicating unified blank infilling outperforms specialized pretraining architectures.

Cite This Study

Du et al. (2022) studied this question.

synapsesocial.com/papers/69db57d474ec163421835b60 https://doi.org/10.18653/v1/2022.acl-long.26

View Full Paper

Ask AI

Bookmark

Implication

Model evaluation demonstrates improved understanding and generation across diverse benchmarks, indicating unified blank infilling outperforms specialized pretraining architectures.

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

Key Points

Discussion

Implication

Cite This Study

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

Key Points

Discussion

Implication

Cite This Study

Also Consider

Also Consider