Language Models as Knowledge Bases?

Fabio Petroni; Tim Rocktäschel; Sebastian Riedel; Patrick Lewis; Anton Bakhtin; Yuxiang Wu; Alexander Miller

doi:10.18653/v1/d19-1250

January 1, 2019Open Access

Language Models as Knowledge Bases?

View Full Paper

Ask AI

Bookmark

Share

Key Points

To assess whether pretrained language models store factual and relational knowledge without explicit supervision or fine-tuning.
Constructed the Language Model Analysis (LAMA) probe using cloze-style query templates across multiple knowledge sources including Wikidata, ConceptNet, and SQuAD.
Evaluated zero-shot retrieval accuracy across diverse pretrained architectures, including BERT, ELMo, and Transformer-XL.
Compared unsupervised language model predictions against supervised relation extraction pipelines and knowledge graph completion baselines.
Pretrained BERT models retrieved relational factual knowledge without fine-tuning, matching or outperforming traditional knowledge base baselines on selected relation sets.
Masked language models demonstrated higher factual precision than standard unidirectional autoregressive language models across cloze-style probes.

Implication

Comparative evaluation reveals competitive factual knowledge retrieval in pretrained neural language models, highlighting their capacity to act as open knowledge bases.

Abstract

Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Discussion

No takes yet. Share an insight, caveat, or question.

Cite This Study

Petroni et al. (2019) studied this question.

synapsesocial.com/papers/698c9cb8cf5273e909337845 https://doi.org/10.18653/v1/d19-1250

View Full Paper

Ask AI

Bookmark

Share

Language Models as Knowledge Bases?

Key Points

Implication

Abstract

Discussion

Cite This Study

Language Models as Knowledge Bases?

Key Points

Implication

Abstract

Discussion

Cite This Study

Keep reading

Keep reading