Transformer models as predication machines

Predication is the process by which the meaning of words are altered as a consequence of the contexts in which they appear. Kintsch provides an algorithm to capture this process. The model is based on Latent Semantic Analysis and has the advantage that it relies only on the statistical analysis of word occurrence—and hence provides a coherent and practical account of how representational content might come to be. Similarly, large language models—such as the Bidirectional Encoder Representations from Transformers (BERT) update the initially context-independent representations of words based on their context of use in a data driven way. In a sense, BERT models are a series of predication layers. In this paper, predication, BERT and the relationship between them is explained.