Logic, inference, understanding: cross-domain generalization for generative language models

Neural systems for Natural Language Inference (NLI) have seen impressive performance over the last ten years, but their ability to generalize beyond their training data has repeatedly been questioned. The NLI task has long been considered as a proxy for the wider problem of Natural Language Understanding (NLU), implicitly motivated by relying on an inferentialist conception of semantics. This paper draws on insights from work in formal logic and semantics to introduce distinctions between different notions of generalizations (in-domain vs. cross-domain, and linguistic vs. inferential) in an attempt to disentangle the problem of generalization. We leverage the theoretical contributions in experiments addressing the inferential generalization power of autoregressive NLI models.