Inducing Grammatical Knowledge from Indirect Evidence in Language Models

Key Points

• The study aims to understand what features and data quantity are needed for language models to correctly judge sentence acceptability using indirect evidence.
• Created the Wug InDirect Evidence Test (WIDET) dataset for evaluating grammatical knowledge.
• Inserted artificial training instances containing unknown pseudo-words into pre-training data.
• Analyzed model responses using evaluation instances with varying levels of indirectness and frequency.
• Models failed to acquire grammatical knowledge despite repeated exposure to training examples that only differed in vocabulary.
• Findings suggest challenges in leveraging indirect evidence for acquiring grammar knowledge in language models.