Language models have achieved remarkable performance gains across multiple natural language processing and understanding tasks. They were shown to capture many high-level aspects of natural human language. However, the complexity of these models and their black-box nature make it difficult to understand their behaviour based on fine-grained explanations. In this paper, we present high-level concept-based explanations for neural language models with a classification task setup using the quantitative testing with concept activation vectors (TCAVq). TCAVq explains a neural model based on its activations in response to concepts present in the data. We propose a pipeline that automates the discovery of these concepts by clustering the model’s activations. The pipeline was tested on (BERT) but can be applied to different neural architectures. We perform ablation and injection studies with the top ranking concepts to evaluate the causality and impact of the explanations provided in terms of change in the model’s predictive performance. The ablation studies show a 2.6% impact on the model’s sensitivity while injection shows up to a 12.1% impact on specificity attributed to the top scoring concepts. This illustrates the potential of using concept-based explanations to provide a human intelligible insight into the model’s decisions by examining the concepts extracted and how they contribute to the model’s predictions. An implementation of our pipeline can be found on GitHub 1
ElSayed et al. (Thu,) studied this question.