Unpacking Bias in LLMs: A Causal Testing Perspective

Large Language Models (LLMs) – the backbone of modern Natural Language Processing applications (NLP) - are notoriously subject to biases that reflect stereotypes present in their training. Fairness testing typically aims to expose bias by generating texts that should yield equivalent outputs, yet differ only in protected attributes such as gender or religion. However, fairness testing for LLMs is more complex than it seems. Bias depends not only on variations in the protected attribute, but also on its causal interactions with non-protected attributes and their combined influence on the output. Ignoring these dependencies risks producing unrealistic scenarios or misleading conclusions about bias. We introduce CFTA ( Causal Fairness Testing and Analysis ), a causality-driven technique for generating test datasets that uncover multiple forms of bias while enabling fine-grained decomposition of (real or apparent) bias-inducing paths. We evaluate CFTA on nine LLMs on a sentiment analysis task, exposing contributions of both obvious and hidden biases between models.