What Is Your Favorite Gender, MLM? Gender Bias Evaluation in Multilingual Masked Language Models

Bias is a disproportionate prejudice in favor of one side against another. Due to the success of transformer-based masked language models (MLMs) and their impact on many NLP tasks, a systematic evaluation of bias in these models is now needed more than ever. While many studies have evaluated gender bias in English MLMs, only a few have explored gender bias in other languages. This paper proposes a multilingual approach to estimating gender bias in MLMs from five languages: Chinese, English, German, Portuguese, and Spanish. Unlike previous work, our approach does not depend on parallel corpora coupled with English to detect gender bias in other languages using multilingual lexicons. Moreover, a novel model-based method is presented to generate sentence pairs for a more robust analysis of gender bias. For each language, lexicon-based and model-based methods are applied to create two datasets, which are used to evaluate gender bias in an MLM specifically trained for that language using one existing and three new scoring metrics. Our results show that the previous approach is data-sensitive and unstable, suggesting that gender bias should be assessed on a large dataset using multiple evaluation metrics for best practice.