Fairness Evaluation in Text Classification: Machine Learning Practitioner Perspectives of Individual and Group Fairness

Mitigating algorithmic bias is a critical task in the development and of machine learning models. While several toolkits exist to aid learning practitioners in addressing fairness issues, little is known the strategies practitioners employ to evaluate model fairness and what influence their assessment, particularly in the context of text. Two common approaches of evaluating the fairness of a model are fairness and individual fairness. We run a study with Machine Learning (n=24) to understand the strategies used to evaluate models. presented to practitioners (group vs. individual fairness) impact which they consider fair. Participants focused on risks associated with/overpredicting and model sensitivity relative to identity token. We discover fairness assessment strategies involving personal or how users form groups of identity tokens to test model fairness. provide recommendations for interactive tools for evaluating fairness in classification.