Fairness in Audio Sentiment Analysis: A Cross-Gender Evaluation of Machine Learning Models

Audio-based sentiment analysis models in machine learning may show biases across different demographic groups, and fairness in audio sentiment analysis regarding gender remains an unexplored area. This study explored fairness and cross-group generalization in machine-learning-based audio sentiment analysis using the binary female/male speaker metadata provided by the EmoFilm datasets. In this research, we develop an end-to-end pipeline that converts speech audio into spectrograms and extracts visual features like ORB key points with a bag-of-visual-words representation. We also compare classical feature extraction techniques (ORB+BoVW) with modern self-supervised learning embeddings such as wav2vec 2.0 and Whisper, demonstrating that while modern embeddings enhance overall accuracy, fairness issues persist. Using a balanced dataset of 442 audio clips, evenly split between female and male voices, each expressing happiness or sadness, we optimize model hyperparameters for gender-specific and gender-agnostic models. Our findings indicate that gender-specific models achieved the highest accuracy (79.2% for females, 75.3% for males), markedly exceeding the 67.1% accuracy of the gender-agnostic model. Furthermore, cross-gender evaluation revealed accuracy drops while wav2vec 2.0 and Whisper improve overall accuracy and reduce these generalization gaps. These differences reflect acoustic distributions associated with the dataset-provided binary labels, not evidence that sentiment expression is inherently determined by gender identity. To address this, the two-step ensemble approach is retained only as a diagnostic illustration of model specialization under the available binary metadata, not as a recommended deployment strategy. Our research emphasizes importance of subgroup-specific fairness auditing and cautious interpretation of demographic metadata in audio sentiment analysis.