Stress-Testing Explainable Hate Speech Detection for Roman Urdu: Baselines, Faithfulness and Religious–Political Bias

Hate speech detection for Roman Urdu (Urdu written in Latin script, with inconsistent spelling and frequent code-switching with English) remains under-studied. We re-implement and extend the BERT+CNN-gram classifier of Rizwan et al. (2020) on the RUHSOLD dataset and ask three questions: How does it compare with baselines re-trained under identical conditions? Are its explanations faithful? Where does it fail? Key findings Performance: On the official test split, the model reaches macro-F1 0.902, ROC-AUC 0.962 and MCC 0.805. Five-fold cross-validation gives a lower, more conservative macro-F1 of 0.872 ± 0.011. Baselines: The model is statistically indistinguishable from XLM-R+CNN-gram, FastText+CNN-gram and a TF-IDF baseline. It is only marginally better than plain mBERT fine-tuning (McNemar p = 0.0498). Explanations: Word-level SHAP and Integrated Gradients agree on 16 of their 20 most important words. Faithfulness: In an ERASER-style deletion test, removing the words ranked most important by occlusion lowers the predicted-class probability by 0.42 on average, against 0.10 for random removal. Bias: False-positive rates are 0.22 and 0.24 on posts with religious or political terms, compared with 0.08 on other posts. Embedding space: Fine-tuning raises the silhouette score of the two classes from 0.02 to 0.52. Code and data: All code and results are publicly available at https://github.com/quratulain77-saeed/Hate-Speech-detection-in-text