Phishing detection by Transformers: Adversarial Robustness and Cross-Lingual Generalization

Abstract Objectives:To evaluate cross-lingual performance and effectiveness against adversarial attacks. Method: this study fine-tuned four transformer models: BERT, RoBERTa, mBERT and XLM-RoBERTa. The proposed methodology involves training the models on the baseline data (English dataset G1) and then evaluating them on other data organized into four groups. Findings: Results revealed that the performance of RoBERTa and BERT collapses in cross-lingual transfer, whereas XLM-RoBERTa maintains its performance (93.43% versus 93.41% on the baseline). This suggests the benefit of the multilingual structure beyond the data. Furthermore, adversarial evaluation allowed to show that transformer models struggle to detect sophisticated phishing emails. Through adversarial training, the proposed method notably achieved a 6.54% increase in F1 Score for RoBERTa. Novelty: This study proposes that combining multilingual models with adversarial training ensures effectiveness against adversarial attacks while providing a high-performing multilingual system for phishing detection. Keywords: Phishing, Cross-Lingual, Adversarial Robustness, Zero-shot transfer, Transformer