FedSER-XAI: PSO-Optimized Multi-StreamCross-Attention Transformer with Graph Featuresfor Explainable Federated Speech EmotionRecognitione

Abstract Federated learning for speech emotion recognition faces fundamental challengesin simultaneously achieving high performance, privacy preservation, and modelinterpretability. This paper introduces FedSER-XAI, a novel framework thatintegrates Particle Swarm Optimization (PSO)-based feature selection, multi-stream cross-attention mechanisms, and graph-based feature extraction withinan explainable federated learning architecture. Our approach combines VisionTransformer processing of mel-spectrograms with temporal-spatial graph convo-lutional networks to capture both contextual and structural speech relationships.The PSO algorithm achieves 78.1% dimensionality reduction (228→50 features)while improving discriminative power. The multi-stream architecture processestraditional acoustic features alongside novel graph-based representations derivedfrom visibility and correlation graphs, fused through Transformer-based cross-attention mechanisms. Extensive evaluation on EMODB and SAVEE datasetsdemonstrates exceptional performance: 99.9% and 97.2% accuracy in central-ized settings, with remarkable federated performance of 99.7% and 97.2% using8 emotion-specialized clients. The framework achieves rapid convergence within110 communication rounds, representing minimal performance degradation (0.2%for EMODB) while preserving privacy. Cross-dataset evaluation on CREMA-Dyields 68% accuracy, demonstrating reasonable generalization. The compre-hensive explainability framework using SHAP and LIME provides global andlocal interpretations, validating that graph-based features contribute significantlyto emotion discrimination. FedSER-XAI represents the first explainable feder-ated speech emotion recognition system, advancing trustworthy AI for sensitivehealthcare and human-computer interaction applications