Multimodal sEMG and Accelerometer Data Fusion for Fine-Grained Hand Gesture Recognition

Surface electromyography (sEMG) has become a key sensing modality for hand gesture recognition in rehabilitation, prosthetic control, and human–computer interaction. However, its sparse activation patterns and sensitivity to subject- and recording-related variability pose significant challenges, particularly for fine-grained gestures involving subtle finger and wrist motions. To address these limitations, we propose an attention-enhanced multimodal fusion framework that integrates raw sEMG and accelerometer (ACC) signals. The proposed network employs narrow-kernel temporal convolutions in dedicated modality-specific streams to preserve fine-grained neuromuscular patterns, followed by a dual-attention fusion module combining multi-head self-attention (MHSA) and gated attention to model interactions between sEMG and ACC embeddings and improve gesture separability. Experiments on the NinaPro DB2 and DB7 databases show that the proposed fusion model outperforms both the sEMG-only and ACC-only models. The feature-level fusion model achieves accuracies of 96.42% on DB2 and 97.03% on DB7 without data augmentation. After applying the selected combined augmentation strategy based on time warping and scaling, the accuracy further increases to 97.56% and 98.19%, respectively. A decision-level fusion variant also achieves comparable performance while keeping the sEMG and ACC classifiers independent. Detailed gesture-level analysis further shows that the largest gains are obtained for low-amplitude and highly similar gestures, highlighting the value of multimodal fusion for improving the separability of subtle hand movements.