Deep Learning with Additive Angular Spatio Temporal Vision Transformer for Multimodal Biometric Recognition Vision Attention Fusion Transformer

Objectives: To achieve audio-visual multimodal biometric recognition with higher accuracy. Method: Deep learning method that incorporates CNN and ViT into a unified framework, called, hybrid convolutional neural network with additive angular vision transformer (CNN-AAViT) introduced for audio-visual multimodal biometric recognition. The key contribution of the proposed method is the deep feature fusion layer that incorporates local spatial features and global context features in one shared feature space. This enables the implementation of face detection and audio recognition utilizing streamlined framework. Findings: results of CNN-AAViT improved in PSNR by 84dB, precision by 0.91, sensitivity by 0.90 and specificity by 0.80 and accuracy by 0.91 using face images. Proposed CNN-AAViT enhanced PSNR by 84dB, precision by 0.89, sensitivity by 0.89 and specificity by 0.87 and accuracy by 0.88 for voice signals when compared to existing methods techniques. Novelty: This work involves the integration of hybrid CNN–ViT architectures for both face and voice biometric modalities within a unified one-shot learning framework. Specifically, CNNs capture fine-grained spatial patterns such as facial features and micro-expressions, while ViTs model long-range dependencies in facial and spectrogram representations. The extracted multimodal features are projected into a shared joint embedding space, enabling direct face–voice association. Furthermore, the use of Additive Angular Margin Loss (AAM-Softmax) enhances inter-class separability and intra-class compactness using cosine similarity, thereby improving the reliability of multimodal matching, particularly when only limited samples are available.