Hybrid learning for detection of audio deepfakes and manipulation attacks

Nowadays, the deployment of automatic speaker verification (ASV) systems is a common practice in security-critical applications, such as telemedicine, banking, forensics, and access control. Audio deepfake increased significantly, spreading misinformation, presenting threats to individuals, organizations, and governments, and eroding the trustworthiness of ASV systems. It is essential to develop robust detection methods to address the potential misuse, and maintain authenticity in audio communications, safeguarding voice-based authentication systems. Current works widely employ log-Mel, MFCC for feature extraction, where static logarithmic compression limits dynamic range and sensitivity to acoustic variations. Moreover, conventional methods often struggle to simultaneously capture both spatial and temporal audio features. When effective, they tend to be computationally complex, limiting their capability to extract multiscale contextual features efficiently. This research introduces a novel approach to audio deepfake detection that leverages trainable Per-Channel Energy Normalization (PCEN) integrated within our proposed residual cascaded dilated convolutional network (RCDLNet) architecture to overcome the aforementioned challenges efficiently by capturing multi-scale contextual features while remaining computationally feasible. Complementing this, we propose a Hybrid Focal-BEC loss function that addresses the class imbalance often present in audio deepfake datasets while enforcing discriminative decision boundaries. By integrating margin-based confidence regularization into our loss function, we enhance the model’s ability to make accurate classifications and reduce uncertainty in boundary regions, thereby improving detection accuracy. We evaluate our approach on four benchmark datasets: ASVspoof 2019 LA (12,483 real / 108,978 spoof samples), where our method achieves an EER of 2.81% and a t-DCF of 0.077, outperforming most single-system approaches on this metric; FoR-norm (34,605 / 34,695), where we obtain state-of-the-art performance with 97.45% accuracy; ODSS (11,032 / 18,993), achieving 99.58% accuracy; and GGMDDC (40,000 / 40,000), achieving 99.67% accuracy, demonstrating the effectiveness of the proposed approach across multilingual and cross-lingual conditions while maintaining user trust in automated authentication technologies.