EnCoNet: An Ensemble-based Synthetic Speech Attribution Scheme with Convolutional Neural Networks

Abstract Due to the advancement of generative artificial intelligence (AI) technologies, differentiating between human voice and generated voice has become a challenging problem. Addressing this problem requires a better understanding and analysis of the characteristics of natural speech and AI-generated speech. However, these characteristics can differ in real-world scenarios, where multiple sources interact together. Relying on manual feature engineering or large neural networks as a solution to this speech attribution problem is often not suitable for practical implementations. To this end, lightweight deep learning-based solutions can help identify the subliminal artifacts present in the AI-generated speech by abstract- ing away the complexities and providing end-to-end solutions for synthetic speech attribution tasks. In this work, we propose an ensemble-based synthetic speech attribution algorithm EnCoNet that employs light-weight one-dimensional (1D) convolutional neural networks. Additionally, the effectiveness of a semi-supervised learning approach is demonstrated to ensure the generalization and robustness of the proposed algorithm. The proposed network is trained on the Signal Processing Cup (SP Cup) 2022 dataset. The evaluation scores in terms of accuracy of the proposed model on two weakly distorted evaluation datasets of Signal Processing Cup 2022 are 96.4% and 98.2%, respectively, whereas for the highly distorted evaluation dataset, the score is 50.87%. Additionally, the proposed model’s evaluation score on custom-made synthetic classes (from ASVspoof 2019 dataset) is 98%, which strongly indicates the robustness and generalization capabilities of the proposed approach. Finally, the results obtained using the proposed method are compared with the existing works and it outperforms them in terms of accuracy, which indicates the effectiveness of the proposed ensemble-based network.