Empowering singers: A deep learning framework for real-time vocal feedback

Automated vocal performance evaluation requires the simultaneous modeling of diverse acoustic attributes, including stylistic technique, melodic accuracy, and subjective quality. In this study, the Disentangled Adversarial Vocal Analyzer (DAVA) is proposed as a novel Multi-Task Learning (MTL) framework for singing analysis through the joint optimization of technique classification, pitch detection, and aesthetic scoring. Unlike traditional monolithic architectures, a dual-branch Transformer-based system is employed in DAVA and integrated with an Adversarial Disentanglement module. Through this module, timbral and pitch-related features are explicitly decoupled using a conditional Stein score objective, thereby reducing task interference and improving feature purity. To capture fine-grained vocal nuances, dual-modality inputs, including Mel-Frequency Cepstral Coefficients (MFCCs) and log-Mel spectrograms, are used and further refined through self-supervised pre-training with the data2vec algorithm. Stable convergence across heterogeneous tasks is supported by GradNorm-based dynamic gradient balancing, while Focal Loss is used to address class imbalance within the VocalSet dataset. Experimental results show that DAVA achieves a macro-F1 score of 92.96% for technique classification, a top-3 pitch accuracy of 88.47%, and an R 2 of 0.846 for aesthetic scoring. In addition, cross-domain evaluation on amateur mobile recordings from the DAMP dataset demonstrates the generalizability of the proposed framework, with 86.15% F1 achieved on clean amateur recordings and 81.40% F1 maintained under 10 dB Signal-to-Noise Ratio (SNR) noise. An average inference latency of 47.2 ms and a Real-Time Factor (RTF) of 0.0094 further indicate that DAVA can provide a computationally efficient and robust solution for personalized vocal coaching and automated vocal performance evaluation.