What question did this study set out to answer?

Investigate a hybrid CNN and Transformer model for automatic music transcription across diverse instruments.

March 27, 2026Open Access

Automatic Music Transcription Using CNN + Transformer For Zero-Shot Cross-Domain Performance

Key Points

Investigate a hybrid CNN and Transformer model for automatic music transcription across diverse instruments.
Utilized Constant-Q Transform for time-frequency representation.
Employed CNNs for local feature extraction including harmonic and percussive elements.
Applied a Transformer Encoder for modeling long-range dependencies across musical notes.
Joint-training strategy with artificial oversampling to address dataset imbalance.
Achieved high F1-scores for polyphonic pitch estimation across piano and guitar datasets.
Demonstrated significant recall improvements for note identification despite initial model limitations.
Proved efficient real-time inference with detailed computational profiling of the architecture.

Abstract

Automatic Music Transcription (AMT) remains a fundamental challenge in Music Information Retrieval (MIR), particularly when generalizing across instruments with divergent acoustic signatures. This paper presents a hybrid deep learning architecture designed to perform polyphonic pitch estimation by leveraging the complementary strengths of Convolutional Neural Networks (CNNs) and Transformers. Our methodology utilizes the Constant-Q Transform (CQT) to provide a musically aligned time-frequency representation, followed by a CNN-based acoustic frontend to extract local spectro-temporal features such as harmonic structures and percussive attacks. These features are subsequently processed by a Transformer Encoder backend, which utilizes multi-head self-attention mechanisms to model long-range temporal dependencies and polyphonic relationships across an 88-key output space.To address the performance gap often observed in cross- domain scenarios, we evaluate the model on the MAESTRO (piano) and GuitarSet (guitar) datasets. Initial results indicate that models trained exclusively on piano data suffer from a significant recall deficit when applied to string instruments due to distinct differences in excitation-resonant patterns. However they perform extremely well in the note identification process and identification of the end of any frame. To mitigate the error caused identifying the start of a f, we propose a joint- training strategy employing artificial oversampling of the smaller GuitarSet corpus to prevent dataset imbalance. Experimental results demonstrate that the proposed hybrid model achieves high F1-scores across both domains, benefiting from the CNN's local feature extraction and the Transformer's global context modeling. Furthermore, we provide a detailed computational profiling of the architecture, demonstrating its efficiency for real-time inference applications. The system is deployed as a web-based application that generates standardized sheet music and guitar tablature from raw audio input.

Read Full Paperexternally

Mark Helpful

Bookmark

Relay

View Full Paper