KTU-MEDAFE: A Newly Developed Multimodal Dataset for Emotion Recognition Using EEG–Speech Decision-Level Fusion

Q: What is the clinical evidence from this study?

Population: Emotion recognition (n=40). Intervention: Multimodal decision-level fusion of EEG and speech vs. Unimodal EEG and speech models. Primary outcome: Average classification performance across binary emotion pairs.

Q: What does this research mean for the field?

The newly developed KTU-MEDAFE dataset provides a synchronized multimodal resource for emotion recognition, demonstrating that EEG-speech decision-level fusion outperforms unimodal approaches while exhibiting substantial subject-dependent variability. Novelty: ClaimNovelty.METHODOLOGICAL. Consensus alignment: ConsensusAlignment.NEUTRAL.

Bahar Hatipoğlu Yılmaz; Betul Mumcu; Busra Ozkellekci

doi:10.3390/s26175608

September 3, 2026SensorsOpen Access

KTU-MEDAFE: A Newly Developed Multimodal Dataset for Emotion Recognition Using EEG–Speech Decision-Level Fusion

View Full Paper

Ask AI

Bookmark

Share

Key result

Multimodal EEG and speech fusion improves emotion classification performance over unimodal models.

n=40

Figures

Why the study?

Achieving reliable emotion recognition for natural and effective human-computer interaction remains a central challenge in affective computing.

Key Points

To introduce the KTU-MEDAFE multimodal dataset and evaluate whether decision-level fusion of EEG and speech signals improves emotion classification performance.
Acquired synchronized EEG, Turkish speech recordings, and facial video across two separate sessions from 40 participants under controlled emotion elicitation conditions.
Converted EEG and speech signals meeting quality criteria into 2D image representations using the Angle–Amplitude Graph (AAG) approach.
Evaluated binary emotion pair classification using transfer learning with ResNet-50 and GoogLeNet architectures across multiple decision-level fusion strategies.
Multimodal decision-level fusion outperformed unimodal EEG and speech baseline models in average classification accuracy across evaluated binary emotion pairs.
Classification performance exhibited substantial subject-dependent variability across participants, fusion methods, and model architectures.

Structured PICO

P

Population

40 participants providing synchronized EEG signals, speech recordings, and facial videos under controlled emotional elicitation conditions.

I

Intervention

EEG and speech integration using Angle-Amplitude Graph (AAG) method and transfer learning with ResNet-50 and GoogLeNet architectures with decision-level fusion

C

Comparator

Unimodal EEG and speech models

O

Outcome

Classification performance across evaluated binary emotion pairs

Combining EEG and speech for multimodal emotion recognition improves classification performance compared to unimodal approaches.

Limitations

Substantial subject-dependent variability in classification performance

Implication

Multimodal EEG-speech fusion may improve emotion classification; leaves open clinical translation pending validation across diverse cohorts.

Abstract

One of the central challenges in affective computing is achieving reliable emotion recognition for natural and effective human–computer interaction. In this study, we introduce KTU-MEDAFE (Karadeniz Technical University Multimodal Emotion Dataset using Audio, Facial Images, and EEG), a newly developed multimodal dataset containing synchronized EEG signals, speech recordings, and facial videos collected from 40 participants under controlled emotional elicitation conditions. The dataset includes Turkish emotional speech and two recording sessions conducted on separate days, providing a language-specific resource that supports both participant-dependent baseline evaluation and future session-separated analysis. Although KTU-MEDAFE comprises three modalities, the present study focuses on EEG and speech integration. EEG and speech recordings meeting signal quality criteria were transformed into image representations using the Angle–Amplitude Graph (AAG) method and classified using transfer learning with ResNet-50 and GoogLeNet architectures. To exploit complementary information across modalities, multiple decision-level fusion strategies were evaluated. Experimental findings show that multimodal fusion provides higher average classification performance than unimodal EEG and speech models across the evaluated binary emotion pairs, with performance varying according to subject, fusion strategy, and model architecture. Overall, the results support the potential benefit of combining EEG and speech for multimodal emotion recognition while highlighting substantial subject-dependent variability in classification performance.

Discussion

No takes yet. Share an insight, caveat, or question.

Cite This Study

Yılmaz et al. (2026) studied Emotion recognition (n=40). Multimodal decision-level fusion of EEG and speech vs. Unimodal EEG and speech models was evaluated on Average classification performance across binary emotion pairs. Multimodal decision-level fusion of EEG and speech provided higher average emotion classification performance than unimodal models.

synapsesocial.com/papers/6a9bd4046b95aff0620eb584 https://doi.org/10.3390/s26175608

View Full Paper

Ask AI

Bookmark

Share

KTU-MEDAFE: A Newly Developed Multimodal Dataset for Emotion Recognition Using EEG–Speech Decision-Level Fusion

Figures

Key Points

Structured PICO

Limitations

Implication

Abstract

Discussion

Cite This Study

KTU-MEDAFE: A Newly Developed Multimodal Dataset for Emotion Recognition Using EEG–Speech Decision-Level Fusion

Figures

Key Points

Structured PICO

Limitations

Implication

Abstract

Discussion

Cite This Study

Keep reading

More related papers

More from Sensors

Keep reading

More related papers

More from Sensors