TinyML-Based Speech Emotion Recognition on Microcontrollers Using Quantized CNNs and Mel-Filterbank Energy Features

This paper presents a real-time, resource-efficient Speech Emotion Recognition (SER) system trained on a modified version of the Toronto Emotional Speech Set (TESS) that includes both male and female voice samples for five emotional labels—Angry, Disgust, Fear, Happy, and Neutral—plus a custom Noise class added to improve robustness. The final model was deployed on the M5Stack C Plus, an ESP32-based microcontroller, using TinyML techniques, performing fully offline inference rather than relying on cloud-based processing. Edge Impulse was used to design, train, and optimize a quantized 2D Convolutional Neural Network, leveraging Mel-filterbank energy features extracted from 16 kHz audio captured via a PDM digital microphone interfaced over I²S. The model classifies six emotional states with a validation accuracy of 99.5% and a total inference latency of approximately 571–572 ms, including both DSP and classification phases. The final quantized int8 model occupies less than 2 MB of flash and consumes under 50 KB of RAM. This work demonstrates that accurate, low-latency emotion recognition is feasible on ultra-low-power microcontrollers, making it suitable for privacy-preserving, always-on, embedded human-computer interaction systems.