Integrating Acoustic and Image Data Features for Melon Ripeness Classification Using Convolutional Neural Network

This study evaluates three classification scenarios: image-based only, acoustic-based using Mel Frequency Cepstral Coefficients (MFCC), and a combined multimodal CNN architecture integrating both modalities. The experiments are conducted on a relatively small dataset comprising only 230 samples. To mitigate the risk of overfitting arising from the limited dataset size, data augmentation is applied to both image and audio data, with audio augmentation performed before the construction of the MFCC spectrogram. Experimental results demonstrate that the multimodal CNN with data augmentation achieves the best performance, with precision, recall, and F1-score, respectively, 0.95, 0.94, and 0.94. These results indicate that augmenting both image and audio data effectively enhances data diversity and model robustness, significantly improving classification performance. The findings confirm that combining complementary feature representations from multiple modalities with proper augmentation strategies substantially improves audio-visual classification tasks.