Multimodal Approach for Fruit Disease Identification Using Vision Language Models and Image Embeddings

Precise detection of fruit diseases is essential for preventing quality degradation, minimizing yield losses, and enabling timely intervention in orchard systems. Although Convolutional Neural Network (CNN) and Transformer-based models demonstrate strong performance on curated datasets, their reliance on large annotated datasets, fine-tuning, and careful model optimization limits their performance on unseen fruit types, disease categories, and heterogeneous real-field imagery. This work proposes a non-parametric multimodal fruit disease diagnosis framework that integrates visual and semantic representations derived from foundation models. Two vision–language models, Qwen-VL and LFM-VL, are used to generate natural-language descriptions of disease symptoms, while DINO-v2, a self-supervised vision encoder, and a Vision Transformer (ViT) are employed to extract robust visual embeddings from fruit images. These complementary features are integrated, and classification is performed using a centroid–based approach, enabling disease recognition without requiring parameter updates or supervised training. For orchard-scale imagery, an open-vocabulary zero-shot detector (LLMDet) is applied to localize fruit instances, after which the detected fruits are classified and the predicted disease labels are reassigned to the original images for accurate object-level diagnosis. To support systematic evaluation, a diverse collection of fruit images is considered from multiple public datasets and challenging real-field orchard environments. Experimental results reveal that the proposed multimodal approach achieves an average accuracy of 92% across eight datasets with six fruits types, comprising 9,520 images characterized by substantial variations in scale, symptom appearance, and background conditions. The proposed approach reduces reliance on extensive manual annotation and complex training pipelines while enabling robust and scalable fruit disease diagnosis in real-world agricultural environments.