AgriMAPO: A multimodal automatic prompt optimization approach for crop disease classification using large language models

Accurate identification of crop diseases is a core challenge in promoting precision agriculture and reducing yield losses. Current deep learning-based recognition methods heavily rely on large-scale annotated data, which limits their scalability across diverse crop scenarios. While large language models (LLMs) show potential for zero-shot classification, their performance on fine-grained disease recognition tasks is often inadequate due to a lack of domain-specific expertise and cross-modal alignment capabilities. To address these issues, this paper proposes a multimodal, automatic prompt-optimization-based method for crop disease classification (AgriMAPO) that enhances the discriminative power of LLMs through automated prompt engineering, enabling high-accuracy, few-shot disease recognition. The AgriMAPO framework integrates three core components: a deep learning-based, dual-stage screening strategy to construct a high-quality few-shot image-label example library; an Automatic Prompt Optimization (APO) agent that uses an evolutionary algorithm to generate a library of fine-grained disease symptom descriptions; and a GPT-4o-based classification agent guided by chain-of-thought (CoT) reasoning. Experimental results on an eight-class soybean disease dataset demonstrate that the proposed method achieves an overall classification accuracy of 90.3%, significantly outperforming the native GPT-4o model, which scored 40.3%. Further analysis showed that the optimized multimodal prompts exhibit excellent generalization, transferring effectively across different datasets and models. This study provides a training-free, highly generalizable solution for few-shot, fine-grained disease recognition in agriculture, significantly reducing reliance on annotated data and expert knowledge. • AgriMAPO: a multimodal auto prompt optimizer for training-free crop disease classification. • Dual-stage strategy screens high-quality few-shot examples with diversity. • APO agent auto-generates fine-grained symptom descriptions via evolution. • Achieves 90.3% overall accuracy on soybean diseases, 50% higher than GPT-4o. • Optimized prompts show strong cross-dataset and cross-model transfer.