Multimodal language models in agriculture: A tutorial and survey

The integration of artificial intelligence (AI) in agriculture is rapidly evolving, marked by increasing adoption of machine learning (ML), deep learning (DL), and the recent emergence of large language models (LLMs) and multimodal language models (MLMs). These technologies are transforming traditional agricultural practices through advanced data analysis and offering innovative solutions for smart agriculture. Although earlier methods often relied on unimodal data, primarily images, current research is shifting toward information fusion with multimodal AI systems that fuse text, imagery, and other types of data, such as agricultural knowledge graphs and videos of animal activities, for enhanced decision making. General-purpose MLMs struggle with agriculture-specific nuances due to domain gaps and limited multimodal datasets. The high computational demands of foundation models (FM) and domain-specific needs also restrict broader adoption. This study presents a tutorial on applying MLMs in agriculture, covering their main concepts. Building on this foundational knowledge, the tutorial surveys key developments in the literature and provides a comprehensive step-by-step guide for implementing and tailoring MLMs to agricultural applications. This will address the domain gaps by knowledge integration, synthetic multimodal data generation and efficient learning methods, thereby expanding their practical applications.