Designing Intelligent Multimodal Assistants for Digital Humanities: A Comparative Study of Models, Modalities, and Domains

The study addresses the problem of underutilization of the capabilities of modern Multimodal Large Language Models in the Digital Humanities. We propose an architecture for a scalable and adaptive multimodal virtual intelligent assistant based on the Retrieval-Augmented Generation approach. The architecture was tested on heterogeneous data in Russian from three subject domains: history, literature, and botany. Three modality processing strategies were used: text-only, images-only, and combined (text and images). In the retrieval experiments, full-text search, semantic text search, image search, and hybrid variants were compared. The most effective combination proved to be full-text search, semantic text search using the KaLM-Embedding model, and image search using the MetaCLIP 2 model. In the generation experiments, six open-weight Multimodal LLMs were compared using the LLM-as-Judge approach. The most performative model for the combined modality was Gemma 3, while for the text-only modality it was Qwen-3-VL Instruct. When working with images all models demonstrated relatively low performance. The study confirms the practical applicability of the proposed architecture and identifies candidate models for developing virtual intelligent assistants capable of working with complex multimodal databases in digital humanities projects.