Enhancing the use of paintings as historical evidence with multimodal models

“Using images as historical evidence” is an important approach in historical research, comparing paintings with textual records to support, refine, or challenge historical interpretations. However, this evidential comparison remains manual and fragmented, requiring labor-intensive connections across dispersed visual and textual materials. To explore how advanced multimodal models could support this workflow, we conducted expert workshops to identify key challenges and workflow stages, and developed HistoriWitness, a multimodal visual analytics system that connects Chinese historical paintings with textual records from classical Chinese sources. HistoriWitness uses a generative module to propose image-text search cues, fragment-level clustering to filter visual and textual candidates, and expert corrections to recompute cross-modal similarities. Case studies, including portrait verification and user feedback, show its potential to accelerate evidence discovery, organize cross-modal evidence, and support traceable, expert-guided interpretation, offering methodological insights into the design of multimodal systems for heritage science.