How to Get Enriched Metadata? A Multi‐modal Model Fusion Strategy for Automatic Metadata Enhancement in GLAM Art Collections

Cultural heritage resource metadata is the foundation and precious asset for GLAM institutions to provide knowledge services which enables users to efficiently search relevant collection information. However, current GLAM institutions (e.g. museums), face significant challenges to gather comprehensive high‐quality collection metadata. Motivated by the complementary advantages of multi‐modal large language models (MLLMs) and pretrained small ones (MPSMs), we proposed “AGGM”, a model fusion approach for automatic metadata enrichment including two key components: MPSM‐based module for demonstration detection and MLLM‐based module for prediction calibration. The merit of AGGM is to fully leverage the powerful semantic understanding capabilities to generate accurate results based on MLLMs in limited computation cost, and also exert the strength of MPSM domain‐specific knowledge to obtain informative demonstrations. The experimental results showed that AGGM outperformed baseline models in two regular metadata generation tasks, demonstrating enormous potential of this proposed model fusion approach in automatic generation of GLAM metadata.