Leverage images in named entity recognition: When and how

Multimodal Named Entity Recognition aims to identify named entities such as persons, locations, and organizations from text by considering both text and image. However, existing research overly emphasizes the importance of image, neglecting the fact that the core of the task is still text. Due to the inherent differences between image and text, introducing image may inevitably introduce noise. Therefore, we advocate that image should only be introduced to assist recognition when text is insufficient. Consequently, we propose an Adaptive Cross-modal Re-matching and Core Semantic Enhancement Framework (ACRSE). This framework evaluates the sufficiency of textual information. If text is sufficient, image is not introduced; conversely, image is incorporated, by leveraging topic relevance in social media, semantically aligned images are retrieved to provide effective visual support. Our framework maps image into keywords and extracts its core semantics to enhance cross-modal semantic consistency. Specifically, the framework evaluates a textual sufficiency score by considering contextual information, semantic associations, etc., to determine whether image information is needed; subsequently, the framework retrieves and matches images, generates descriptive texts and extracts keywords based on textual structure, relevance, and both local and global semantic relationships. Extensive experiments demonstrate that our model achieves state-of-the-art performance on benchmark datasets.