Research Progress on Multimodal Named Entity Recognition with Multi-Task Learning

Multimodal Named Entity Recognition (MNER) leverages heterogeneous data sources such as text and images to improve the accuracy of named entity recognition. However, MNER performance remains limited due to the semantic gap between visual and textual representations and the bias introduced by ignoring visual context (or incorporating it inappropriately). To address these issues, multi-task learning is adopted to assist MNER from two perspectives: feature extraction and processing, and multimodal fusion, by jointly learning multiple related auxiliary tasks. This paper tracks recent advances in MNER and conducts a comparative analysis of representative mainstream approaches. According to the auxiliary tasks used in multi-task learning, existing methods are categorized into three types: visual optimization network–assisted methods, local matching–assisted methods, and text-assisted multi-task methods. The first two categories enhance model performance by improving the quality of visual features and modeling image–text similarity, respectively, whereas the third category reduces visual bias through auxiliary tasks based on textual signals. Moreover, twelve representative methods, including MFD and SMRG, are evaluated on Twitter-2015 and Twitter-2017. Experimental results demonstrate that multi-task learning effectively alleviates the semantic discrepancy between image and text modalities and mitigates visual bias, improving the accuracy and robustness of MNER systems.