Match me if you can: aligning historical posters and newspaper articles via multimodal deep learning

Multimodal deep learning (MDL) has potential for digital history (DH) research, from simple tasks such as image or text classification, to more sophisticated ones such as information retrieval (IR). We propose a novel IR method based on vision language models (VLMs), a class of MDL models, to extract embeddings from images and text to find connections among unrelated image and text datasets that are challenging to find using conventional approaches such as keyword search. We evaluate our method with a proof-of-concept case study on debates surrounding the civil usage of nuclear power in Europe, based on poster images from the Landelijk Kernenergie Archief website and German newspaper articles found in the Impresso corpus, where both datasets are assumed to share some common themes. We find that poster images and newspaper articles can be automatically matched from independent sources based on representations extracted by pre-trained VLMs. We further find that VLMs achieve better performance when translating newspaper articles to English or summarizing them in English. Ultimately, our work bridges the gap of cross-modal and cross-lingual IR for DH, showing real-world applicability of MDL and opening new avenues in DH research. Our code and data are available at https://github.com/StevenXuf/MatchMeIfYouCan.