Distance-aware multilayer aggregation transformer for image captioning

Recently, the Transformer based on grid features has achieved great success in image captioning, but it still has some problems: the flattening operation of Transformer will destroy the positional information among visual objects, and only the output of the last layer encoder is sent to the decoder, which will lose low-level semantic information. To solve the above problems, we first introduce Distance-aware self-attention (DA), which considers the original geometric distance between visual objects in a two-dimensional image during the self-attention modeling process, and integrates distance information into attention calculation through a mapping function, better capturing the relational information among visual objects. Second, we propose the Multilayer Aggregation (MA) module, which aggregates the output of the encoder and establishes a weighted residual connection as the final output, sent to the decoder separately. It aggregates information from different encoder layers to achieve cross-layer semantic complementarity, features with rich semantics can be explored simultaneously from both low-level and high-level coding layers. To verify the validity of our proposed two designs, we applied them to a standard Transformer and conducted extensive experiments on MS-COCO, a benchmark dataset for image captioning. The experimental results demonstrate the effectiveness of our proposed Distance-aware Multilayer Aggregation Transformer (DMAT) model.