Selective Depth Attention Networks for Adaptive Multi-scale Feature Representation

Existing multi-scale methods lead to a risk of just increasing the receptive field sizes while neglecting small receptive fields. Thus, it is a challenging problem to effectively construct adaptive neural networks for recognizing various spatial-scale objects. To tackle this issue, we first introduce a new attention dimension, i.e., depth, in addition to existing attentions such as channel-attention, spatial-attention, branch-attention, and self-attention. We present a novel selective depth attention network to treat multi-scale objects symmetrically in various vision tasks. Specifically, the blocks within each stage of neural networks, including CNNs, e.g., ResNet, SENet, and Res2Net, and ViTs, e.g., PVTv2, output the hierarchical feature maps with the same resolution but different receptive field sizes. Based on this structural property, we design a depth-wise building module, namely an SDA module, including a trunk branch and an SE-like attention branch. The block outputs of the trunk branch are fused to guide their depth attention allocation through the attention branch globally. According to the proposed attention mechanism, we dynamically select different depth features, which contributes to adaptively adjusting the receptive field sizes for the variable-sized input objects. Moreover, our method is orthogonal to multi-scale networks and attention networks, so-called SDA- x Net. Extensive experiments demonstrate that the proposed SDA method significantly improves the original performance as a lightweight and efficient plug-in on numerous computer vision tasks, e.g., image classification, object detection, and instance segmentation. Our source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/QingbeiGuo/SDA-xNet.git</uri> .