Bootstrapping vision–language transformer for monocular 3D visual grounding

In the task of 3D visual grounding using monocular RGB images, it is a challenging problem to perceive visual features and accurately predict the localization of 3D objects based on given geometric and appearances descriptions. Traditional text‐guided attention‐based methods have achieved better results than baselines, but it is argued that there is still potential for improvement in the area of multi‐modal fusion. Thus, Mono3DVG‐TRv2, an end‐to‐end transformer‐based architecture that employs a visual‐text multi‐modal encoder for the alignment and fusion of multi‐modal features, incorporating an enhanced transformer module proven in 2D detection, is introduced. The depth features predicted by the multi‐modal features and the visual‐text features are associated with the learnable queries in the decoder, facilitating more efficient and effective acquisition of geometric information in intricate scenes. Following a comprehensive comparison and ablation study on the Mono3DRefer dataset, this method achieves state‐of‐the‐art performance, markedly surpassing the prior approach. The code will be released at https://github.com/Jade-Ray/Mono3DVGv2 .