What question did this study set out to answer?

The aim is to improve dense prediction tasks using a novel hybrid architecture, ViT-Hook, for vision transformers.

April 30, 2026Pattern Recognition0 citationsOpen Access

Vision transformer Hook for dense predictions

SMSiyuan MeiFriedrich-Alexander-Universität Erlangen-Nürnberg MTMareike ThiesFriedrich-Alexander-Universität Erlangen-Nürnberg YXYan XiaUniversity of Science and Technology of China

Key Points

The aim is to improve dense prediction tasks using a novel hybrid architecture, ViT-Hook, for vision transformers.
Introduced a transformer-decoder-inspired hook module for integrating CNN features with ViT features.
Tested ViT-Hook on various dense prediction tasks including semantic segmentation and object detection.
Utilized a partial fine-tuning strategy for performance enhancement while reducing computational demands.
Achieved 59.7 mIoU on ADE20K val, a 4.7 increase compared to previous methods.
Reached 55.0 box AP and 48.5 mask AP on COCO val2017, improvements of 3.6 and 3.3 respectively.
ViT-Hook surpasses state-of-the-art results across multiple benchmarks.

Abstract

Pre-trained vision transformers (ViTs) have demonstrated remarkable capability in learning semantically rich image representations. However, their underlying plain architectures yield low-resolution feature maps, lacking essential fine-grained spatial details required for dense prediction tasks. To better transfer the learned visual features, we present ViT-Hook, a novel hybrid backbone compatible with plain ViTs that effectively bridges the gap between global semantic understanding and local spatial encodings. Specifically, our method aims to broaden the scope and impact of ViT from the following perspectives: (1) We propose a simple transformer-decoder-inspired hook module that receives hierarchical CNN features as spatial queries and interacts with expressive ViT features from large-scale pre-training, therefore instantiating general-purpose representations into task-suited ones. (2) ViT-Hook is a plug-and-play solution for powerful vision foundation models, such as DINOv2 and RADIO. In this case, we find that only partially fine-tuning several intermediate ViT layers can outperform previous full fine-tuning methods, while substantially reducing compute and memory burdens with most parameters frozen. (3) We evaluate ViT-Hook with various pre-trained sources on multiple dense prediction tasks, including semantic segmentation, instance segmentation, and object detection. Notably, tested on the unified UperNet and Mask R-CNN frameworks, our ViT-Hook surpasses state-of-the-art by a large margin, achieving 59.7 (+4.7) mIoU on ADE20K val, 55.0 (+3.6) box AP and 48.5 (+3.3) mask AP on COCO val2017. • We propose ViT-Hook, a hybrid backbone that effectively enhances Vision Transformer performance on various dense prediction tasks. • The proposed spatial query and hook modules are lightweight yet powerful, achieving competitive results compared to SoTA on widely used benchmarks. • We introduce a novel partial fine-tuning strategy, which outperforms full fine-tuning while using only a fraction of compute and memory. • We validate the generalizability of ViT-Hook on multiple types of upstream pre-training methods, including the most recent vision foundation models.

AI से पूछें

Bookmark

View Full Paper

Cite This Study

Mei et al. (Wed,) studied this question.

synapsesocial.com/papers/69f2f1471e5f7920c6386ff3 https://doi.org/https://doi.org/10.1016/j.patcog.2026.113818

AI से पूछें

Bookmark

View Full Paper