Hierarchical Region-level DecouplingKnowledge Distillation for Semantic Segmentation

Abstract Region-level distillation can facilitate the transfer of salient region information for each channel, but computing Kullback-Leibler (KL) divergence for the entire soft probability map can significantly limit the function of Non-target Class Knowledge Distillation (NCKD). To address the aforementioned issue, we propose the Region-level Decoupling Knowledge Distillation (RDKD), and the simple and efficient RDKD implicitly decouples region-level distillation into Target Region Knowledge Distillation (TRKD) and Non-target Region Knowledge Distillation (NRKD), which ensures the effective transfer of the region-level dark knowledge present in both TRKD and NRKD. In order to progressively integrate global information, we further propose the Hierarchical Region-level Decoupling Knowledge Distillation (HRDKD), which gradually aggregates global information through a simple average pooling operation, thereby facilitating the distillation of multi-scale semantic information. Extensive experiments are conducted on six benchmark datasets: Cityscapes, Pascal VOC, ADE20k and COCO Stuff164k for natural images, and Synapse and FLARE22 for medical images. The experimental and visualization results demonstrate that our proposed distillation methods achieve state-of-the-art (SOTA) performance without the need for introducing auxiliary modules in the corresponding semantic segmentation tasks.