Causal-Patched Attention Network: Mitigating Contextual Bias and False Associations in Multi-Label Image Classification

Multi-label image classification (MLIC) is vulnerable to contextual bias, where models may exploit spurious label–context associations rather than object evidence, leading to degraded generalization under distribution shifts. To address this issue, we propose CPAN, a causal-inspired framework that integrates label-specific feature decoupling, prototype-based mediator modeling, patch-level evidence aggregation, and adaptive fusion. Specifically, CPAN uses a Transformer decoder to extract label-specific representations from the whole image and local patches. We introduce a prototype dictionary as a surrogate mediator space to encourage the model to rely on object-relevant intermediate patterns rather than context-sensitive shortcuts. We further aggregate patch-level predictions to enhance direct object evidence and fuse them with whole-image predictions through a learnable gate. Experiments on two benchmark datasets show that CPAN consistently improves both recognition accuracy and robustness. On MS-COCO, CPAN achieves 85.26 mAP, 80.67 CF1, and 82.52 OF1; on NUS-WIDE, it reaches 66.11 mAP, 64.42 CF1, and 75.95 OF1. Under context-shifted evaluation on MS-COCO, CPAN further obtains 80.93 mAP, 75.84 CF1, and 77.87 OF1, indicating stronger robustness to contextual bias. These results show that CPAN learns more object-centered representations and reduces reliance on spurious contextual correlations.