Hierarchical Concept Alignment Meets Counterfactual Invariance for Semantic-Faithful Person Retrieval

Text-Based Person Retrieval (TBPR) refers to retrieving target person images using flexible natural language descriptions. A key challenge in TBPR isalignment asymmetry, arising from the discreteness of language and the continuity of visual data. This asymmetry leads to dispersed cross-modal mappings and semantic drift. We aim for semantic-faithful person retrieval, where cross-modal alignment respects the compositional structure of language and the holistic nature of visual perception. In this paper, we propose a Hierarchical Concept Alignment Meets Counterfactual Invariance, HCA-CI, framework for alleviating alignment asymmetry and semantic drift in TBPR. We first propose Cross-Granular Concept Alignment (CGCA), which progressively refines multi-granularity concepts through concept-intensive alignment and attribute-level semantic modeling, thereby reducing semantic mismatches arising from granularity asymmetry. Then, a Counterfactual Contrastive Learning (CCL) mechanism introduces controlled concept perturbations to enforce attribute-compositional invariance across modalities, thereby mitigating semantic drift. Finally, an Intra-ID Commonality Learning (ICL) module performs prototype-guided cross-modal alignment to enhance identity-level semantic consistency while preserving instance diversity. In addition, we propose Semantic-Prioritized Rank-k (SPRank-k), a semantic-consistency evaluation perspective that prioritizes semantic/visual consistency beyond strict ID matching. Extensive experiments on the CUHK-PEDES, ICFG-PEDES, and RSTPReid benchmarks demonstrate that our method achieves competitive performance.