Hybrid Causal Action Reasoning for Temporal Action Detection

Human action understanding aims to detect action instances and classify them in videos. Existing methods capture contextual dependencies through temporal feature aggregation or attention-based interactions. However, these methods primarily rely on visual features to establish relationships, making them susceptible to interference from background content or irrelevant action frames. To address this limitation, we propose Hybrid Causal Action Reasoning (HCAR), a framework for modeling causal correlations between action proposals. Specifically, we design a Hybrid Causality-Aware Attention (HCAA) to refine the semantic representations of proposal classes. To further characterize relations between adjacent proposals, an ordered relation branch evaluates the refined representations in temporal order, while a hyperbolic branch measures hierarchical semantic relatedness between action categories in the Poincaré ball. The outputs of these branches are combined to obtain causal correlation scores, which are subsequently used to refine action proposals. Extensive experiments on THUMOS14 and ActivityNet-1.3 demonstrate the effectiveness of our HCAR.