Capsule Activation Separability-Driven domain-adversarial attention network for fine-grained open-set domain generalization fault diagnosis with intrinsic decision

Achieving fine-grained (FG) open-set domain generalization (OSDG) fault diagnosis for mechanical systems stands as a pivotal challenge for the practical implementation of intelligent maintenance. However, existing studies largely provide coarse-grained diagnostics, and their open-set criteria heavily rely on external modules, limiting model efficiency and reliability. To address these challenges, this paper proposes a novel deep network termed Capsule Activation Separability-Driven Domain-Adversarial Attention Network (CAS-DAANet). The core innovation of the CAS-DAANet lies in developing the Capsule Activation Separability (CAS) as an intrinsic criterion. The CAS ingeniously quantifies the model's decision certainty for a given input as its core metric, thereby unifying and intrinsically assessing open-space risk, classification confidence risk, and model hallucination risk. This enables open-set recognition that requires no external modules and offers inherent interpretability. Besides, to further enhance the model's ability to learn hierarchical fault features, the CAS-DAANet integrates self-attention mechanisms and domain adversarial training to learn robust, domain-invariant fault representations. By optimizing the designed CAS loss and incorporating an adaptive threshold calibration strategy, CAS-DAANet can accurately diagnose known faults within unseen target domains and effectively identify unknown faults. Extensive experimental validation on the HUST and CWRU datasets demonstrates that our proposed CAS-DAANet achieves average H-Scores of 86.28% and 90.05% respectively, affirming the effectiveness and superior generalization capability of the proposed method.