Diagnosing Pseudo-Robustness: A Boundary-Conditioned Audit Methodology for Perturbation Evaluation in NLP

Robustness evaluation in NLP typically treats prediction invariance as the target, but invariance alone cannot distinguish valid robustness from pseudo-robustness—the failure to respond when the task boundary has been crossed. We propose the Boundary Conditioned Audit Profile (the BCAP methodology) methodology, a validation-gated diagnostic methodology that interprets prediction behavior relative to task boundaries. Rather than asking whether a model’s prediction changed under perturbation, BCAP asks whether it should have changed, given the perturbation’s normative relation to the task-relevant distinction. Perturbations are classified into three boundary relations: boundary-preserving (Ppre), which should leave the label unchanged; boundary-crossing (Pbrk), which should change it; and boundary-irrelevant (Pirr), which vary aspects irrelevant to the task. The same observable behavior—retention or change—maps to opposite diagnostic states depending on this relation: retention can indicate valid robustness, pseudo-robustness, or valid stability; change can indicate fragility or boundary sensitivity. We evaluate four boundary regimes—sentiment polarity (SST-2), topic category (AG News), inferential relation (MNLI), and semantic equivalence (MRPC)—across four transformer checkpoints (BERT, RoBERTa, DistilBERT, ALBERT), comprising 11,122 perturbation pairs. Human validation with three annotators on 600 stratified pairs yields perfect intended-effect agreement (α = 1.000) and 94.5% full agreement for Pbrk relation and task validity, with disagreements concentrated in Ppre task-validity judgments. Empirically, boundary audit profiles cluster by boundary regime rather than by model slot (Boundary Structuring Index = 42.3), yielding a three-tier sensi tivity gradient: sentiment is boundary-sensitive (BSR ≈ 0.63 across all checkpoints), NLI is partially and label-asymmetrically sensitive (BSR ≈ 0.20), and topic/MRPC exhibit severe pseudo-robust collapse (BSR ≈ 0.05–0.07, PRR > 0.91). A complementary response-only insufficiency analysis confirms the invariance para dox: on the invariance-only subset (37,736 rows where predictions were unchanged), boundary relation perfectly separates the opposite causal meanings of identical observ able invariance (balanced accuracy = 1.000), while boundary-free classifiers using only response traces remain below 0.56. The central implication is that aggregate perturba tion metrics underdetermine diagnosis; robustness should be evaluated as boundary relative appropriate invariance and responsiveness, and benchmark designers should specify task boundaries and externally validate intended perturbation-boundary rela tions before interpreting perturbation outcomes.