Evaluating Diagnostic Evidence in Multimodal Network Fault Classification

Machine learning using multimodal telemetry is now standard in network fault diagnosis, but diagnostic models are compared almost exclusively by accuracy. On a virtual SONiC testbed with five injected fault types, five classifiers from three model families reach a similar macro-F1. We therefore grade the evidence behind each verdict with an attribution protocol, an answer key computed from the injected interventions and validated by two separability gates, and three metrics: onset localization, physical consistency, and deletion faithfulness. The grades separate models that accuracy cannot; on the switching loop, strict physical consistency ranges from 0.960 to 0.000 across the five models with non-overlapping intervals. No model’s link-failure verdict rests on the traffic collapse that defines the fault. Within a collection regime, the evidence is a small traffic dip or the log block. Each model also depends on baseline channels set by the collection conditions, which accounts for part of the normal–link-failure confusion but not for the accuracy saturation. When applied unchanged to events collected later under changed conditions, all five models transferred on these classes, with keys validated at the input resolution whenever the defining signal reached the input and failed on the class whose key is only validated below it. The most accurate model produced false confirmations on nearly half of the normal windows. The procedure reports the classes without a validated key.