X-MAG-COS: Compact Evidence Messaging for Source-Partitioned Open-Set Intrusion Detection in 5G–IoT Networks

Open-set intrusion detection must distinguish known traffic classes, reject previously unobserved attacks, and respect the available communication between a traffic monitor and a coordinator. We study X-MAG-COS, a compact evidence-message architecture that separates known-class prediction from rejection using predictive uncertainty, local anomaly evidence, and a message-prototype residual. Beyond structural properties, we derive bounds that propagate packet-scalar quantization through a normalized one-versus-rest classifier, possible predicted-class prototype changes, the composite score, and the calibrated rejection decision. The primary evaluation uses eight leave-one-attack-family-out tasks and five seeds on 5G-NIDD with training-only feature screening, packet-decoded inference, and disjoint normalization and calibration subsets. The ownership map contains two highly imbalanced metadata partitions, not eight active physical monitors. A matched extension evaluates six algorithmic replacements and four controls at identical realized false-positive counts within each task. At the all-known FPRs targeted at 0.1%,1%, and 5%, the 16-byte reference attains mean unknown recalls of 0.0934, 0.5484, and 0.7242; centralized random forest attains 0.6781, 0.7557, and 0.7942. Replacing the local forest with extra trees raises near-5% recall to 0.8062 but lowers stringent-level recall, while mean fusion has the opposite operating-point tradeoff. These substitutions do not establish a uniformly superior configuration. Fixed-model precision diagnostics satisfy the derived score and decision bounds within a stated 10−8 numerical tolerance without constituting a machine-verified proof. The 12-byte detection-only record remains a competitive alternative; the 16-byte record additionally contains an optional attribution proxy whose analyst-facing value is unmeasured. UDPFlood, the largest attack family and 37.6% of the corpus, retains below-chance mean ranking with AUROC 0.4818 ± 0.0190. Controlled peer-context experiments and separately retrained CICIoT2023 results further delimit the approach. The findings establish a characterized communication–detection tradeoff, not universal detection superiority, validated explainability, or deployment readiness.