Pre-Masking, Preprocessing Completion, and Anomaly Sensitivity in HDFS Log Classification with TF–IDF and LinearSVC

Log preprocessing determines both the representation supplied to a detector and whether the intended observations can be evaluated. We compared Drain3 without external regular-expression pre-masking (E), pre-masked Drain3 (D), and a regex bag-of-words representation (R) on HDFS-v1, comprising 11,175,629 log lines and 575,061 labeled block identifiers. Drain3 processing for E and D preceded a fixed 80/20 stratified block split; only D completed the full unlabeled corpus. TF–IDF and LinearSVC were fitted only on the training blocks. E was terminated after approximately 16 h, with 200,109 templates and 42,879 of 115,013 test block identifiers reached, before complete block-document coverage was established. D and R met the operational block-document coverage criterion and produced predictions for all 115,013 test blocks. D yielded 14 false negatives and 81 false positives, whereas R yielded 84 false negatives and 81 false positives. The anomaly recall was 99.584% for D and 97.506% for R. Among 3368 anomalous test blocks, D uniquely detected 73 and R uniquely detected 3, with 11 shared misses (exact two-sided p = 1.94 × 10−18). In this fixed-split setting, external pre-masking enabled the Drain3 branch to complete corpus processing. Under the study’s transductive representation setting, D achieved higher fixed-threshold anomaly sensitivity than R on the common test partition, with the same marginal false-positive count.