Model Comparisons in Behavioral Malware Detection Are Noise-Dominated: A Tuning-Parity Study and Corpus Audit

Architecture comparisons in behavioral malware detection are usually decided by margins of one or two points, and this study asks whether those comparisons can resolve differences that small. On a corpus of 127,930 DRAKVUF system-call traces evaluated with folds that are simultaneously family-disjoint and duplicate-disjoint, every model — including the order-free frequency baseline that neural architectures are normally ranked against — is given the same twenty-trial tuning budget and refit across multiple training seeds. Under that parity the frequency baseline moves from 0.964 to 0.984 area under the ROC curve and from 0.871 to 0.881 average precision and takes the lead on both measures, reversing the verdict the conventional protocol produces; neural tuning spreads and seed variability exceed the baseline's by twenty to a hundred times, which dwarfs the margins routinely reported as architectural progress. The work is paired with an audit of the corpus itself, which finds that nearly half of samples share a duplicate feature representation and that enforcing family-disjoint evaluation costs the published baseline seven points of ranking performance.