ZeroDayBench-22: A Reproducible Protocol and Evaluation Harness for CVE-Linked Source-Code Vulnerability Detection Benchmarks

Empirical evaluation of deep-learning source-code vulnerability detectors has been shown to be highly sensitive to how the underlying corpus is constructed and how the train/test split is defined. Recent methodological work has documented that reported accuracies collapse toward chance level once repository leakage, class-style leakage, and duplicate leakage are removed. Rebuilding evaluations under these constraints requires a corpus construction procedure that is transparent, reproducible, and separated from any single detector implementation. This paper introduces ZeroDayBench-22 as a protocol contribution rather than as a single fixed dataset. The contribution has three parts. First, a reconstruction-oriented benchmark protocol that maps CVE records to specific fix commits, extracts vulnerable and matched non-vulnerable code spans from the same source files, and enforces a repository-disjoint temporal split. Second, a metadata schema that records, per span, the identifiers necessary to reconstruct the exact input from public sources (repository, commit hash, file path, timestamp, language, CWE, label, and normalised-text hash) without redistributing third-party source code. Third, a model-agnostic evaluation harness that takes per-span score files as input and computes ROC-AUC, PR-AUC, Precision, Recall, F1, Recall at fixed FPR operating points, and Recall at top-k, with an explicit sample-size guard that refuses to report Recall@FPR values below the granularity supported by the negative-class count. A prototype metadata release of 240 candidate spans across nine programming languages is documented as an initial reference instance of the protocol. No detector accuracy is reported in this paper; the artefact is the protocol, the schema, and the harness. All code and metadata are released at a public repository under the MIT licence.