Improving vulnerability-scores: Leakage-Free Splits and Consistent CVSS 4.0 Scoring

This technical report describes a new version of CIRCL/vulnerability-scores, the dataset used to train the VLAI vulnerability severity classifiers (Bonhomme & Dulaunoy, 2025). The new version gives the models a stricter, more reliable evaluation and more consistent severity labels. The changes follow an external review by Christian Bucher, which found two problems in the previous dataset revision (34e6a14): - Twin rows across the train/test split. 58.45% of the labelled test rows had a train row with the same description, mostly a CVE record and the GitHub advisory for the same CVE. Counting shared identifiers and aliases too, the figure is 74.6%. The published RoBERTa classifier scores 81.17% on that test split, but 73.02% on the test rows it had never seen in any form. The 73.02% is the more representative figure for new vulnerabilities.- Two readings of CVSS 4.0. GitHub, PySec and CSAF advisories were scored with the threat metric Exploit Maturity applied (CVSS-BT), while CVE records kept the CNA's published score. As a result, the same vulnerability could get two different severity labels. The new dataset is split by vulnerability group: rows sharing a description, an identifier or an alias always end up in the same split. Every CVSS 4.0 score is read as CVSS-B (base metrics only), a validation split is used for checkpoint selection, and repeated documents and rejected CVEs are removed. Splits are reproducible across rebuilds. An automatic leakage check confirms that no group spans train, validation and test, and it now runs after every rebuild. The report covers the measurements, the design choices, the rebuilt dataset and the next steps for the VLAI models. All changes are implemented in VulnTrain, and the dataset is published on Hugging Face.