Research Model for Improving the Detection Accuracy of Obfuscated Malware

This article researches the problem of calibrating probabilistic predictions in machine learning models for detecting obfuscated malware. Isotonic regression was used as the research method. The research was conducted in a virtual environment using obfuscated malware samples. The analysis showed that isotonic regression, when used in neural network calibration, improves the detection accuracy of obfuscated malware using machine learning methods. The datasets used were CIC-IDS 2017, EMBER 2018 and datasets generated using malware obtained from public sources. To evaluate the calibration accuracy of the machine learning model under noisy and class-imbalanced conditions, the following evaluation metrics were used: area under the receiver operating characteristic curve, area under the precision-recall curve, brier score, log loss, and expected calibration error. The results of the research confirm the applicability of isotonic regression to imbalanced and noisy datasets typical for obfuscated malware detection tasks.