Enhancing malware detection reliability in non-executable files using confidence score prediction

In the rapidly evolving landscape of cybersecurity, malware attacks targeting widely used non-executable formats, namely Microsoft Office and PDF files, have become a prevalent threat. These files, which encompass a broad spectrum of data types, including text, images, executable codes, and media formats, are classified as complex files. Their flexibility in accepting different data types makes them of significant interest to attackers. Existing learning-based malware detection approaches suffer from a high false-positive rate making their results unreliable. Two complementary solutions are proposed to address the problems of current malware detections. The first methodology involves the development of binary classification models, utilizing a dataset of complex file formats collected from various sources, with an extended feature set extracted to achieve high accuracy rates. The second methodology employs the development of regression models to ascribe a confidence score to each sample qualifying the reliability extent of the binary classifier results. A weighted reliability score is assigned to various antivirus programs when computing their reliability for labeling malicious samples with a confidence score. Our findings demonstrate a significant enhancement compared to existing state-of-the-art malware detection models, with improvements of approximately 2% for PDF files and 1.9% for MS Office. Specifically, the gradient-boosting model achieved 99.3% accuracy for PDF malware detection, while a random forest model achieved 99.4% accuracy for MS Office files. Moreover, the random forest regressor achieves a mean squared error (MSE) of 0.0006 for PDF and 0.003 for Microsoft Office files in predicting confidence score. Our models enable malware analysts to make informed decisions based on such accurate outputs.