PhD Forum: Deep Learning-Based Real-Time Malware Detection with Multi-Stage Analysis

Protecting computer systems is a critical and ongoing problem, given that real-time malware detection is hard. The state-of-the-art for defense cannot keep pace with the increasing level of sophistication of malware. The industry, for instance, relies heavily on anti-virus technology for threat, which is effective for malware with known signatures, but not sustainable given the massive amount of malware samples released daily, as well as and its inefficacy in dealing with zero-day and polymorphic/metamorphic malware (practical detection rates range from 25% to 50%). Behavior-based approaches attempt to identify malware behaviors using instruction sequences, computation trace logic, and system (or API) call sequences. These solutions have been mostly based on conventional machine learning (ML) models with hand-craft features, such as K-nearest neighbor, SVM, and decision tree algorithms. However, current solutions based on ML suffer from high false-positive rates, mainly because of (i) the complexity and diversity of current software and malware, which are hard to capture during the learning phase of thealgorithms, (ii) sub-optimal feature extraction, and (iii) limited/outdated dataset. Since malware has been continuously evolving, existing protection mechanisms do not cope well with the increasedsophistication and complexity of these attacks, especially those performed by advanced persistent threats (APT), which are multi-module, stealthy, and target- focused. Furthermore, malware campaigns are not homogeneous--malware sophistication varies depending on the target, the type of service exploited as part of the attack (e.g., Internet Banking, relationship sites), the attack spreading source (e.g., phishing, drive-by downloads), and the location of the target. The accuracy of malware classification depends on gaining sufficient context information and extracting meaningful abstraction of behaviors. In problems about detecting malicious behavior based on sequence of system calls, longer sequences likely contain more information. However, classical ML- based detectors (i.e., Random Forest, Naive Bayes) often use short windows of system calls during the decision process and may not be able to extract enough features for accurate detection in a long term window. Thus, the main drawback of such approaches is to accomplish accurate detection, since it is difficult to analyze complex and longer sequences of malicious behaviors with limited window sizes, especially when malicious and benign behaviors are interposed. In contrast, Deep Learning models are capable of analyzing longer sequences of system calls and making better decisions through higher level information extraction and semantic knowledge learning. However, Deep Learning requires more computation time to estimate the probability of detection when the model needs to be retrained incrementally, a common requirement for malware detection when new variants and samples are frequently added to the training set. The trade-off is challenging: fast and not-so-accurate (classical ML methods) versus time-consuming and accurate detection (emerging Deep Learning methods). Our proposal is to leverage the best of the two worlds with Spectrum, a practical multi-stage malware- detection system operating in collaboration with the operating system (OS).