Identifying Malware Family with String Matching Algorithms Based on API Calls and Entire Strings

Malware is a growing concern for governments, organizations, and individuals, as it can corrupt or steal confidential or sensitive data. Although security software constantly identifies and blocks malware, adversaries can produce slightly altered variants within the same malware family by making minor alterations to the original. Although this makes it challenging for the system to detect, it also provides an opportunity to detect malware from the same family early since many of its strings remain similar. In this study, we apply three similarity algorithms to nine malware families to identify whether a malware string is from the same family. First, we extract the strings of each executable and divide these into API calls and entire strings. Next, we use the Jaccard Coefficient, Longest Common Subsequence (LCS), and N-gram algorithms to determine the optimal threshold to determine whether two malware strings belong in the same malware family or not. Finally, we determine that threshold values 26.84% for LCS, 28.02% for Jaccard, and 54.16% for N-gram are optimal for detecting similar strings within all malware families, and we conduct both within-family and cross-family analyses to substantiate our results. Our analysis shows that the N-gram algorithm achieves high accuracy rates of 91.16% for entire strings and 93.11% for API calls. Similarly, the Jaccard algorithm also demonstrates strong accuracy, with rates of 91% for entire strings and 91.68% for API calls. On the other hand, the LCS algorithm's accuracy is comparatively lower, with 83.78% for entire strings and 71.65% for API calls.