VoteDroid: a new ensemble voting classifier for malware detection based on fine-tuned deep learning models

Abstract In this work, VoteDroid a novel fine-tuned deep learning models-based ensemble voting classifier has been proposed for detecting malicious behavior in Android applications. To this end, we proposed adopting the random search optimization algorithm for deciding the structure of the models used as voter classifiers in the ensemble classifier. We specified the potential components that can be used in each model and left the random search algorithm taking a decision about the structure of the model including the number of each component that should be used and its location in the structure. This optimization method has been used to build three different deep learning models namely CNN-ANN, pure CNN, and pure ANN. After selecting the best structure for each DL model, the selected three models have been trained and tested using the constructed image dataset. Afterward, we suggested hybridizing the fine-tuned three deep-learning models to form one ensemble voting classifier with two different working modes namely MMR (Malware Minority Rule) and LMR (Label Majority Rule). To our knowledge, this is the first time that an ensemble classifier has been fine-tuned and hybridized in this way for malware detection. The results showed that the proposed models were promising, where the classification accuracy exceeded 97% in all experiments.