Multilayer Ensemble Pruning via Novel Multi-sub-swarm Particle Swarm Optimization

Abstract: Recently, classifier ensemble methods are gaining more and more attention in the machine-learning and data-mining communities. In most cases, the performance of an ensemble is better than a single classifier. Many methods for creating diverse classifiers were developed during the past decade. When these diverse classifiers are generated, it is important to select the proper base classifier to join the ensemble. Usually, this selection process is called pruning the ensemble. In general, the ensemble pruning is a selection process in which an optimal combination will be selected from many existing base classifiers. Some base classifiers containing useful information may be excluded in this pruning process. To avoid this problem, the multilayer ensemble pruning model is used in this paper. In this model, the pruning of one layer can be seen as a multimodal optimization problem. A novel multi-sub-swarm particle swarm optimization (MSSPSO) is used here to find multi-solutions for this multilayer ensemble pruning model. In this model, each base classifier will generate an oracle output. Each layer will use MSSPSO algorithm to generate a different pruning based on previous oracle output. A series of experiments using UCI dataset is conducted, the experimental results show that the multilayer ensemble pruning via MSSPSO algorithm can improve the generalization performance of the multi-classifiers ensemble system. Besides, the experimental results show a relationship between the diversity and the pruning technique.