Dynamic Optimistic Ensemble Clustered Feature Selection Techniques for Big Data Analysis

Feature selection and clustering are essential for managing high‐dimensional datasets efficiently. The machine learning model performance can be improved by utilizing only the most relevant features and minimizing computational needs through feature selection techniques. The existing feature selection techniques fail to perform efficiently in the case of high‐dimensional data. In this research, a dynamic optimistic ensemble clustered feature selection (DOECFS) method is proposed that integrates multiple feature selection and clustering techniques dynamically to enhance feature relevance. The proposed method is evaluated using nine diverse datasets from the University of California Irvine (UCI) machine learning repository. Data preprocessing is performed using the min–max scaler to normalize feature values, ensuring consistency and enhancing the performance of the feature selection process. The DOECFS method integrates clustering techniques, including dynamic adjustments of clustering parameters based on data characteristics and adaptive feature selection using stability scores, PCA, and optimization through the spiral dynamic algorithm (SDA) to enhance feature relevance by exploring the feature space using a spiral search strategy. Feature distance metrics, including squared Euclidean, cosine, maximum, Mahalanobis, Manhattan, and Euclidean distances, are used to evaluate the effectiveness of the selected features, and the Wilcoxon rank‐sum test (WRST) evaluates if two independent groups have significantly different distributions in the Python platform. The proposed method’s average cluster accuracy for different linkage strategies is 99.66% and the proposed ensemble cluster accuracy is 99.44% as compared to other cluster approaches such as CSPA, LCE and K‐modes (94.3%, 90.98%, and 93.8%, respectively).