A Semi-Supervised Ensemble Classification with Data Augmentation to Reduce Imbalanced Class Distribution in Big Data

With the age of digital information, managing, retrieving, and processing information in the form of big data has become the most substantial and difficult task. Though several tools are available for data analysis, they may not perform well with high-dimensional data. Machine learning algorithms play a major role in acquiring useful knowledge from this massive data. Consequently, a better technique that is extremely fast in handling a huge volume of data with numerous dimensions and processing ability in a distributed setting is indispensable. As the data has been gathered from various sources, data preprocessing plays a vital role in transforming it into a suitable form for acquiring useful insights. Despite the data being pre-processed, class imbalance remains a significant concern that must be addressed to enhance the outcomes. This work proposes a semi-supervised ensemble classifier utilizing data augmentation and the classroom rebalancing sampling to mitigate uneven class distribution in large datasets. The model employs ensemble techniques with two heterogeneous classifiers: extreme gradient boosting as a base learner and random forest for pseudo-labelling and classification. The Hadoop framework implements the model, which is suitable and effective for very large datasets. Following the execution of the experimental evaluation for the suggested framework, the evaluation metrics of the results indicate that the approach achieves an average accuracy of 84.3% over 16 datasets, reflecting an increase of around 17.65% in prediction performance.