SFC-SMOTE: A Sample Filtering Clustering Oversampling Algorithm for Imbalanced Clinical Diagnostic Data

Background: Medical data mining is an important research direction in the data mining field and has always been a research hotspot in the computer and medical fields for many years. Data mining algorithms usually assume that the sample distribution of data is balanced and the misclassification cost is equal. However, due to the characteristics of medical research, there are serious sample imbalances in the clinical diagnostic data collected, which leads to poor classification effects of data mining algorithms. Methods: For the problem of sample imbalance in clinical diagnosis data, this paper proposes a sample filtering clustering oversampling algorithm (SFC SMOTE). Firstly, the k-means algorithm is used to group the clinical diagnosis data so that the similarity of the samples within the group is high while the similarity of the samples between the groups is low. Then, the sample filtering strategy is used to identify and filter the "noisy samples" in the minority and the majority and merge the filtered samples of the two classes according to the original class of the samples. Finally, the dynamic rate strategy is set according to the original two-class sample numbers, and the minority and majority samples are dynamically synthesized to restore the original structure of the data. Results: Experimental results show that the three classification algorithms have the best classification effect on the dataset sampled by the SFC SMOTE algorithm. Among them, the average Sensitivity, Specificity and MCC of Random forest are 92.48%, 92.07% and 84.57%, respectively, which are significantly better than the existing algorithms. In addition, the Wilcoxon and Friedman test results show that the SFC SMOTE algorithm is significantly better than the existing algorithms. Conclusion: The classification effect of the three classification algorithms has been significantly improved after all sampling algorithms. In addition, the oversampling algorithm is significantly better than the clustering oversampling algorithm. More importantly, no matter which classification algorithm is used, the classification effect of the dataset sampled by the SFC SMOTE algorithm is better.