When Resampling Isn't Enough: A Case Study in Distinguishing Class Imbalance from Label Noise in Sentiment Classification

Class imbalance in text classification is usually treated as a resampling problem — duplicate the minority class, generate synthetic examples, or drop majority-class data until the classes are balanced. In this case study, we tested five such techniques (Random Oversampling, Random Undersampling, SMOTE, Borderline-SMOTE, and ADASYN) on a three-class Amazon review sentiment classifier (SVM and Neural Network), and found that minority-class ("negative") recall stayed stuck at around 14% no matter which technique we used, except Random Undersampling, which raised recall to as high as 57% but crashed overall accuracy to 38.2%. Since four algorithmically different resampling methods all hit the same wall, we treated that as a clue rather than a dead end. Reading the actual misclassified reviews by hand showed several "negative" labels didn't match the review content at all. The cause turned out to be a bug in the labeling function: it only scored the sentiment of the first word in each review, not the review as a whole, because a return statement inside a for loop exited before reaching the rest of the text. Fixing this one line changed the class distribution itself (negative reviews grew from 27 to 194 out of ~4,000) and improved every result we had, more than any resampling technique managed on the original data. Negative recall rose to 20-84% depending on method, and the Neural Network's earlier instability under undersampling (38.2% accuracy) resolved to 68.8% once it was trained on correct labels. The takeaway: when resampling fails the same way across multiple different techniques, that's a sign to check the labels, not to try another algorithm.