Unearthing Code Smells: An In‐Depth Exploration of Machine Learning Techniques in Code Smell Detection

Code smells are clear indicators of design flaws that compromise maintainability, reliability, and overall performance in software systems. As systems become complex, traditional detection methods become inadequate, prompting the need for automated approaches. In this study, we evaluate the potential of conventional machine learning algorithms for detecting critical code smells. Specifically, we examine Classification and Regression Trees (CART), Gradient Boosting Classifiers, Extreme Gradient Boosting (XGBoost), RuleFit, Adaptive Boosting (AdaBoost), K ‐Nearest Neighbors (KNN), Stochastic Gradient Descent (SGD), and Random Forest to identify six key code smell categories: Large Class, Long Parameter List, Switch Statements, God Class, Data Class, and Long Method. Advanced resampling techniques such as SMOTE–ENN and SMOTE–Tomek are employed to mitigate dataset imbalances alongside tree‐based feature extraction for effective dimensionality reduction. Systematic experimentation demonstrates that SMOTE‐based strategies significantly enhance detection performance, yielding robust results across multiple evaluation metrics. By focusing on the core challenges of code smell detection, this study provides a cost‐effective alternative to deep learning methods, which often demand extensive computational resources and large labeled datasets.