A hybrid random forest–LSTM framework for breast cancer diagnosis using cytological features

Breast cancer remains one of the leading causes of cancer-related mortality among women worldwide, emphasizing the need for accurate and early diagnostic approaches. Machine learning (ML) techniques have shown significant potential in improving diagnostic precision by leveraging structured biomedical data. In this article, we propose a hybrid framework combining random forest (RF)–based feature selection with a long short-term memory (LSTM) model for breast cancer classification using cytological features derived from fine needle aspiration (FNA) samples. The Breast Cancer Wisconsin (Diagnostic) dataset was utilized, consisting of 569 instances with 30 quantitative features describing cell nucleus characteristics. RF was employed to identify the most relevant features contributing to malignancy classification, and the top-ranked features were used as input to the LSTM model. Although the dataset is non-temporal, features were structured into a sequential format to enable LSTM-based learning of complex feature interactions. The proposed model achieved an accuracy of 98.25% on the test dataset and demonstrated robust performance during 5-fold cross-validation with an accuracy of 94.95%, outperforming traditional ML models, including gradient boosting, support vector machines (SVMs), and artificial neural networks (ANNs). The results indicate that the integration of RF feature selection with LSTM-based modeling provides an effective and partially interpretable framework for breast cancer diagnosis. This approach highlights the potential of hybrid ML models in enhancing diagnostic accuracy and supporting clinical decision-making.