Synergistic fusion of deep and handcrafted features for accurate and robust machine learning based cervical cancer diagnosis

Cervical cancer remains a leading cause of morbidity and mortality among women worldwide. Automated Pap smear image analysis with recent advances in machine and deep learning offer a solution by enabling the development of early detection systems for classifying cervical abnormalities improving accuracy and efficiency and reducing the manual interpretation which is labor-intensive and subject to interobserver variability. This study proposes and evaluates a segmentation free methodologies framework for classifying cervical cancer, using machine learning focusing on deep learning, feature extraction methodology evaluation, and hybrid approaches. The Herlev Pap smear dataset used which contains 917 single cell images distributed across seven diagnostic categories. The framework specifically analyzes the performance of various machine learning classifiers combined with multiple feature extraction techniques, including Haralick and wavelet-packet texture features, autoencoder-learned features, and deep features from a pretrained VGG network. ANOVA is used for texture feature selection, while PCA followed by LDA is used to obtain compact class-discriminative representations from high-dimensional deep features. Six classifiers were evaluated under tiered five-fold cross-validation (CV) across three distinct tasks: multi-class classification, normal versus abnormal classification, and a more specific multi-class classification to differentiate between various cancer types and multiple normal categories. The strongest classification performance on our results demonstrates that hybrid approaches, particularly those integrating autoencoder features with texture descriptors, consistently achieved near-perfect binary classification and reached approximately 99.85%/99.96% test accuracy / F1-score for cancer-class classification using random forest. VGG-based representations also showed stable performance, especially for normal-class classification, reaching approximately 100% accuracy with tree-based classifiers. These findings underscore the value of combining unsupervised feature learning with classical handcrafted interpretable texture features to develop robust and complementary information for Pap smear classification. The proposed framework offers a transparent and computationally efficient alternative to fully end-to-end deep learning pipelines, while acknowledging the need for larger multi-center external validation before clinical deployment.