Handwritten digit recognition: Comparative analysis of ML, CNN, vision transformer, and hybrid models on the MNIST dataset

Abstract Handwritten digit recognition (HDR) remains challenging due to variations in writing styles. To address this challenge, this study comprehensively compares ML (ML) and deep learning (DL) models. We explored a variety of approaches. We evaluated these models on the Modified National Institute of Standards and Technology (MNIST) and Extended Modified National Institute of Standards and Technology (EMNIST) datasets to assess their generalization capabilities. Initially, we investigated standalone ML and DL models trained from scratch to learn features directly. Logistic regression (LR) achieved an accuracy of 92.5% on MNIST and 86.63% on EMNIST. A multi-layer perceptron demonstrated improved performance with 98.10% accuracy on MNIST. Convolutional neural networks exhibited superior performance, reaching 99.90% accuracy on MNIST and 99.57% on EMNIST. To further enhance performance, we explored ensemble learning techniques, combining CNNs with RF (98.20 and 99.86% accuracy on MNIST and EMNIST, respectively), LR (88.67 and 99.79% accuracy on MNIST and EMNIST, respectively), and VC (99.27 and 99.83% accuracy on MNIST and EMNIST, respectively). We then introduced a ViT model, leveraging self-attention for long-range dependency modeling, achieving an accuracy of 98.70% on MNIST and 99.58% on EMNIST. Finally, we proposed a hybrid model combining CNN and ViT, that yielded the highest accuracy of 99.97% on MNIST and 98.26% on EMNIST. Throughout our experimentation, we employed various techniques such as regularization, weight initialization, and optimization strategies to improve model performance. The impact of each technique is analyzed and discussed. Overall, this study provides a comprehensive comparison of different HDR models, highlighting each approach’s strengths and weaknesses. The results demonstrate the effectiveness of DL models, particularly CNNs and hybrid architectures, in achieving high accuracy in HDR.