A Hybrid CNN–BiLSTM Architecture Leveraging Transfer Learning for Medical Prescription Recognition

Recognizing handwritten medical prescriptions is a challenging problem due to the diverse styles of handwriting, similar character shapes, image noise and the absence of annotated data. To overcome these problems, two major changes are suggested: in the paper, a transfer learning-based CNN-BiLSTM network is introduced to accurately and efficiently digitize prescriptions. The approach introduced in this work leverages the Extended Modified National Institute of Standards and Technology (EMNIST) data set to pre-train the convolutional feature extractor using common handwritten character representations, and then fine-tunes the network with the BD Prescription data set to be suitable for real-world medical handwriting. In the pre-processing stage, prescription images are preprocessed to ensure input quality and reduce variability by converting to gray scale, reducing noise, normalizing and resizing the images. It uses Bi-Directional Long Short-Term Memory (Bi-LSTM) layers to better handle the context information of the sequential information between handwritten characters and words, and the convolutional neural network captures high-level spatial information. Finally, the whole layer whose activation is Softmax is utilized to create a final prediction for the prescription text recognition. The proposed framework is constructed using transfer learning and sequence modeling, which are effective in decreasing the complexity in training, encouraging convergence and improving the generalization to unseen handwritings. The results of the experiments demonstrate that the proposed method provides better recognition performance compared to the conventional methods, when trained on EMNIST and fine-tuned on BD Prescription datasets. The proposed solution provides automated prescription digitization which is scalable and dependable, and can help enable intelligent healthcare systems, intelligent pharmacy automation, and intelligent management of medical records. The proposed Knowledge transfer framework CNN-BiLSTM has been validated with BD Prescription dataset with an accuracy of 94.62% and the accuracy drop due to label misclassification has been corrected by semantic correction leads to improvement in validation accuracy to 96.28% and the test accuracy on BD Prescription dataset attained a gain of 4.49%.