Medical X-ray image annotation using deep learning with local-global feature fusion

Image annotation refers to generating textual descriptions for images in a manner compatible with human language understanding. This technology holds significant importance in industries, particularly in the medical field, where it aids in reducing medical errors and associated costs. Although numerous methods have been proposed for automatic medical image annotation, existing approaches often fail to effectively extract deep local features from images, which limits their annotation accuracy. This study proposes a novel method combining nonlinear local descriptors, specifically Wavelet Scattering, with deep and local features to enhance feature extraction. The proposed framework introduces an encoder-decoder architecture for automatic medical image annotation. It leverages deep features extracted from the InceptionV3 network, along with local features derived from Wavelet Scattering and three additional descriptors—Zernike moments, Local Binary Patterns (LBP), and Gabor filters. A Principal Component Analysis (PCA)-based dimensionality reduction step is incorporated to reduce the transformer model's complexity. Moreover, the framework employs Relational Memory (RM) and Memory-Conditioned Layer Normalization (MCLN) to effectively capture and utilize critical information. Evaluations conducted on the IU X-Ray dataset demonstrate the effectiveness of the proposed model. For lateral lung images, it achieves a BLEU1 score of 0.393; for frontal lung images, a BLEU1 score of 0.518; and for both views combined, a BLEU1 score of 0.497. These promising results highlight the potential of the proposed approach in generating accurate annotations for medical images.