Attention-Based Medical Knowledge Injection in Deep Image Classification Models

Deep learning for medical image classification is extremely important for decision support in medical healthcare settings. General-purpose neural network architectures for image classification have become increasingly sophisticated in recent years. Among them, attention-based models have provided significant advancements in deep learning. However, attention mechanisms adopted thus far focus on minimizing task-specific losses and do not fruitfully exploit medical knowledge, such as lesion-specific characteristics, during the training process, resulting in a potential reduction in accuracy. In this paper, we propose an attention-based approach that leverages knowledge of the localization of specific lesion types to guide the model training process. To this end, gradient-based activation mapping is used for incentivizing models to focus on the right area for a given lesion type. The approach is general since it can be applied to any deep learning architecture in end-to-end model training. Our experiments on two real-world medical image datasets show the ability of our approach to improve the classification performance of popular deep-learning model architectures over the classical cross-entropy loss.