Hybrid Vision Transformer and Convolutional Neural Network for Multi-Class and Multi-Label Classification of Tuberculosis Anomalies on Chest X-Ray

Tuberculosis (TB), caused by Mycobacterium tuberculosis, remains a leading cause of global mortality. While TB detection can be performed through chest X-ray (CXR) analysis, numerous studies have leveraged AI to automate and enhance the diagnostic process. However, existing approaches often focus on partial or incomplete lesion detection, lacking comprehensive multi-class and multi-label solutions for the full range of TB-related anomalies. To address this, we present a hybrid AI model combining vision transformer (ViT) and convolutional neural network (CNN) architectures for efficient multi-class and multi-label classification of 14 TB-related anomalies. Using 133 CXR images from Dr. Cipto Mangunkusumo National Central General Hospital and 214 images from the NIH datasets, we tackled data imbalance with augmentation, class weighting, and focal loss. The model achieved an accuracy of 0.911, a loss of 0.285, and an AUC of 0.510. Given the complexity of handling not only multi-class but also multi-label data with imbalanced and limited samples, the AUC score reflects the challenging nature of the task rather than any shortcoming of the model itself. By classifying the most distinct TB-related labels in a single AI study, this research highlights the potential of AI to enhance both the accuracy and efficiency of detecting TB-related anomalies, offering valuable advancements in combating this global health burden.