Design, training, and applications of foundation model for chest computed tomography volumes

Recent development in deep learning based image processing has spurred research in development of large and complex imaging models for detection of disease or segmentation of regions of interest like masses/nodules or organs in high-dimensional medical imaging data like CPT and MRI volumes. Given the complexity of the model and dimensionality of the input, large amounts of annotated training data is required to build such models. However, the annotation process for medical imaging data is challenging because of tedious processes and limited availability of clinical experts such as radiologists. Hence, only small amounts of annotated imaging data is generally available for any novel task – resulting in overfitted models with poor generalization capacity. A common solution to this challenge has been to pre-train the models on large imaging dataset of generic images like ImageNet in a supervised manner using class/object labels available with the data and then fine tuning them on small amounts of annotated medical imaging data [1]. This strategy often fails when pre-training and downstream tasks do not share the domain and thus the low-level visual features (e.g. classification of tumor molecular subtype using model pretrained on ImageNet dataset). In contrast to fine-tuning, foundation models [2] are deep models with multiple layers trained on large datasets in self-supervised fashion at a scale which attempt to learn generic low-level features, with the potential to transfer their knowledge of a variety of downstream target tasks - even when the self-supervised and target tasks are not similar. Attempts have been made to train such models in various domains including natural language processing (NLP), image processing, as well as combined vision-language research tasks [3] [4] [5] [6] [7] [8] [9] [10]. Recently, the NLP domain has arguably undergone revolutionary advances with foundation models allowing release of large language models (LLMs), such as GPT [3]. Similar ideas of self-supervision were translated in the domain of image processing. Such models are used to learn image representations, encode fundamental visual characteristics, and are tested for downstream prediction tasks like object detection on datasets like ImageNet. Simultaneously, complex vision-language models like CLIP [6], DALL-E [8], FLAVA [9], and Socratic models [12] were trained in a self-supervised way for understanding correlation in multi-modal data, and enabling reasoning across multiple modalities, and even applied in generation tasks. Inspired by vast research performed in the field of image and natural language processing for self-supervised training of foundation models [13] [4] [5], we proposed radiology-specific task agnostic self-supervision techniques. We experimented with a systematic solution for handling limited annotated data problems for a commonly performed imaging exam (CT of the chest) that can be independent of the downstream clinical tasks. Potential downstream applications for chest CT are vast including segmentation of multiple organs like lungs and heart and/or segmentation of lung nodules, and detection of various abnormalities, such as pulmonary embolism, COVID-19, pneumonia, and lung cancer, coronary calcium scoring. Many of these tasks have been released with datasets as multiple public challenges. To counteract scarcity of annotated data at individual institutions, data in challenges are often pooled from multiple healthcare organizations and can make large, annotated sets available for relevant tasks like PE detection and lung nodule segmentation. However, this approach is not scalable for producing annotated sets for all relevant applications outside of the intended target (e.g., using publicly available PE dataset for cardiac segmentation). Thus, a foundation model could fill this gap by extending its ability for various downstream tasks outside of its core training task. The foundation model can reduce the need for large volume annotated data for any chest CT axial view based downstream task providing a task-independent solution to data annotation challenge. We evaluated the performance of the foundation model on two separate clinical tasks (pulmonary embolism (PE) detection, lung nodule segmentation) and compared them to state-of-the-art models (SOTA).