Abstract C036: From pixels to prognosis: Attention-based chain of thought reasoning for automated spine cancer image analysis

Current vision– Large Language Models (V-LLMs) for spinal oncology imaging use a black-box approach towards generating caption from an MRI/CT scan image. This conflicts with what a real radiologist goes through when interpreting the same images. These models take the pixels, traverse a billion plus parameter latent space, and generate a caption in one shot. This “all-at-once” approach ignores the multi-pass workflow that expert radiologists follow—checking image quality and vertebral levels, mapping baseline anatomy, surveying for disease, measuring epidural tumor extension and spinal-canal compromise and then integrating these findings into a structured report that drives surgical or radiotherapy decisions. Because the vision model’s intermediate reasoning remains hidden, clinicians cannot confirm that the AI model has examined every clinically critical cue, they can also not trace the origin of potential errors when the AI model misses a small sacral lesion or overstates the degree of canal stenosis. This opacity limits trust, complicates regulatory review, and ultimately slows the adoption of AI in oncologic imaging. It also slows down clinical adoption at scale. A total of 1978 expert-captioned studies—radiographs, CT, and MRI—were collected from the public ROCO-v2 corpus which also included images for spine. The proposed solution is an inference-time five-step Chain-of-Thought (CoT) reasoning that uses a fine tuned 7-billion-parameter vision–language model that guides the model to generate the following:Image-quality & modality check, Systematic vertebral and soft-tissue mapping, Disease detection, Tumor-specific feature analysis, Clinical synthesis. After following these five steps, the system generates a structured clinical report while 32 layer-wise attention mechanism expose where in the image the AI model looked at every stage of image processing. This allows a clinician visibility into why of what the model generated as a caption to explain what’s happening in the image. This type of model scaffolding provides insights into model’s reasoning ability, why it generated a specific language, in addition to that, it lifts caption fidelity and provides an auditable trail that oncologists and regulators can inspect—all without the cost and delay of re-training the model for each new cancer-imaging task. The chain of thought reasoning improved caption accuracy by nearly 3x when compared to just the fine tuned AI vision model. Corpus BLEU-4 increased from 0.005 to 0.013 (+161%) and image level BLEU-4 score rose from 0.005 ± 0.002 to 0.014 ± 0.003 (+194 %). During testing, all 8 COT runs for spine related cancer completed without error. Patch-level attention captured from all 32 vision layers of the AI vision model showed early layers attending diffusely across vertebrae and later layers concentrating on tumor margins, corroborating the step-wise reasoning flow. The five-tage, attention based CoT scaffold transforms a 7 B-parameter V-LLM from a black box to a radiologist-like, auditable spine-cancer reader. Citation Format: Vedant Malik, Krish Malik. From pixels to prognosis: Attention-based chain of thought reasoning for automated spine cancer image analysis [abstract]. In: Proceedings of the 18th AACR Conference on the Science of Cancer Health Disparities; 2025 Sep 18-21; Baltimore, MD. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2025;34(9 Suppl):Abstract nr C036.