Attention U-Net-Based Segmentation and Hybrid Classification for Detection of Circulating Tumor-Associated Cells

Background/Objectives: Circulating tumor-associated cells (CTACs) are rare among peripheral blood nucleated cells (PBNCs), creating a challenge for image-based multi-cancer detection. We evaluated a predefined CTAC-detection pipeline incorporating Attention U-Net segmentation, post-processing, cytological feature extraction, and Random Forest classification. Methods: Model suitability was explored in asymptomatic individuals and patients with advanced solid tumors. Clinical performance was assessed in a case–control cohort of therapy-naive stage I/II cancers, benign conditions, and asymptomatic individuals, followed by four prospective cohort evaluations performed within the same laboratory and imaging workflow: recurrent cancer with low radiological tumor burden, peri-operative solid tumors, suspected cancer, and asymptomatic screening. PBNCs were stained with EpCAM/Hoechst 33342 and imaged. Pathologists’ review established ground truth annotations. Results: The model had 90.68% sensitivity and 99.53% specificity in the exploratory study. In the case–control cohort, sensitivity was 88.65% in therapy-naive stage I/II cancers, while specificity was 78.95% in benign conditions and >99.9% in asymptomatic individuals. In the prospective cohorts, CTAC detection sensitivity was 91.96% in pretreated low tumor burden cases; CTACs were detected in 100% of pre-surgery specimens and 29.41% of post-surgery specimens; and in suspected cancer cases, the Positive Predictive Value (PPV) and Negative Predictive Value (NPV) were 96.34% and 32.35%, respectively. In the asymptomatic screening cohort, 44/7183 participants were CTAC-positive; 16 had confirmed Stage I/II cancer, 10 had no radiologically detectable disease at the available assessment, and 18 remained unresolved. The conservative lower-bound PPV was 36.36%, and the NPV was 99.97%; estimates remain provisional pending complete follow-up. Conclusions: The integrated Attention U-Net/feature-based classification pipeline demonstrated consistent CTAC detection across the evaluated cohorts and supports its potential clinical utility for cancer detection.