Prostate-Specific Membrane Antigen PET/CT Segmentation Guided by Vision Language Models Embedding Clinical Report Semantics

Whole-body prostate-specific membrane antigen (PSMA) PET/CT segmentation remains challenging: manual contouring is unsustainable, and image-only artificial intelligence models often misclassify physiologic uptake as malignant, causing high false-positive rates. These limitations underscore the need for segmentation strategies that incorporate the nuclear medicine expertise encoded in PET/CT reports but not used by current models. We investigated whether routine PET/CT reports improve automated lesion segmentation for prostate cancer. Methods: We developed a language-guided 3-dimensional SegResNet fusing report text with 68Ga-PSMA-11 PET/CT image features, encoding report semantics via 4 strategies (BiomedVLP, RadGraph, GPT-raw, GPT-engineered) compared with an image-only model (baseline). Of 320 patients who underwent 68Ga-PSMA-11 PET/CT, 297 studies were available; 13 were excluded for missing reports or incomplete image-mask pairs, leaving 284 for analysis, split at the patient level into 243 development cases and a fixed 41-case held-out test set. Metrics included Dice, intersection over union (IoU), precision, recall, average symmetric surface distance, 95th percentile Hausdorff distance, surface Dice similarity at 5 mm, and lesion-level false positives, using 3 matching criteria (any overlap; 1-to-1 maximum IoU matching, IoU ≥ 0.5; and correct localization of the SUVmax voxel). Significance was assessed using paired 1-sided Wilcoxon tests with Holm adjustment. Results: Report guidance consistently reduced false positives and improved whole-mask boundary agreement. False-positive lesions were reduced by nearly 4 per patient with BiomedVLP and by approximately 3 per patient with RadGraph and GPT-raw. Average symmetric surface distance decreased from 105.4 ± 62.5 mm (image only) to 23.4 ± 18.2 mm with BiomedVLP, 36.9 ± 22.4 mm with RadGraph, and 36.3 ± 26.4 mm with GPT-raw; 95th percentile Hausdorff distance improved from 455.0 ± 169.3 mm to 123.8 ± 80.3 mm with BiomedVLP, and surface Dice similarity at 5 mm increased from 0.638 ± 0.172 to 0.705 ± 0.181. However, when these surface metrics were recomputed on matched true-positive lesions only using the same 3 criteria, no language-guided model differed significantly from the image-only model for any surface metric (all Holm-adjusted P = 1.0), indicating that the whole-mask improvements were attributable to false-positive suppression rather than genuine gains in contour accuracy on correctly detected lesions. Precision increased for BiomedVLP (0.099) and RadGraph (0.060), with a modest decrease in voxel-level recall (−0.051, BiomedVLP), reflecting a precision-recall trade-off from suppression of spurious uptake. Dice improved by 0.032 but was borderline significant; IoU increased for BiomedVLP (0.036). Lesion-level sensitivity changed by no more than 0.04 for all models and did not reach significance. Total lesion PSMA remained highly correlated with manual measurements (r ≥ 0.85), whereas PSMA tumor volume correlations were more variable across models (r = 0.37–0.70). Conclusion: The integration of PET/CT report semantics into automated PSMA PET/CT segmentation substantially reduced false-positive lesions and preserved total lesion PSMA agreement with manual measurements; a matched true-positive-only reanalysis indicated that the accompanying whole-mask boundary-agreement gains reflected false-positive suppression rather than improved contour accuracy on correctly detected lesions. This approach may reduce manual cleanup and support more reliable quantitative workflows, although precision gains should be weighed against a modest recall reduction. External validation, extension of the true-positive-only analysis to independent cohorts, and PSMA-specific language pretraining are warranted.