Semi-Automated Image Labeling and Object Tracking for MR-Guided Radiotherapy Using a Foundation Vision Model

Magnetic resonance-guided radiotherapy (MRgRT) enables real-time tracking of tumor and organ motion for more precise radiation delivery. However, existing clinical tracking software struggles with high-velocity or non-rigid motion. Foundation models, such as Segment Anything Model 2 (SAM2) and MedSAM2, offer robust tracking with fast inference speeds and strong zero-shot accuracy that can improve with fine-tuning. Fine-tuning typically uses expert-labeled datasets, which are expensive to acquire. We explored SAM2’s utility in accelerating the labeling process by developing a workflow where novices label < 1% of frames and SAM2 automatically labels the rest. We then evaluated SAM2 and MedSAM2 for tumor tracking on an open-access MRgRT dataset in three configurations: zero-shot, fine-tuned using manually labeled data, and fine-tuned using the semi-automatically labeled data. We measured accuracy with the Dice similarity coefficient, 95th percentile (P95) Hausdorff distance, and centroid distance. MedSAM2 and SAM2 demonstrated strong zero-shot tracking accuracy with no significant difference among the model variants. Fine-tuning significantly improved Dice coefficients (0.874 to 0.891, p = 0.006), P95 Hausdorff distances (9.069 to 4.378 mm, p < 0.001), and centroid distances (5.299 to 1.769 mm, p < 0.001) relative to zero-shot, with no significant difference between using manually labeled and semi-automatically labeled data. Therefore, leveraging SAM2 to semi-automatically generate labeled training data can substantially reduce the cost of data curation and long-term model maintenance without compromising accuracy.