A deep learning framework for automatic detection of arbitrarily shaped fiducial markers in intrafraction fluoroscopic images

PURPOSE: Real-time image-guided adaptive radiation therapy (IGART) requires accurate marker segmentation to resolve three-dimensional (3D) motion based on two-dimensional (2D) fluoroscopic images. Most common marker segmentation methods require prior knowledge of marker properties to construct a template. If marker properties are not known, an additional learning period is required to build the template which exposes the patient to an additional imaging dose. This work investigates a deep learning-based fiducial marker classifier for use in real-time IGART that requires no prior patient-specific data or additional learning periods. The proposed tracking system uses convolutional neural network (CNN) models to segment cylindrical and arbitrarily shaped fiducial markers. METHODS: The tracking system uses a tracking window approach to perform sliding window classification of each implanted marker. Three cylindrical marker training datasets were generated from phantom kilovoltage (kV) and patient intrafraction images with increasing levels of megavoltage (MV) scatter. The cylindrical shaped marker CNNs were validated on unseen kV fluoroscopic images from 12 fractions of 10 prostate cancer patients with implanted gold fiducials. For the training and validation of the arbitrarily shaped marker CNNs, cone beam computed tomography (CBCT) projection images from ten fractions of seven lung cancer patients with implanted coiled markers were used. The arbitrarily shaped marker CNNs were trained using three patients and the other four unseen patients were used for validation. The effects of full training using a compact CNN (four layers with learnable weights) and transfer learning using a pretrained CNN (AlexNet, eight layers with learnable weights) were analyzed. Each CNN was evaluated using a Precision-Recall curve (PRC), the area under the PRC plot (AUC), and by the calculation of sensitivity and specificity. The tracking system was assessed using the validation data and the accuracy was quantified by calculating the mean error, root-mean-square error (RMSE) and the 1st and 99th percentiles of the error. RESULTS: The fully trained CNN on the dataset with moderate noise levels had a sensitivity of 99.00% and specificity of 98.92%. Transfer learning of AlexNet resulted in a sensitivity and specificity of 99.42% and 98.13%, respectively, for the same datasets. For the arbitrarily shaped marker CNNs, the sensitivity was 98.58% and specificity was 98.97% for the fully trained CNN. The transfer learning CNN had a sensitivity and specificity of 98.49% and 99.56%, respectively. The CNNs were successfully incorporated into a multiple object tracking system for both cylindrical and arbitrarily shaped markers. The cylindrical shaped marker tracking had a mean RMSE of 1.6 ± 0.2 pixels and 1.3 ± 0.4 pixels in the x- and y-directions, respectively. The arbitrarily shaped marker tracking had a mean RMSE of 3.0 ± 0.5 pixels and 2.2 ± 0.4 pixels in the x- and y-directions, respectively. CONCLUSION: With deep learning CNNs, high classification performances on unseen patient images were achieved for both cylindrical and arbitrarily shaped markers. Furthermore, the application of CNN models to intrafraction monitoring was demonstrated using a simple tracking system. The results demonstrate that CNN models can be used to track markers without prior knowledge of the marker properties or an additional learning period.