A Mask R-CNN-Based Approach for Brain Aneurysm Detection and Segmentation from TOF-MRA Data

Background: Accurate detection of intracranial aneurysms, especially those smaller than 3 mm, remains a critical challenge in neurovascular imaging due to their subtle morphology and low contrast in Time-of-Flight Magnetic Resonance Angiography (TOF-MRA) scans. This study presents a Mask R-CNN-based deep learning framework designed to automatically detect and segment intracranial aneurysms, with specific architectural modifications aimed at improving sensitivity to small lesions. Method: A dataset of 447 TOF-MRA volumes (161 aneurysmal, 286 healthy) was used, with patient-level deduplication and 5-fold cross-validation to ensure robust evaluation. Bayesian hyperparameter optimization was applied using Optuna, and two key innovations were introduced: a Small Object Aware ROI Head to better capture micro-aneurysms and customized anchor configurations to improve region proposal quality. Healthy scans were incorporated as negative samples to enhance background modeling, and targeted data augmentation increased model generalization. Results: The proposed model achieved a Dice coefficient of 0.8832, precision of 0.9404, and sensitivity (recall) of 0.8677, with consistent performance across aneurysm sizes. Conclusions: These results demonstrate that the integration of architectural innovations, automated optimization, and negative-sample modeling enables a clinically viable deep learning tool that could serve as a reliable second-reader system for assisting radiologists in intracranial aneurysm detection.