Confidence-Calibrating Regularization for Robust Brain MRI Segmentation Under Domain Shift

The Segment Anything Model (SAM) exhibits strong zero-shot performance on natural images but suffers from domain shift and overconfidence when applied to medical volumes. We propose CalSAM, a lightweight adaptation framework that (i) reduces encoder sensitivity to domain shift via a Feature Fisher Information Penalty (FIP) computed on 3D feature maps and (ii) penalizes overconfident voxel-wise errors through a Confidence Misalignment Penalty (CMP). The combined loss, \ (L₂₀₋ₒ₀₌\) fine-tunes only the mask decoder while keeping SAM's encoders frozen. On cross-center and scanner-shift evaluations, CalSAM substantially improves accuracy and calibration: e. g. , on the BraTS scanner split (SiemensGE) CalSAM shows a +7. 4\% relative improvement in DSC (80. 1\% vs. \ 74. 6\%), a -26. 9\% reduction in HD95 (4. 6 mm vs. \ 6. 3 mm), and a -39. 5\% reduction in ECE (5. 2\% vs. \ 8. 6\%). On ATLAS-C (motion corruptions), CalSAM achieves a +5. 3\% relative improvement in DSC (75. 9\%) and a -32. 6\% reduction in ECE (5. 8\%). Ablations show FIP and CMP contribute complementary gains (p<0. 01), and the Fisher penalty incurs a modest 15\% training-time overhead. CalSAM therefore delivers improved domain generalization and better-calibrated uncertainty estimates for brain MRI segmentation, while retaining the computational benefits of freezing SAM's encoder.