Improving the performance of radiology report de-identification with large-scale training and benchmarking against cloud vendor methods

To enhance automated de-identification of radiology reports by scaling transformer-based models through extensive training datasets and benchmarking commercial cloud vendor systems using a privacy-preserving evaluation framework for protected health information (PHI) detection. In this retrospective study, we built upon our state-of-the-art, transformer-based, PHI de-identification pipeline by further training on two large annotated radiology corpora from Stanford University, encompassing chest X-ray, chest CT, abdomen/pelvis CT, and brain MR reports and introducing an additional PHI category (AGE ≥ 90) into the architecture. Model performance was evaluated on test sets from Stanford and the University of Pennsylvania (Penn) for token-level PHI detection. We further assessed (1) the stability of synthetic PHI generation using a “hide-in-plain-sight” method and (2) agreement with commercial systems under a privacy-preserving synthetic-label evaluation framework. Precision, recall, and F1 scores were computed across all PHI categories. Our model achieved overall F1 scores of 0.973 on the Penn dataset and 0.999 on the Stanford dataset, achieving increases of up to approximately 0.7 F1 on the Stanford test set in the HOSPITAL, PATIENT, and VENDOR categories while maintaining strong overall performance on the independently annotated Penn dataset. Synthetic PHI evaluation showed consistent detectability (overall F1: 0.959 [0.958–0.960]) across 50 independently de-identified Penn datasets. Under a privacy-preserving evaluation framework using synthetic PHI surrogate labels as reference markers, our model demonstrated higher agreement with the synthetic labels than the evaluated cloud vendor systems on synthetic Penn reports (overall F1: 0.960 vs. 0.632–0.754). Because these reference labels were generated by our own de-identification pipeline, these findings reflect relative agreement rather than independent comparative detection accuracy. Large-scale, multimodal training supported strong performance across two institutions while synthetic PHI generation enabled privacy-preserving evaluation. A transformer-based de-identification model that performs highly in PHI detection on diverse radiology datasets, supporting privacy-preserving evaluation of radiology report de-identification systems.