A Seven-Criteria Evaluation Framework for the Responsible Use of Synthetic Medical Data

Purpose To develop a framework for evaluating and reporting the quality of synthetic medical data (SMD). Method As SMDs become increasingly common, their responsible utilization necessitates a detailed characterization of their quality and value within the intended use context. Here, we introduce a framework to assess SMD across seven dimensions: Congruence, Coverage, Constraint, Consistency, Comprehension, Compliance, and Completeness. We also introduce an approach for reporting the quality of synthetic data to stake-holders. Results We applied the proposed framework to seven digital mam-mography datasets. Our analysis underscored the strengths and limita-tions of each dataset across these dimensions. For example, while datasets generated by generative AI methods show high Congruence, they often fall short in Constraint and Completeness compared to those generated by knowledge-based methods. Additionally, our findings highlighted that the quality of generated data varies across different subgroups (e.g., breast density) with certain subgroups showing lower quality. This suggests that subgroup-specific fine-tuning of the generative process may be necessary, as these disparities could impact the downstream tasks. Conclusion The proposed framework provides stakeholders with a tool for assessing data quality, which can be used to better understand and analyze synthetic datasets. Advances in knowledge This study introduces a framework for assessing SMD quality based on clinically and technically relevant crite-ria. The framework offers standardized and transparent mechanism for reporting SMD quality, which can facilitate the use of synthetic datasets in medical imaging AI device development.