Introduction: Synthetic data generation is a rapidly evolving field, with significant potential for improving data privacy. However, evaluating the performance of synthetic data generation methods, especially the tradeoff between fairness and utility of the generated data, remains a challenge. Methodology: In this work, we present our comprehensive framework, which evaluates fair synthetic data generation methods, benchmarking them against state-of-the-art synthesizers. Results: The proposed framework consists of selection, evaluation, and application components that assess fairness, utility, and resemblance in real-world scenarios. The framework was applied to state-of-the-art data synthesizers, including TabFairGAN, DECAF, TVAE, and CTGAN, using a publicly available medical dataset. Discussion: The results reveal the strengths and limitations of each synthesizer, including their bias mitigation strategies and trade-offs between fairness and utility, thereby showing the framework’s effectiveness. The proposed framework offers valuable insights into the fairness-utility tradeoff and evaluation of synthetic data generation methods, with far-reaching implications for various applications in the medical domain and beyond. Conclusion: The findings demonstrate the importance of considering fairness in synthetic data generation and the need for fairness focused evaluation frameworks, highlighting the significance of continued research in this area.
Warnecke et al. (Wed,) studied this question.