Abstract This work introduces an image quality assessment (IQA)-driven optimization framework for synthetic data generation (SDG) to improve the simulation-to-real-world (sim2real) transfer performance of synthetic vision systems. By using IQA metrics—particularly structural similarity (SSIM) based—as fitness functions for genetic algorithms (GAs), we automate the tuning of synthetic environments to enhance the generalizability of synthetic-trained object detectors in new or unobserved real-world scenarios. Digital-twin environments were constructed using both 3D Gaussian splatting from real images and by procedural generation methods. Two GAs were developed: a pre-process GA that modifies the environment before image generation for background focused similarity, and a post-process GA that adjusts noise and visual properties in the synthetic images, focusing on foreground-object similarity. The Synthetic Image Quality Analysis Calculator (SIQAC) software was extended to define regions of interest and evaluate transferability. Experimental results show sim2real accuracy improvements of up to 36% with the pre-process GA, 19% with the post-process GA, and 33% when combined, reaching near real vision system performance levels. These findings demonstrate the benefits of IQA-informed optimization in reducing the sim2real gap for synthetic vision systems.
Mardikes et al. (Thu,) studied this question.