From synthetic images to clinical reports: effect of data augmentation on transformer-based chest X-ray report generation with cross-modal attention

In the process of training report generating models, the scarcity of large-scale labelled medical datasets remains a fundamental challenge. This study investigates the impact of three data augmentation methods, i.e., Latent Diffusion Model, Progressive GAN, and Conditional DCGAN on the performance of chest X-ray report generation systems. We developed a novel transformer-based encoder-decoder architecture that incorporates a visual encoder with dual spatial-channel attention and a GPT-style decoder with cross-modal attention and gated fusion mechanism. We also propose a novel accuracy-stability combined scoring metric for robust model selection. The experimental results on the IU X-ray dataset demonstrated that YOLOv8m backbone had better feature extraction capability and CXR-BERT outperformed other language models due to its domain-specific pre-training on chest X-ray reports. Meanwhile, among augmentation methods, the Diffusion-based model achieved superior image quality. We also concluded that data augmentation consistently enhanced report generation performance across all language models by 5−12% improvement over baseline.