A novel vision transformer-based approach to detect generative model fingerprint

Generative model fingerprint detection has become of very vital importance in the evolving field of medical image processing. The generative adversarial networks (GAN) fingerprint is crucial for improving generative model interpretability, solving real vs. synthetic image challenges, and addressing data scarcity and ethical issues in image generation. This study focuses on identifying those fingerprints that are left in the images by GAN, particularly in 3-D CT scans of patients affected with lung tuberculosis generated by multiple GAN and diffusion models. A design of such a new multiformer-based ensemble model, including CLIP and BLIP2 architectures, would have better distinction between these fingerprints incomparably with the traditional methods. In this work, thresholding, late fusion, and morphological techniques are applied, producing very promising results in terms of the ARI metric: 0.90 and 0.8137 with fine-tuned variants. Accordingly, it demonstrates rich potential for quality control concerning synthetic images in medical applications by effectively separating images created with different models.