RaGAN-Seg: A relativistic adversarial enhancement and segmentation framework for nucleus segmentation in IHC images

Objective: Accurate nucleus segmentation in immunohistochemistry (IHC) images is essential for quantitative analysis in digital pathology. However, segmentation remains challenging due to heterogeneous tissue types, staining differences, and multi-modal imaging variations. Existing segmentation models exhibit poor generalization across diverse IHC tissues. This study aims to develop a robust and generalizable framework for nucleus segmentation across diverse IHC image types. Methods: We propose RaGAN-Seg, a two-stage segmentation framework comprising an image enhancement module and a segmentation module. The enhancement stage employs R3GAN, a lightweight generative adversarial network with grouped convolutions, residual connections, and interpolative upsampling, optimized with relativistic adversarial loss and R1/R2 gradient penalties. The segmentation stage utilizes U-Net for classification combined with watershed post-processing for contour refinement. The framework was evaluated on a comprehensive private IHC dataset encompassing 480 images across 8 tissue types (4 seen and 4 unseen) with both brightfield and fluorescence imaging modalities. Results: RaGAN-Seg achieves significant performance improvements over state-of-the-art methods. On brightfield images, RaGAN-Seg outperforms DistSeg-Net by 11.7% in AJI, 14.4% in AJI+, 2.8% in Dice, 4.0% in DQ, and 7.9% in PQ. On fluorescence images, corresponding improvements are 12.9%, 12.1%, 1.2%, 13.1%, and 11.5%, respectively. Ablation studies confirm the necessity of relativistic adversarial loss design. The framework demonstrates strong generalization to unseen tissue types. Conclusion: RaGAN-Seg effectively combines adversarial enhancement with U-Net segmentation to address IHC image heterogeneity. The two-stage design enhances adaptability to complex histological variations, providing a robust and generalizable solution for nuclear segmentation in both research and clinical digital pathology applications.