Learning pentary embedding costs for enhancing image steganography security via curriculum-based GAN

Embedding-cost design is central to distortion-minimization steganography, yet most GAN-based cost-learning methods are restricted to ternary modifications ( \0,± 1\ { 0 , ± 1 } ), which can become limiting at high payloads. We propose Curriculum-based Pentary Embedding GAN (CuPE-GAN) for learning pentary ( \0,± 1,± 2\ { 0 , ± 1 , ± 2 } ) embedding costs. CuPE-GAN uses a two-layer ternary training surrogate: Stage 1 learns the first-layer ± 1 ± 1 allocation, whereas Stage 2 activates the second layer with forced learning-rate decay; a directional-consistency loss discourages canceling layerwise changes; and an Improved Decoupled Ternary Approximation (IDTA) represents the two modification directions with separate one-sided soft indicators and exact zero-probability consistency at finite λ λ . The learned layer probabilities are converted to a normalized five-symbol distribution and then to pentary costs, while actual message embedding is performed by a standard pentary STC encoder. Trained on CelebA-HQ and ALASKA1 and evaluated on BOSSBase and BOWS2, CuPE-GAN achieves higher average detection error rates than Steg-GMAN, WOW, and S-UNIWARD at medium and high payloads. Relative to Steg-GMAN, the average gains at 0.4 and 0.8 bpp are 6.16 and 5.23 percentage points on BOSSBase, 5.05 and 7.92 percentage points on BOWS2.