A novel 3D hyperchaotic system for secure RGB image encryption with coupled bidirectional diffusion: design, dynamical analysis, and cryptographic evaluation

The development of nonlinear dynamical systems with complex chaotic behavior has attracted considerable attention for secure information processing and cryptographic applications. In this study, a three-dimensional hyperchaotic (HC-3D) system is introduced using nonlinear state coupling, sinusoidal modulation, and a modulo-one transformation to generate bounded complex dynamics. Its dynamical behavior is investigated through equilibrium analysis, bifurcation diagrams, phase portraits, Lyapunov exponent analysis, complexity evaluation, and parameter-space studies. Based on the generated chaotic sequences, an RGB image encryption scheme is developed using session-dependent spatial and channel permutations, hybrid chaotic–hash-based keystream generation, and coupled bidirectional diffusion. A cryptographic master key and a session-dependent nonce are used to derive session-specific initial conditions and encryption sequences, providing separation between encryption sessions. The security performance is evaluated using histogram analysis, information entropy, adjacent-pixel correlation, chi-square testing, NPCR, UACI, key sensitivity, wrong-key decryption, noise contamination, cropping, ciphertext perturbation, and NIST SP 800-22 statistical randomness tests. The NIST analysis is performed independently on the x -, y -, and z -components of the HC-3D sequences and on the final RGB ciphertext. The experimental results demonstrate near-uniform ciphertext distributions, entropy values close to 8 bits, low adjacent-pixel correlations, and NPCR and UACI values close to their expected ideal ranges. The proposed framework integrates the HC-3D dynamics with session-dependent permutation, keystream generation, and cross-channel diffusion to provide a unified approach for RGB image encryption.