Coupled improved nested array and correntropy operator for robust DOA estimation under impulsive noise

Reliable underwater direction of arrival (DOA) estimation is challenged by impulsive noise and the limited aperture of compact hydrophone arrays. To address these issues, this paper develops two correntropy based operators and an improved nested array (INA). A polynomial correntropy operator (PCO) and an absolute polynomial correntropy operator (APCO) are proposed for robust covariance construction. PCO is proved bounded in the real valued case, while APCO extends boundedness to complex-valued hydrophone observations under symmetric 𝛼 stable (S 𝛼 S) impulsive noise without requiring prior knowledge of the characteristic exponent. The proposed operators suppress large amplitude outliers and enable robust subspace processing. The INA is obtained by splitting the dense subarray of a prototype nested array and relocating one sensor. Without adding sensors, it yields a hole free difference coarray, increases uniform degrees of freedom, and reduces coupling leakage. The operators and INA are combined with difference coarray processing and spatial smoothing multiple signal classification (MUSIC). Simulations under different generalized signal to noise ratios (GSNR), snapshot numbers, characteristic exponents, and coupling conditions demonstrate improved accuracy and robustness, especially under strong impulsive noise and low GSNR conditions.