Robust Hybrid Beamforming and Dynamic Subarray Design for Near-Field mmWave ISAC Systems Under Unknown Interference

This paper investigates a near-field millimeter-wave (mmWave) integrated sensing and communication (ISAC) system under unknown interference. A base station equipped with a partially connected dynamic subarray hybrid architecture serves a legitimate user while performing target-oriented transmit beampattern shaping. Unlike existing works that assume perfect interference knowledge, we characterize the unknown interference channels via a robust spatial covariance uncertainty model. To exploit spatial degrees of freedom for interference suppression, the user employs a fully connected hybrid receiver. We formulate a robust transmit power minimization problem subject to worst-case communication signal-to-interference-plus-noise ratio (SINR) and sensing beampattern constraints, alongside constant-modulus and dynamic subarray hardware constraints. To solve this highly non-convex mixed discrete–continuous problem, we propose a two-layer alternating optimization framework. The inner layer optimizes the continuous and phase-quantized beamformers using successive convex approximation, while the outer layer refines the binary subarray connections via a penalty-augmented local discrete search. Extensive simulations demonstrate that explicitly modeling worst-case uncertainties ensures reliable ISAC performance in adversarial environments, and the dynamic subarray architecture systematically outperforms conventional fixed topologies in power efficiency. Additional robustness and sensitivity analyses show that these gains are most pronounced when sufficient spatial degrees of freedom remain, whereas excessive antenna failures, unmodeled strong multipath, or covariance drift outside the uncertainty envelope can erode the communication and sensing margins.