Dynamic antenna selection for hybrid beamforming in ISAC systems: a joint transceiver optimization framework

Integrated sensing and communication (ISAC) is a key enabler for 6G networks, yet its performance is critically constrained by the trade-off between hardware complexity and system flexibility. Conventional hybrid beamforming (HBF) architectures suffer from rigid antenna connections, limiting their adaptability in dynamic environments such as unmanned aerial vehicle (UAV) networks. To overcome this, we propose a dynamic antenna selection (AS) frame-work for HBF-based ISAC systems with joint transceiver optimization. The proposed architecture adaptively recon-figures both transmit and receive antenna subsets, forming a closed-loop system where high-precision sensing continuously guides robust communication. A weighted objective function is formulated to maximize both spectral efficiency and sensing accuracy (characterized by the Cram´er–Rao bound). To solve this mixed-integer non-convex problem, we introduce a Gumbel-Softmax relaxation that transforms the combinatorial AS task into a differentiable form, enabling gradient-based end-to-end optimization within an alternating optimization framework. Furthermore, an iterative adaptive approach (IAA) is employed for robust single-snapshot angle estimation under the dynamically formed non-uniform receive array. Simulation results demonstrate that the proposed joint transceiver optimization significantly outperforms both transmitter-only and full-array baselines, achieving a superior communication-sensing trade-off and establishing a new Pareto frontier for AS-aided ISAC systems.