Array processing with compressive measurements and coherent signals

Array signal processing uses multiple, spatially separated sensors for signal enhancement, localization, and interference cancellation. To achieve these, techniques such as direction of arrival (DOA) estimation, beamforming, and multiple-input multiple-output (MIMO) processing are widely used in radar, sonar, wireless communications, ultrasound, and radio astronomy. This dissertation addresses two intertwined research goals: optimization of the compressive measurement matrix (CMM) in massive MIMO systems and DOA estimation of coherent signals. Massive MIMO technology, characterized by the use of a large number of antennas, is a key enabler for next-generation wireless communication and radar sensing. Despite its potential, implementing a massive MIMO system poses numerous technical challenges, such as high hardware complexity, increased cost, and elevated power consumption due to the large number of antennas and associated radio frequency (RF) front-end chains. One solution to these challenges is the use of hybrid beamforming, which divides the transceiving process into analog and digital domains. To perform hybrid beamforming efficiently, it is necessary to optimize the CMM, which projects high-dimensional signals onto a low-dimensional manifold, allowing to implement the massive MIMO system with a reduced number of RF chains. This dissertation presents several approaches to optimize the CMM. In the first approach, a deep learning framework based on a fully connected deep neural network (FCDNN) is developed to optimize the CMM in a massive MIMO system. This approach offers a computationally efficient solution while enhancing robustness and generalizability. The second approach also employs a deep learning framework, but it is based on a long short-term memory (LSTM) network. Unlike the first approach, it accounts for temporal dependencies between adjacent snapshots of received data to sequentially optimize the CMM. Preserving time-dependent information enables more effective optimization in each time snapshot, leading to faster convergence and improved DOA estimation performance. Both approaches consider the fully connected hybrid beamforming (FHBF) configuration. The third approach explores a more hardware-efficient architecture, namely non-overlapping subarray-based hybrid beamforming (NSHBF), in which the array is partitioned into multiple identical non-overlapping subarrays. The CMM is optimized for one subarray and based on identical subarray partitioning, applied to the remaining subarrays. This strategy offers further computational efficiency compared to the FHBF counterparts. The second focus of this dissertation is the DOA estimation of coherent signals. Most of the popular DOA estimation methods, such as subspace-based methods, are designed for uncorrelated signals. In many real-world applications, however, coherent signals are commonly encountered, and popular subspace-based algorithms often fail to detect them due to the rank-deficiency of the covariance matrix. This dissertation considers several approaches to address coherent signal detection. First, mixed coherent and uncorrelated signals in a uniform linear array (ULA) are considered, and a technique is developed to decorrelate the covariance matrix by utilizing multiple Toeplitz matrices, each of which is constructed by one row and one column of the original matrix. Next, detection of mixed coherent and uncorrelated signals in a two-dimensional (2D) sparse array is addressed by applying a structured tensor reconstruction method to decorrelate the covariance tensor, followed by canonical polyadic decomposition (CPD) for signal DOA estimation. We then develop a general framework for 2D DOA estimation of coherent signals in a uniform rectangular array (URA), with a detailed study of the tradeoffs among the achievable number of degrees of freedom (DOFs), DOA estimation accuracy, and computational complexity. In this framework, we first develop a forward-backward tensor reconstruction (FBTR) strategy, which provides larger DOFs than existing tensor reconstruction methods. We further develop an optimized sensor placement strategy for the URA to maximize the number of DOFs for a given number of antennas, and derive closed form expressions for the achievable DOFs of both the FBTR and forward-backward spatial smoothing (FBSS) approaches under the proposed sensor placement. Next, a moving URA-based coherent signal detection strategy is developed, which provides improved DOA estimation performance and higher DOFs than the static array. The minimum number of movements, minimum movement step size, and motion support are analyzed for effective decorrelation of coherent signals. Finally, to bridge the two focuses of this dissertation, coherent signal detection under a hybrid beamforming scheme is investigated, and a overlapping subarray-based hybrid beamforming architecture (OSHBF) is developed to enable effective coherent signal detection in hybrid beamforming systems.