Adaptive detection of second-order subspace signals: derivation and characterization of the maximal invariant statistic

We address the problem of detecting a multidimensional subspace signal among colored noise with unknown covariance matrix using a second-order model, i.e. assuming that the presence of the signal of interest results in a rank- R modification of the noise covariance matrix. The problem is tackled through the prism of invariance. We derive the group of transformations that leave the problem invariant and subsequently derive the maximal invariant statistic (MIS) along with the induced maximal invariant (IMI). A statistical representation of the MIS is obtained which is quite different from that of a first-order model. Hints at possible test statistics based on the MIS are also provided.