An efficient generalized sphere decoder for rank-deficient MIMO systems

We derive a generalized sphere decoder (GSD) for rank-deficient multiple input multiple output (MIMO) systems using N transmit antennas and M receive antennas. This problem arises when N>M or when the channel gains are strongly correlated. The upper triangular factorization of the Grammian yields an under-determined system and the standard sphere decoding (SD) fails. For constant modulus constellations, we modify the maximum likelihood (ML) cost metric so that the equivalent Grammian is rank N. The resulting GSD algorithm has significantly lower complexity than previous algorithms. A method to handle nonconstant modulus constellations is also developed.