Better than best low-rank approximation with the singular value decomposition

The Eckhart-Young theorem states that the best low-rank approximation of a matrix can be constructed from the leading singular values and vectors of the matrix. Here, we illustrate that the practical implications of this result crucially depend on the organization of the matrix data. In particular, we will show examples where a rank 2 approximation of the matrix data in a different representation more accurately represents the entire matrix than a rank 5 approximation of the original matrix data -- even though both approximations have the same number of underlying parameters. Beyond images, we show examples of how flexible orientation enables better approximation of time series data, which suggests additional applicability of the findings. Finally, we conclude with a theoretical result that the effect of data organization can result in an unbounded improvement to the matrix approximation factor as the matrix dimension grows.