On Least Squares with Insufficient Observations

Abstract The optimality of the method of least squares is reconsidered when multicollinearity is present. An analysis is presented of the relationship between estimability and identifiability. The criterion of best linear minimum bias is developed, and shown to be equivalent to that of best linear conditionally unbiased estimation subject to complementary (non-estimable) linear restrictions. Imposition of erroneous estimable linear restrictions is shown to lower variances of estimators if and only if it biases them. All these results rely heavily on the use of the generalized inverse of a matrix, for which a new proof of existence and uniqueness is presented from the viewpoint of duality in linear spaces. Finally, estimation by minimum mean square error is proposed, and this is shown to reduce to the least squares method when either (a) regression coefficients have infinite prior variances, or (b) least squares estimators have small sampling variances.