Reduced space optimal analysis for historical data sets: 136 years of Atlantic sea surface temperatures

A computationally efficient method for analyzing meteorological and oceanographic historical data sets has been developed. The method combines data reduction and least squares optimal estimation. The data reduction involves computing empirical orthogonal functions (EOFs) of the data based on their recent, high‐quality portion and using a leading EOF subset as a basis for the analyzed solution and for fitting a first‐order linear model of time transitions. We then formulate optimal estimation problems in terms of the EOF projection of the analyzed field to obtain reduced space analogues of the optimal smoother, the Kalman filter, and optimal interpolation techniques. All reduced space algorithms are far cheaper computationally than their full grid prototypes, while their solutions are not necessarily inferior since the sparsity and error in available data often make estimation of small‐scale features meaningless. Where covariance patterns can be estimated from the available data, the analysis methods fill gaps, correct sampling errors, and produce spatially and temporally coherent analyzed data sets. As with classical least squares estimation, the reduced space versions also provide theoretical error estimates for analyzed values. The methods are demonstrated on Atlantic monthly sea surface temperature (SST) anomalies for 1856–1991 from the United Kingdom Meteorological Office historical sea surface temperature data set (version MOHSST5). Choice of a reduced space dimension of 30 is shown to be adequate. The analyses are tested by withholding a significant part of the data and prove to be robust and in agreement with their own error estimates; they are also consistent with a partially independent optimal interpolation (OI) analysis by Reynolds and Smith 1994 produced in the National Centers for Environmental Prediction (NCEP)(known as the NCEP OI analysis). A simple statistical model is used to depict the month‐to‐month SST evolution in the optimal smoother algorithm. Results are somewhat superior to both the Kalman filter, which relies less on the model, and the optimal interpolation, which does not use it at all. The method generalizes a few recent works on using a reduced space for data set analyses. Difficulties of methods which simply fit EOF patterns to observed data are pointed out, and the more complete analysis procedures developed here are suggested as a remedy.