Entropy as a basis for comparing and blending forecasts

The use of Shannon's entropy for comparing forecasts, and for blending the forecasts into a single forecast with higher skill, is discussed. The basic idea is to examine the difference in the entropy of the error distributions. A particularly appealing approach is to decompose the entropy difference into a complete set of vectors in phase space that maximize or minimize the entropy difference. This decomposition identifies the patterns that the first model predicts with less error variance than the second, and those that the second model predicts with less error variance than the first. A ‘blended’ forecast with higher skill than either of the original forecasts can be constructed by an appropriate superposition of these patterns. In contrast to a linear regression correction, which minimizes only the total mean square error, the blended forecast predicts every pattern, and every local region, with less (or equal) error variance than either of the original forecasts. A filtering procedure is introduced to deal with the problem that, at long lead times, error variance fails to be an appropriate measure of skill. The above procedure is used to compare the 1 to 8‐day forecast skill of 500 hPa geopotential height predicted by the National Centers for Environmental Prediction (NCEP) reanalysis forecast model over a 40‐year period, against the forecast skill of a 200‐variable linear inverse model. The results show that the NCEP model predicts virtually every pattern with less error variance than the linear inverse model, provided that the pattern can be predicted with skill by at least one of the models. The best predicted patterns of each model (in an error variance sense) also have the longest decorrelation times, suggesting a strong relation between predictability and persistence.