Spatial and Interannual Variability of the Reliability of Ensemble-Based Probabilistic Forecasts: Consequences for Calibration

Reliability is an essential attribute of the quality of probabilistic forecasts. It indicates the correspondence between a given probability, and the observed frequency of an event in the case this event is forecast with this probability. The variability of the reliability of ECMWF ensemble forecasts has been investigated. Probabilistic forecasts of a850-hPa temperature anomaly have been considered during four consecutive winter seasons. Reliability appears highly variable in space and time. A proper evaluation thus requires a local, seasonwise verification, in order to avoid an overestimation of the performance. On the other hand, stratification of the data is likely to lead to an underestimation of the performance due to ill-sampling, so that a compromise has to be found. Variations of model bias contribute for a major part to the interannual variability of the reliability. After bias correction, reliability is virtually constant during the considered period of 4 yr, despite two major changes that have been implemented in the ECMWF Ensemble Prediction System (EPS). Local variations of model bias contribute highly to the spatial variability of the reliability, but this contribution is only revealed when considering separate winter seasons. Due to sampling limitations, local calibration and/or local bias correction are unlikely to bring a large improvement of reliability in operational conditions. A good scheme might rather be a combination of domain bias correction and domain calibration. Because calibration and bias correction require large samples of data for computing statistics, and at the same time other, independent samples are needed for validation, the implementation of an operational scheme for improving local reliability might be an arduous challenge.