Serial correlation considerations when assessing differences in prediction skill

When comparing two series of numerical weather prediction (NWP) skill scores, the serial dependence of the scores needs to be taken into account. The problem may be more generally stated as one of testing for the statistical significance of the difference between two samples, when serial dependence may be present within each sample. Difference may be defined by location (mean or median) when scores are quantitative, or may be defined as 'better' or 'worse' when the two series are considered as ranked pairs. A methodology is demonstrated that is applicable within the framework of a classical significance test, but the assumptions of that test are relaxed to the extent of allowing a population serial dependence of a specified form. A test statistic for comparing two series of skill scores is computed in the same way as in the classical test, but the statistic is then referred to as an extended distribution (possibly synthesised by Monte Carlo simulation) that depends upon the population serial correlation. The methodology produces a similar asymptotic result 'to that of the traditional 'effective sample size' approach for large samples, but it is also applicable to small samples and to non-normal populations. In the case of a NWP model with a long operational history, the population serial dependence may be well known from historical data. However, if the relevant population serial correlation is poorly known (as is often the case in practice) it is recommended that a result which would be statistically significant for serially independent data, be qualified by stating the level of the population serial correlation necessary to negate statistical significance. The application of the methodology to several well known parametric and non-parametric tests is presented, for the case where the serial dependence is first order autoregressive.