Evaluating density forecasts: forecast combinations, model mixtures, calibration and sharpness

This paper reviews current density forecast evaluation procedures, and considers a proposal that such procedures be augmented by an assessment of ‘sharpness’. This was motivated by an example in which some standard evaluation procedures using probability integral transforms cannot distinguish the ideal forecast from several competing forecasts. We show that this example has some unrealistic features from a time series forecasting perspective, and so provides insecure foundations for the argument that existing calibration procedures are inadequate in practice. Our alternative, more realistic example shows how relevant statistical methods, including information-based methods, provide the required discrimination between competing forecasts. We introduce a new test of density forecast efficiency. Copyright © 2010 John Wiley & Sons, Ltd.