What Is a Good Forecast? An Essay on the Nature of Goodness in Weather Forecasting

Differences of opinion exist among forecasters—and between forecasters and users—regarding the meaning of the phrase “good (bad) weather forecasts.” These differences of opinion are fueled by a lack of clarity and/or understanding concerning the nature of goodness in weather forecasting. This lack of clarity and understanding complicates the processes of formulating and evaluating weather forecasts and undermines their ultimate usefulness. Three distinct types of goodness are identified in this paper: 1) the correspondence between forecasters’ judgments and their forecasts (type 1 goodness, or consistency), 2) the correspondence between the forecasts and the matching observations (type 2 goodness, or quality), and 3) the incremental economic and/or other benefits realized by decision makers through the use of the forecasts (type 3 goodness, or value). Each type of goodness is defined and described in some detail. In addition, issues related to the measurement of consistency, quality, and value are discussed. Relationships among the three types of goodness are also considered. It is shown by example that the level of consistency directly impacts the levels of both quality and value. Moreover, recent studies of quality/value relationships have revealed that these relationships are inherently nonlinear and may not be monotonic unless the multifaceted nature of quality is respected. Some implications of these considerations for various practices related to operational forecasting are discussed. Changes in these practices that could enhance the goodness of weather forecasts in one or more respects are identified.