Judging Air Quality Model Performance

Under the direction of the AMS Steering Committee for the EPA Cooperative Agreement on air quality modeling, a small group of scientists was convened to review and recommend procedures to evaluate the performance of air quality models. Particular attention was paid to the operational use of models in regulatory settings. A number of general recommendations resulted from the workshop. The usefulness of models to aid decision makers in air quality management was reiterated. Concerns about the absolute, rather than the statistical, nature of air quality standards were raised, with a recommendation to reformulate standards accordingly. Model performance evaluation was suggested on the basis of differences between observed and predicted concentrations. The bias (average), the variance (noise), and the gross variability (gross error) of such differences were the recommended performance measures. In addition, correlation measures calculated in time, space, and jointly between observation and predictions were recommended. These measures and suggestions on how to apply them are outlined. Some qualification on the use of these measures for application to point sources using limited data sets was made. The workshop did not recommend any specific numerical standards for acceptable model performance; there was insufficient information to do this. Rather, considerations for evaluating model performance using statistically constructed confidence intervals and comparison against EPA's currently used models were suggested. Finally, areas in need of further research were identified. These include development and description of performance measures; application, especially to point source models; analysis of meteorological and air quality data, both existing and newly developed; and evaluation of diffusion modeling techniques.