Prediction Efficiency Skill Scores for Event Detection Analysis

Prediction efficiency (PE) is a skill score that compares the data‐model metric of mean square error against the variance of the observations (i.e., using the average of the observed values as the “reference model” in the general skill score formula). This has proven to be highly useful for analysis using “continuous metrics”—those data‐model comparison techniques that employ the exact values of the observations and model results—especially when comparing the new model's performance against an existing model. Another major grouping of data‐model comparisons is “event detection analysis” in which all observational and numerical values are converted into a binary yes‐no categorization of being in or out of “event state.” There is, however, no equivalent skill score to PE within the event detection toolkit. This study proposes two such options, one based on the proportion correct metric and another based on the critical success index metric. Like PE, these new skill scores use the observations as the reference model, which provides complete independence of the reference model from the new model. It is demonstrated, reevaluating two space weather data‐model comparison studies, that these skill scores provide context for model evaluation that is unique to other existing metrics and valuable for the assessment, especially with respect to comparing the new model's performance against an existing model. Reference models in existing skill scores are based in part on the new model's performance against the data, which leads to ambiguous interpretation for intermodel comparison usage.