Accuracy statistics for judging soft classification

In the literature one can find different accuracy measures that are built from the error matrix. However, standard accuracy assessment, which is based on the error matrix, is incomplete when dealing with fuzzy sets or when errors do not have the same importance. In this paper, we propose an extension of the error concept for soft (or crisp) classification that will be able to extend standard accuracy measures (e.g., overall, producer's, user's or Kappa statistic) that can be used in any framework: errors with different importance, soft classifier and crisp reference data (expert) or with a fuzzy expert. In particular, a weighted measure is built that takes into account the preferences of the decision maker in order to differentiate some errors that must not be considered equal. Acknowledgments This research was supported by the Government of Spain, grant number TIN2006‐06190 and a grant from the Del Amo joint programme between Complutense University of Madrid and the University of California. We want to thank the anonymous referees for their valuable comments.