Small-sample error estimation: mythology versus mathematics

Error estimation is a key aspect of statistical pattern recognition. The true classification error rate is usually unavailable since it depends on the unknown feature-label distribution. Hence, one needs to estimate the error rate from the available sample data. This paper presents a concise, mathematically rigorous review of the subject of error estimation in statistical pattern recognition, pointing to the pitfalls that arise in small-sample settings due to the use of "rules of thumb" and a neglect for proper mathematical understanding of the problem.