A bound on performance for object detection and classification for machine learning

An objective, quantitative method for assessing image interpretability for machine learning (ML) would be a valuable tool to support sensor design, collection management, and algorithm selection. The National Imagery Interpretability Rating Scale (NIIRS) has served as a useful standard for image analysis in support of intelligence, surveillance, and reconnaissance (ISR) missions. However, NIIRS focuses on human perception and empirical studies have demonstrated a tenuous relationship, at best, between NIIRS and observed performance for ML algorithms. We propose a new approach that approximates the Bayes error for object classification to establish an upper bound on ML performance for a given set of imagery. The process starts with high fidelity signatures from the object classes of interest. Degrading these signatures through an emulation of the sensor's image chain produces signatures consistent with observed imagery from that sensor. Various distance metrics quantify the separability between specific object classes. We demonstrate a resampling technique to approximate the Bayes error, which is the theoretical limit for performance. This approach provides a quantitative measure that is independent of any specific machine learning model or methodology.