The Trap Of Random Sampling and How to Avoid It - Alternative Sampling Strategies for a Realistic Estimate of the Generalization Error in Remote Sensing

Many benchmark datasets in remote sensing still consist of only a single image that is used to train and evaluate machine learning algorithms. Random sampling - while being the virtual standard in the field to divide the available data into train and test set - is the worst choice in this scenario as it leads to a large degree of spatial correlation between these two sets which are assumed to be independent. Already pointed out in several earlier works, this paper reconfirms this fact and compares several alternatives. Two of the evaluated techniques allow to draw a large amount of unbiased test samples and lead to significantly less biased error estimates. The urgent recommendation is using multi-image datasets or - if only a single image is available - refraining to use random sampling and instead rely on either of these or similar methods.