A Producer’s Guide to Reference Data (Training and Validation Data) in Leading Geographic Information System Applications

Leading geographic information system (GIS) applications offer producers streamlined tools for conducting digital image classifications and subsequent thematic map accuracy assessments. Such tools are used daily by countless practitioners for critical analyses of land use and cover and other thematic maps due to their ease of implementation and increasingly available remotely sensed imagery. Despite a robust history of discussion and research surrounding the sources of error that could be introduced during such practices, as well as the careful considerations one should undertake when carrying out such methods, many analyses are still performed using invalid or inappropriate methodologies. These issues largely stem from the GIS software applications considering the reference data as points (x, y coordinates) and not as imagery (pixels). In this paper, three key characteristics and their issues in how reference data are handled by leading GIS applications are addressed. These characteristics include first the sample unit (i.e., geometry), in which the trade-offs and limitations of individual reference data sample geometry types are documented. Second is the reference data sample size, which focuses on how default software settings could lead to improper subsampling (i.e., counting/creating multiple samples from a single producer-defined reference data site). Third is the sampling distribution, which relates to the controlled distribution of individual training and validation samples within and among the given strata (map classes). The concluding section provides several recommendations to future producers (practitioners) based on a confluence of evidence gathered while testing these specific characteristics using leading GIS applications.