The Errors of Our Ways

Errors don't exist in our data, but they serve a vital function. Reality is complicated, but our models need to be simple in order to be manageable. We assume that attributes are invariant over some conditions of observation, and once we do that we need some way of accounting for the variability in observed scores over these conditions of observation. We relegate this inconvenient variability to errors of measurement. The seriousness of errors of measurement depends on the intended interpretations and uses of the scores and the context in which they are used. Errors are too large if they interfere with the intended interpretations and uses, and otherwise are acceptable. The errors of measurement have to be small compared to the tolerance for error, and errors that are too large have to be controlled in some way. We have several ways of doing this. We can redefine the attribute of interest, we can standardize the assessments and leave the attribute alone, and/or we can sample the relevant performance domain more thoroughly. It is particularly important to control the larger sources of error. If a source of error (systematic or random) is small compared to the dominant sources of error for a testing procedure, it can generally be ignored.