The Meaning of Validity

Frequently, authors refer to the instruments or measures that they use in their research studies as “validated,” “proven to be valid,” “having established validity,” and the like. For example, consider the terms “valid questionnaire” and “validated questionnaire.” Phrases and terms such as these are misleading and inappropriate, since they suggest that validity is some set characteristic of a questionnaire or other type of measure. Validity is not a property of an instrument or a measure. Rather, it is a property of the scores or data that are obtained when one uses a particular instrument with a particular group of subjects in a particular setting and under certain conditions. The practice, however, of treating both validity and reliability as though they were static or absolute properties of instruments is so common that it has been labeled as a measurement “myth” or “misconception” (1). My purpose in this article is to review the meaning of validity, the changes that have occurred in the definition of validity over the past 60 years, and then to suggest some appropriate ways to present validity information in manuscripts. What is Validity? The meaning of validity has undergone a “metamorphosis” during the past 60 years (2). Early definitions of validity characterized it as a static property of a test or measure. This view of validity was epitomized by Guilford's statement in 1946 that, “in a very general sense, a test is valid for anything with which it correlates” (3). The prevailing view was fairly black-and-white: a test was either valid or not. Between 1950 and the late 1970s, the definition of validity focused on use. In the first edition of the Standards for Educational and Psychological Tests and Manuals, published in 1966 (4), the extent to which a test yielded information useful for a specific purpose was emphasized. Validity was described as comprised of three types: content validity, criterion-related validity (which includes both predictive and concurrent validity), and construct validity. Categorizing validity into these three discrete types was useful for many years. However, it increasingly grew to be a burden. As several measurement specialists have noted in recent years (5,6), the “trinity” view of validity (content-criterion-construct) is problematic for several reasons. It compartmentalizes our thinking about validity, it is incomplete, and the various “types” of validity are not equal. The prevailing definition of validity that emerged during the 1980s focused on the inferences made from test scores. According to the 1985 Standards for Educational and Psychological Testing (7), validity “refers to the appropriateness, meaningfulness, and usefulness of the specific inferences made from test scores. Test validation is the process of accumulating evidence to support such inferences.” A debate started at this time about the social consequences of testing as part of validity (which remains a fairly contentious issue today), and there also was increasing criticism of the tripartite view of validity. Construct validity began to be seen as the unifying type or theme—almost an “umbrella” under which other types of validity could be subsumed. More and more, psychometricians called for the accumulation of several types of validity evidence to support the inferences made from scores or data. Current “best thinking” about the meaning of validity is exemplified by the recently revised Standards for Educational and Psychological Testing (8). Although the definition of validity continues to emphasize interpretations of scores—it is defined as “the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests”—a number of changes in the treatment of validity are noteworthy. One key change is that the breakdown of validity into discrete types of validity has been replaced with a discussion of the various types of validity evidence. While these align quite closely with the types of evidence that could be collected for each of the old types of validity (6), the emphasis has switched to validity as a unitary concept (i.e., construct validity). The definition of the term “construct” has been broadened (a characteristic or concept that a test or other measurement procedure is designed to measure), and five types of validity evidence are described. These are: evidence based on test content; evidence based on response processes; evidence based on internal structure; evidence based on relations to other variables; and evidence based on the consequences of testing. The complex nature of validity, and the difficult and time-consuming nature of validation efforts, was summarized well by Kane (9): “ . . . validity is an integrated, or unified, evaluation of the interpretation [to be made from scores or data]. It is not simply a collection of techniques or tools.' Again, it is important to emphasize that it is the interpretations of scores required by proposed uses of the measure that are evaluated, not the measure itself. Validity is of concern for all types of measures and measurement procedures: formal tests, observation techniques, interview protocols, questionnaires, self-report affective measures, projective devices, and so on. Validation requires an extensive collection of data (across many studies) and analyses of evidence based on explicit expectations about the proposed interpretations or inferences that will be made from the scores or data collected. Consistent with the notion of different types of validity evidence, there are a wide array of possible methodological and statistical approaches to the estimation of validity. These include expert review, logical analyses, correlational studies, descriptive studies, longitudinal studies, experimental studies, “known-group” comparison studies, factor analytical studies, and differential item functioning studies (6). Appropriate Ways to Present Validity Evidence The following suggestions pertain to the interpretation and description of validity evidence in manuscripts and presentations. The language used is important. Describe validity as a property of data or test scores that has been estimated, not “proven” or “established.” Avoid describing validity as though it were a property of a test or measure and, especially, avoid describing it as though it were a static property. For example, do not use the terms “valid questionnaire” or “validated questionnaire.” Instead, describe the steps taken to obtain validity evidence—such as evidence based on test content or evidence based on internal structure, which are often relevant for questionnaires—and the results. Describe all extant types of evidence that support the validity of score interpretations. Be clear about the proposed interpretations as well as the supporting evidence. If possible, provide details about the parameters of empirical validation studies, including subjects, setting, conditions, and results. When reporting the results of correlational validity studies, focus on the size of the correlations rather than on the p values or levels of statistical significance. When reporting the results of expert reviews, provide details about the characteristics of the experts, the tasks that they were asked to complete, and the results. When reporting the results of factor analytical validity studies, do not “overinterpret” the findings. Factor analysis is an appropriate technique for obtaining evidence about the internal structure of a measure; however, it alone does not yield sufficient information about the “whole” of construct validity (10). When appropriate, acknowledge that the study of the validity of score interpretations for many measures is complex, multifaceted, and never-ending. Validity and reliability are separate psychometric properties, although they are related to each other in a classic way: reliability is necessary, but not sufficient, for validity. Avoid intermingling the terms that describe types of validity evidence with types of reliability evidence. For example, “internal consistency, “stability,” “interobserver” or “interrater,” and “equivalence” refer to types of reliability, not validity, evidence. If necessary, consult a measurement specialist (or psychometrician) to determine appropriate ways to collect empirical validity evidence and/or to describe the validity evidence in manuscripts or presentations. Since journal space is often limited, it is challenging to describe the psychometric evidence sufficiently and succinctly at the same time. Be thoughtful and reflective in summarizing the extant evidence, and choose words and phrases carefully.