Under-Appreciated Steps in Instrument Development, Part II: Ending With Validity

Even after an elaborate factor analysis or other evidence of its conceptually strong internal structure, external validity testing of a new measurement tool is needed before a reader can actually use an instrument with confidence. In this usage, external validity is not meant as generalizability to the population at large but as evidence that an instrument's measurement functions in expected ways in relation to other scores or characteristics. With a little advance planning, the same sample used to identify factor structure and assess internal consistency also can be used to estimate external validity. This is the second of two editorials on under-used steps in instrument validation, written in hopes of increasing the number of fully developed instrument development reports received at RINAH. As noted in the first part, none of these ideas is original; many of these points have been made in this journal (Froman Higgins Ferketich, Figuerdo, failure to obtain prescription refills; or even worsening trends in biophysical measures such as blood pressure or weight as a result of these behaviors may be useful correlates. The consequences of others' use of an instrument once it is made public—how users actually respond to and interpret the measurement results—can reveal important characteristics of its real-world validity (Cook & Beckman, 2006). For example, a cut-off score on the instrument may be used to identify respondents a risk of a clinical concern, although use in this manner may have been far from the developer's original intention and not yet estimated systematically. The use of a score as an indicator of success or recovery likewise must be evaluated carefully. When an instrument is developed with a particular target group in mind, use and interpretation in other groups should be subject to close examination. In general, researchers who publish in RINAH are attentive to these concerns, but we encourage continued close attention to how and why an instrument was developed before adopting it for a new purpose or context. Researchers will want to follow closely any later reports of others' use of their instruments and publish corrections of others' misunderstandings or inappropriate uses if needed. If, for example, my discouragement measure were used by healthcare providers to identify those in need of psychological support, this practice might be helpful to patients, but it would not be based on any claim I had made about the measure unless I had conducted analyses to determine a clinically meaningful cut-off score. Some consequences of untested clinical use might even be harmful, such as if a patient's high discouragement score led health care providers to reduce efforts to engage that patient in treatment. When I published the results of my instrument validation study, I would be sure to include cautions about what is and is not known about its relevance in patient care. No instrument is reliable or valid once and for all. Validity and reliability can only be estimated in a specific group at a given time, and an instrument may function very differently under new conditions. In these two editorials, I have argued that going beyond experts' assumptions on content validity and item clarity when developing items, and testing how an instrument functions in relation to other variables when testing its internal reliability and validity, add to the trustworthiness of a measure and enhance its readiness for use. We at RINAH aim to publish strong building blocks for clinical research, whether empirical findings, innovative methods, or tools for measurement of important concepts. I welcome submissions of instrument development papers that move us toward this goal.