Scientific Inquiry Precision, Reliability, and Validity: Essential Elements of Measurement in Nursing Research

Janet Houser Column Editor: Anne Marie Kotzer Scientific Inquiry provides a forum to facilitate the ongoing process of questioning and evaluating practice, presents informed practice based on available data, and innovates new practices through research and experimental learning. Quantitative research depends on identifying and measuring the right things. The process of measurement allows the researcher to determine if and in what quantity a characteristic is present. If the measurement is not correct, then the researcher may draw the wrong conclusion. There are many forms of measurement. Some measures involve sorting subjects into categories; others quantify an objective characteristic. For example, using a thermometer measures the amount of heat in the body, whereas asking about religious affiliation seeks to categorize subjects based on their beliefs. Both types are frequently used in nursing practice. Regardless of the type of measure, the researcher is interested in representing the characteristics of the subject accurately and consistently. The desirable characteristics of a measure are precision, reliability, and validity (Waltz, Strickland, & Lenz, 2004). All three are important in drawing accurate conclusions about the credibility of a research study. When a measure is reproducible, that is, when it generates a consistently accurate value every time it is used, it is considered precise. The reader can have confidence that differences between individuals are not explained by differences in the measurement instrument. Both reproducibility and accuracy must be present for a measure to be precise. For example, a tape measure is often used to measure the length of a neonate. A tape measure that has been used repeatedly may become stretched. While it may be a consistent measure of neonatal length, it is not an accurate one; it will consistently underestimate the neonate's length. Likewise, using a metal yardstick yields an accurate result, but it is unlikely that the nurse will have a quiet baby lying on the instrument each time it is used. While the metal yardstick might be accurate, results are not reproducible. Both characteristics—reproducibility and accuracy—are necessary to describe a measure as precise. The most common way to assure precision is through calibration of instrumentation. Calibration is accomplished by measuring a known quantity of a variable and determining if the instrument generates identical values. Keeping instruments in top working condition and well calibrated minimizes measurement error. But not all instruments are easily calibrated. For example, how do you calibrate an instrument to measure quality of life? In the case of instruments that measure more subjective characteristics, the precision concern is replaced with an emphasis on reliability and validity. An instrument is reliable if characteristics that are detected among subjects are attributable to true differences and not to differences within the instrument or among those administering the measure. Tests of reliability document the degree to which an instrument is stable among individuals, between raters, and over time. Stability within an instrument is called internal reliability and is measured with the alpha coefficient statistic. This coefficient may be called Cronbach's alpha, coefficient alpha, or internal reliability, and should have a value of .7 or greater. The coefficient alpha represents the extent to which individual items represent the variability in the overall instrument. This indicates that changes from item to item represent real changes in the subject and not changes in the way the questions are worded or interpreted. The coefficient alpha has an additional use: It allows the reader to calculate the amount of measurement error in the instrument. One minus the coefficient alpha quantifies error for the sample using the instrument. For example, if coefficient alpha were .92, then measurement error is equal to 1 – .92, or 8%. Minimal measurement error translates into a stronger study. It is desirable, then, to use an instrument that has as large a coefficient alpha as possible. Stability among subjects is measured by an item-total correlation, which should have a positive sign and an absolute value close to .5. An item-total correlation represents whether performance on a single item is consistent with the individual's performance on all items. For example, a student who scores high on an overall knowledge quiz would be expected to be correct on individual items. The item-total correlation is an important statistic when the instrument asks for sensitive information about the subject or when respondent truthfulness on individual items may be of concern. Stability among raters is measured as interrater reliability. Documentation of interrater reliability is needed when multiple raters observe and record a variable. For example, the degree of agreement between two or more nurses who are assigning an Apgar score should be high if the measure has strong interrater reliability. An agreement statistic can be interpreted like a percent; a value of .85 or greater is considered acceptable (Sim & Wright, 2005). Stability over time is quantified by a test–retest correlation coefficient. Stability over time becomes a particular concern when a repeated-measures or pretest/posttest design is used. The researcher must be confident that any differences detected between time periods are due to real changes in the subject, and not to instability of the instrument. Test–retest correlation is accomplished by administering an instrument, waiting a reasonable period of time, and then readministering the instrument. A correlation coefficient is then calculated between the two sets of item scores; a correlation of .5 or greater is desirable (Shultz & Whitney, 2005). At least one test of reliability should be performed and reported for each instrument in a study, whether the instrument has a long track record or is developed specifically for a single study. The selection of an appropriate reliability test depends on the way the instrument will be used in the specific study at hand (Streiner & Norman, 2003). An instrument has to be consistent and precise, but a measure can consistently and precisely measure the wrong thing. An instrument is valid if it accurately represents the underlying characteristic of interest. For example, measuring the length of a neonate can be difficult. A squirming baby is measured from the tip of the heel to the crown of the head—not exactly an exact description. If three people were asked to point to the crown of the head, it is likely that three different locations would be indicated. Measuring over the head to the tip of the nose would be more reliable—it is easy to find the end of the nose—but it would not be an accurate representation of the baby's length. Reliability tells us an instrument will be consistent; validity tells us the instrument will consistently measure the right thing. Reliability constrains but does not assure validity. An instrument cannot be more valid than it is reliable. On the other hand, simply possessing reliability does not ensure validity; separate tests of each are required to draw a comprehensive conclusion about the usefulness of an instrument. It is common to see reliability statistics reported without any mention of tests of validity. This is likely due to the fact that validity is much harder to test than reliability. Assuring that an instrument truly represents an abstract characteristic is one of the most difficult challenges in measurement. Consider the commonly used pain scale of 10 faces, with the lowest end a smile and the upper end a frown and tears. The instrument is intended to represent the quantity and nature of pain reported by pediatric patients. Is it possible, though, that some patients may interpret the crying face as "sadness" instead of pain? Assuring that a measure represents an underlying attribute credibly is the challenge of validity testing. Even with abstract characteristics, there are methods a researcher can use to document validity. One or all types may be used in the same study. Content validity involves a subjective judgment about whether a measurement makes sense. Content validity can mean that face validity has been assessed ("this instrument looks like it should measure pain") or that a panel of experts has verified that the correct concepts are included in the measure. No statistics are reported for content validity, although the credentials of the experts who provide it should be described. Construct validity indicates that a measurement captures the conceptual basis for the variable. This is, as it sounds, very difficult—but extremely valuable in assuring that measures represent reality. A common method of construct validation is factor analysis. Factor analysis groups items within the instrument according to their shared variability (Pett, Lackey, & Sullivan, 2003). The researcher can then review the factor groups and determine if they do, indeed, represent the conceptual basis of the instrument. Criterion-related validity is the correlation of the instrument to some external manifestation of the characteristic. There are several ways to measure criterion-related validity. Concurrent validity is present when an instrument reflects actual performance. An example might be when the reading from a tympanic thermometer is correlated with the reading from a rectal thermometer. Predictive validity indicates that a measure can predict future performance. An example might be an instrument that can predict the risk of postnatal depression. Discriminate validity demonstrates the capacity to differentiate those who have a characteristic from those who do not. For example, an instrument that could detect hyperbilirubinemia noninvasively possesses discriminate validity if it can successfully classify neonates as low-risk or high-risk for kernicterus. All of these considerations, taken together, make up the psychometric characteristics of an instrument. An instrument with acceptable psychometric properties has been tested for and attained a level of reliability and validity acceptable for a research study. It is common, though, to find studies in which instruments have been used that have minimal or borderline psychometrics. This may not automatically translate into a weaker study. There are many research considerations in addition to psychometric properties. Feasibility of administration, costs of instruments, and subject burden also drive the selection of an instrument. Documentation of the precision, reliability, and validity of a measurement tool increases the value of the results as evidence for practice. Doing so enhances confidence that results and conclusions are correctly drawn and assures the reader that differences among subjects were, indeed, true differences and not due to weaknesses in the instrumentation.