Isolated “Low” Test Scores Are Often Normal and Valid

Dear Editor-in-Chief, The recent study from Walton and colleagues (1) focuses on an important and relevant topic in sport-related concussion management. Using baseline testing relies on the assumption that student-athletes extend adequate effort so that a clinician can be confident that their baseline scores represent true ability. Suboptimal effort is an important but difficult phenomenon to capture, and the authors excellently describe the clinical implications of a baseline score obtained from suboptimal effort. Unfortunately, I have significant concerns about the proposed methods for doing so and their underlying assumptions. The Walton et al. (1) criterion for suboptimal performance is that at least one of four separate ImPACT composite scores falls below the 16th percentile. They base this on the assumption that a single score <16th percentile is a “statistical abnormality” or represents impaired cognition. This rationale, however, is questionable. For example, the supporting reference from Iverson and Schatz (2) neither argues for nor demonstrates efficacy of a 16th percentile cutoff for questionable effort. Rather, Iverson and Schatz (2) state how frequently such scores are obtained in healthy populations. This highlights why it is significantly problematic that Walton et al. (1) have attempted to draw meaningful conclusions about effort and validity from one “low” score within a test battery (3–5). In fact, Iverson and Schatz’s (2) classification scheme describes individuals with one ImPACT score <16th percentile as “broadly normal.” The fact that Walton et al. (1) found a higher proportion of ADHD/LD-diagnosed individuals in their valid-but-invalid group only further supports the conclusion that the score is likely a legitimate representation of their cognitive skills, because one would expect ADHD/LD-diagnosed individuals to perform much lower than normative averages. Athletes in the study were retested either one or two more times at test intervals “no less than 7 d” from initial testing, but the overall range appeared to vary from 7 d to over 2 months later. Improved scores within previously questionable domains were interpreted as supporting the initial assumption of suboptimal effort. However, an improved score does not indicate relevant performance change per se, which requires consideration of the domain-specific reliable change indices. Although the authors do note that a higher than expected percentage of valid-but-invalid athletes improved beyond reliable change intervals, it is unclear what percentage of the improvements occurred within the domain of question versus another domain that was already >16th percentile at initial testing. Lastly, there are practical hurdles. The authors importantly note the discrepancy between their sample’s frequency of obtaining one score <16th percentile (15%) and the expected frequency of 40% (2). Because the 40% figure was obtained from a sample of almost 18,000 male and female collegiate athletes, it is reasonable to expect that figure is more broadly generalizable. Suggesting that clinicians should consider repeat testing 40% of their institution’s athletes based solely on their obtaining one score <16th percentile is an extreme commitment of financial and personnel resources. Such a recommendation should not be predicated upon the unsubstantiated assumption that an isolated “low” test score indicates invalid performance. The foundation of the arguments proposed in this study for capturing suboptimal effort in collegiate athletes is simply too uncertain. Normal performance variability should not be confused with suboptimal effort. Use of this nonvalidated definition for suboptimal effort is likely to create far more problems than it aims to solve. Breton M.