Item Orientation as a Confound in Epistemology Scales

Item orientation—whether agreement with a survey item indicates higher (positive) or lower (negative) standing on a construct—can influence the factor structure of psychological assessments in education research. This study investigated how item orientation affected inferences about the factor structure of students’ epistemology in history because previous studies consistently measured factors of epistemology with uniformly oriented items within factors and polarized item orientation across factors. We administered a survey comprised of positively and negatively oriented items for three previously hypothesized dimensions of student epistemology—complex, tentative, and integration beliefs. We used confirmatory factor analysis to examine how different approaches to handling item orientation affected conclusions about the factor structure of epistemology. First, we replicated the factor structure of epistemology in prior research using a subset of uniformly oriented items within factors and polarized item orientation across factors, finding similar fit and small inter-factor associations as seen in previous research. Second, we introduced items of mixed orientation to measure each factor and found that model fit became poor. Third, we simultaneously modeled substantive and method factors, resulting in improved model fit. The method factor explained more variance than substantive factors, indicating item orientation was a significant confound. When controlling for a method effect (i.e., item orientation), associations among epistemology factors were large and positive, suggesting that previous findings of roughly orthogonal epistemology factors may result from offsetting negatively associated method factors and positively associated substantive factors.