The Generalizability of Systematic Direct Observations Across Time and Setting: A Preliminary Investigation of the Psychometrics of Behavioral Observation

This study examined the generalizability of systematic direct observation across setting and time. Participants included 14 students from an intact inclusionary fifth grade classroom. On-task/off-task behavior was directly observed using momentary time-sampling recording, twice a day, for 10 school days. Using Generalizability (G) theory, results suggested that of the total amount of variance present, individual variation among participants accounted for 62% of the observed variance and unexplained error 24%. Although adequate levels of interobserver agreement were attained, resulting reliability coefficients of the observational procedures were in the low.60s. Results are discussed in terms of both behavioral and classic measurement theory and implications for practice.