The Relationship Between Computer Familiarity and Performance on Computer‐based Toefl Test Tasks

ABSTRACT The increasing use of computer‐based testing raises concerns about equity and bias. Specifically, many in the field of language testing are concerned that the introduction of a computer‐based TOEFL test in 1998 will confound language proficiency with computer proficiency and thus bring construct‐irrelevant variance to the measurement of examinees' English‐language abilities. In a Phase I study (Kirsch, Jamieson, Taylor, & Eignor, 1998), TOEFL examinees were surveyed regarding their computer familiarity and classified into one of three computer familiarity groups: low, moderate, and high. In this study, Phase II, more than 1,100 “low‐computer‐familiar” and “high‐computer‐familiar” examinees from 12 international sites were identified from the Phase I survey and administered a computer tutorial and a set of 60 computer‐based TOEFL test items. The relationship between level of computer familiarity and performance on the computer‐based items was then examined. The examinees in Phase II were largely representative of those in Phase I, who were representative of the general TOEFL test‐taking population. Thus, results from this phase of the study are considered generalizable to the current TOEFL examinee population. The effect of computer familiarity after adjustments for language ability was examined by performing a series of analyses of covariance (ANCOVAs), using TOEFL paper‐and‐pencil test score as the covariate. These analyses were followed by a series of ANCOVAs involving the computer familiarity variable and a number of other variables: gender, reason for taking the TOEFL test, times the TOEFL test had been taken, and location where the TOEFL test was taken. In a final set of analyses, the TOEFL paper‐and‐pencil test scores of the low‐ and high‐computer‐familiar examinees were weighted such that the groups had identical distributions on the covariate. After controlling for language ability, the researchers found no meaningful relationship between level of computer familiarity and level of performance on computerized language tasks among TOEFL examinees who had completed the computer tutorial. This finding was consistent for all but one of the subgroups considered. A small but practically significant interaction between computer familiarity and reason for taking the test was found on the set of computerized reading items. Researchers concluded that there was no evidence of adverse effects on the computer‐based TOEFL performance due to lack of prior computer experience.