Judgmental and Statistical DIF Analyses of the PISA-2003 Mathematics Literacy Items

Comparisons of human characteristics across different language groups and cultures become more important in today's educational assessment practices as evidenced by the increasing interest in international comparative studies. Within this context, the fairness of the results across different language and cultural groups draws the attention of researchers. In the present study, the utility of hypothesis generated about differential item functioning (DIF) will be studied using the Program for International Student Assessment (PISA) 2003 data set. Items that were flagged as DIF through Restricted Factor Analysis (RFA), Mantel-Haenzsel (MH), and Item Response Theory Likelihood Ratio Analysis (IRT-LR) were simultaneously considered for content-wise evaluation. A group of judges reported that (1) cognitive skills measured by the items, (2) translation errors, and (3) use of quantitative words are three important sources of error that might be causing DIF in the PISA 2003 study. Results indicated that systematic a-priory analyses of the item content may help anticipate items with DIF in a cross-cultural comparison. Close agreement was obtained between the judges' opinions about DIF and statistical DIF analysis results.