High Stakes Tests with Self-Selected Essay Questions: Addressing Issues of Fairness

This study investigates the effect of reporting the unadjusted raw scores in a high-stakes language exam when raters differ significantly in severity and self-selected questions differ significantly in difficulty. More sophisticated models, introducing meaningful facets and parameters, are successively used to investigate the characteristics of the dataset. The application of the Rasch models to the data showed that examinees could benefit significantly from being marked by lenient raters and by responding to less demanding essay questions. It was also shown that the third rater failed to adjust the raw scores in a way similar to the statistical adjustment by the Rasch models. The study discusses the consequences of reporting unadjusted raw scores with particular emphasis on issues of fairness.