Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction

In this article, multifaceted Rasch techniques are used to investigate inter- and intrarater reliability in the assessment of two alternate versions of an oral interaction test - a direct version and a semi-direct version. Following the rating of trial administration of both versions of the test, an analysis of the bias of each individual rater in respect to test type and particular criteria used in rating the test was undertaken. The results of these analyses were then presented to the rater. The raters then assessed a further administration of the test and their bias with respect to this administration was analysed. The results of the two bias analyses were compared to determine whether rater performance had improved as a result of the feedback. There was some evidence that performance had improved.