Reporting practices of rater reliability in interpreting research

The issue addressed in this study is the reporting practices of rater reliability in interpreting research (IR), given that the use of raters as a method of measurement is a commonplace in IR, and that little is known about to what extent and how rater reliability estimates (RREs) have been reported. Drawing upon 447 articles from 14 translation and interpreting journals (2004--2014), this mixed-methods study attempts to gain quantitative and qualitative insights into the reporting practices. Data analysis reveals that: 1) almost 90% of the articles that needed to report RREs failed to do so; 2) potential problems emerged from those articles that reported RREs: lack of distinction between rater consensus and consistency, underreporting, misinterpretation and misuse of RREs, and lack of justification for the use of rater-generated measurements for subsequent data analysis. These findings highlight an urgent need for increased author awareness of reporting appropriate RREs in IR.