Instrument Reporting Practices in Second Language Research

Second language (L2) researchers often have to develop or change the instruments they use to measure numerous constructs (Norris & Ortega, 2012). Given the prevalence of researcher‐developed and ‐adapted data collection instruments, and given the profound effect instrumentation can have on results, thorough reporting of instrumentation is important for interpreting study results, conducting replications, forming and modifying L2 theory, and performing meta‐analytic research. Transparency in instrument reporting should include information on the origins, development, piloting, and reliability of instruments as well as information about inter‐rater reliability. Prior research, however, indicates gaps in L2 research transparency (Cohen & Macaro, 2010; Plonsky & Gass, 2011). The present study examined instrument reporting practices in three L2 journals over a period of 5 years. Origins were reported for 58% of the instruments, descriptions of piloting were provided for 17% of the instruments, reliability coefficients were reported for 28% of the instruments (with Cronbach's alpha being the most frequently used index: 22%), and inter‐rater reliability information was provided for 42% of the instruments (with percentage agreement being the most frequently used calculation: 16%). Seventeen percent of the instruments were made available either online or in an appendix. This snapshot of current reporting practices indicates areas where transparency could be improved.