Confusing Claims for Data: A Critique of Common Practices for Presenting Qualitative Research on Learning

We question widely accepted practices of publishing articles that present quantified analyses of qualitative data. First, articles are often published that provide only very brief excerpts of the qualitative data themselves to illustrate the coding scheme, tacitly or explicitly treating the coding results as data. Second, articles are often published that treat interrater reliability solely as a matter of justifying the coding scheme, without further attention to the variance it makes evident in the process of coding. We argue that authors should not treat coding results as data but rather as tabulations of claims about data and that it is important to discuss the rates and substance of disagreements among coders. We propose publication guidelines for authors and reviewers of this form of research.