Does Learning Statistics Have to Be So Difficult? Evidence from a Randomized Controlled Trial

Objectives Computing education research often relies on empirical methods. Students in this area must use statistical analysis and formalized reporting standards (e.g., APA JARS). These traditions are complex and prior work consistently shows that many learners experience anxiety around them (e.g., data management, methodology, test selection, complex equations, custom programming languages). Most computer science students receive limited, if any, formal training in statistics beyond an introductory course. In this paper, we conduct a randomized controlled trial with students in and out of computer science under two conditions. In the first, we use standard statistics traditions along with the programming language R. In the second, we modify these traditions to use more plain language and develop a custom statistics language and API based on a recent theory called “say what you do” that we hypothesize may impact anxiety and correctness. Participants A total of 293 participants completed the study, including 158 Computer Science (CS) students. Non-CS participants were included for comparison in how those in other fields engage with this material. Our sample included different genders and for the first time, to our knowledge, participants with disabilities, a group under-studied regarding statistical anxiety. Study Methods We conducted a between-subjects experiment that compared a traditional statistical programming approach with an experimental one based on the theory in question. Changes to test names, code structure, and output provided identical answers to R but were redesigned from the lens of computer science students that receive little or no training. Participants completed six tasks across three phases of statistical work: selecting tests, writing code, and interpreting output. Measures included anxiety, confidence (self-efficacy), perceived difficulty, uncertainty, and correctness. Quantitative and qualitative responses were collected during each task. Findings The re-imagined approach reduced anxiety, uncertainty, and perceived difficulty while increasing confidence and performance, especially in coding and interpretation. Disability status and gender significantly influenced statistical anxiety, addressing a gap in research where disability is rarely considered, if at all. Uncertainty emerged as distinct from correctness: participants selected ‘I am not sure’ at a higher rate in the traditional approach, with qualitative feedback suggesting these responses often reflected hesitation rather than misunderstanding. Prior studies classify such responses as incorrect; this work shows that previous performance metrics can conflate knowledge with anxiety-driven disengagement. To our knowledge, this study represents the first formal test of the “say what you do” theory. We found it was largely supported by the evidence, but also that it had limitations that suggest its impact is uneven. Conclusions Statistics and statistical programming could be altered to be easier for computer science students and potentially everyone. In addition to the study, we developed a prototype environment that implements the design principles we evaluated and supports many of the statistical procedures commonly taught in computing education (e.g., ANOVA, common corrections, factor analysis, nonparametric tests). The changes we made reduced statistical anxiety and uncertainty and increased confidence and performance.