Abstract Language plays a central role in learning processes in higher education, both in the acquisition and processing of information and in the production of written responses to academic tasks. When relying on online sources, these processes can be situated within the framework of Critical Online Reasoning (COR), which addresses students’ ability to search for, evaluate, and integrate online information in order to solve scenario-based tasks in a self-regulated manner. While COR research has mostly considered source-related and processual dimensions of online reasoning, the role of specific grammatical features as indicators of students’ task performance has received little attention. Addressing this gap, the present pilot study tests the hypothesis that a small set of grammatical features is sufficient to predict response quality, thereby supporting the inference of task-specific student performance in COR tasks. To test this hypothesis, we propose an integrated qualitative-quantitative approach applied to written responses from economics students. The qualitative analysis examines grammatical features at the levels of semantics (e.g., modality) or syntax (e.g., adverbial clauses), and relates them to expert evaluations of response quality. The resulting linguistic model is then operationalized computationally and evaluated on a larger dataset using machine-learning methods. The results provide evidence for the predictive, though still limited validity of the linguistic model and show that its feature set can be substantially reduced while improving predictive performance. We compare the model against similarly low-dimensional approaches, identifying promising alternatives from quantitative linguistics. Using evolutionary search and contrast analysis, we ultimately reduce the model to two features. Given the increasing number of AI-based approaches to automated essay scoring, our findings demonstrate the feasibility of a fully linguistically controlled, low-dimensional model that remains interpretable from an educational-science perspective while being computationally efficient.
Mehler et al. (Tue,) studied this question.