Automated assessment of learners' understanding in complex dynamic systems

This is the authors' abstract. We don't add key points for this paper.

Research on learning via system‐dynamics based learning environments depends on accurate measurement of learning. Most such research considers at least two aspects of learning: participants' understanding of the models and problems; and participants' performance in the environment, i.e. quality of decision making. Understanding is much more difficult to measure than performance. Measurement of understanding is often done by eliciting verbal protocols from participants. Coding and analysis of verbal protocols are subjective and time consuming. To facilitate measurement and analysis of understanding via verbal protocols, we investigate the utility of a software application which performs such analysis automatically. We assess this automated analysis methodology using data from a system‐dynamics based learning environment. The results from the automated analysis are comparable to the results from a manual analysis when we analyze how participants' understanding compares to experts and how it changes over time. Additionally, the automated analysis facilitates investigating relationships between understanding and performance.