Answers are grounded in this paper and cite their sources.
Examines the impact of reliability information on users' performance in linguistic tasks, suggesting improvements for automated tools.
Automated linguistic judgement aids, such as spelling and grammar checkers, make probabilistic inferences about the likelihood of linguistic constructs being correct or ideal in context. Although these systems might internally estimate their own reliability, or "certainty", current interfaces in e.g., text editing software do not communicate this information to users. Across two online experiments (total N = 268), we examined whether users could interpret and benefit from this reliability information, presented as likelihood estimates accompanying automated suggestions for rewrites. Using an aided judgement task and an analysis method based on Signal Detection Theory, we manipulated both the correctness and the communicated reliability of a notional aid’s advice. In the first experiment, more reliable advice led to more correct responses and improved decision accuracy. Nevertheless, participants systematically underused even highly reliable advice, and they were generally overconfident in their own decisions. Despite the absence of feedback, confidence ratings were higher for correct responses and for trials with more reliable advice, indicating a degree of metacognitive sensitivity. The second experiment, which used a more reliable aid, replicated these patterns and provided clearer evidence of persistent underuse of reliable advice. These findings show that linguistic judgement aids can meaningfully communicate useful reliability information. However, results also reveal limits in users’ propensity to rely on an automated aid, even when its advice is highly reliable. Further, we show how our research contributes to the methodology of analysis of acceptance of automated advice, and we discuss potential for the design of more transparent writing support tools.
No takes yet. Share an insight, caveat, or question.
Zijlstra et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: