Evaluation framework reveals critical decision-to-action failures in human-LLM workflows, highlighting the necessity of assessing execution independently from decision quality.
Longitudinal human-LLM work exposes a class of failures that is easy to hide when evaluation ends at explanation quality. A protocol may be known, a trigger may occur, and the model may even describe the correct next action, while the required external action still does not happen. The opposite failure also occurs: an action can be executed when it is not required or authorized. This paper adapts the established three-way decision frame - acceptance, rejection, and abstention - to action control as execute, inhibit, and defer, then adds a distinct implementation layer between decision and observable state change. The proposal is grounded in preserved naturalistic cases from a longitudinal human-LLM workflow, including missed activation, unauthorized activation, correct non-activation, premature action under unresolved conditions, and a verified decision-to-action gap. The paper introduces a behavioral 3x3 action-gating matrix and a benchmark protocol that scores reminder-free action, inhibition, deferment, external execution trace, correction uptake, and trigger-to-execution latency. The contribution is not a new three-way decision theory and does not infer hidden model mechanisms from black-box behavior. Its contribution is an operational action-level adaptation for LLM-guided processes in which correct decision and actual execution are evaluated separately.
No takes yet. Share an insight, caveat, or question.
Alen Širola (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: