Large language model agents are often evaluated by one-run task completion: the agent receivesa task, acts in an environment, and is scored by the final state. This is useful, but it is not enoughfor deployment reliability. A system that succeeds once may fail on the fifth repetition, collapseunder a minor paraphrase, corrupt memory across sessions, break when a tool times out, or performan unsafe collateral action while still appearing competent on a benchmark. This paper introducesthe Reliability Half-Life of LLM Agents: the number of repeated, perturbed, tool-stressed executionsafter which reliable action falls to one half of its initial leve
Tony Newton (Sun,) studied this question.