LLM agents are increasingly deployed in long-horizon, multi-step settings–browsing, code editing, tool use–where a single user request expands into hundreds or thousands of model actions.The dominant alignment techniques (RLHF, Constitutional AI, DPO) optimise behaviour at the level of a single response, and they break down in long-horizon use for an arithmetical rea son that is more often stated than taken seriously: per-step alignment errors compound. At a per-step error probability of 10−3, an agent operating for 3,000 steps has a >95% probabilityof at least one misaligned action–regardless of how good its base alignment is on any individualresponse.This paper argues for a small, practical shift: stop optimising single-action policies harder, and start stacking cheap oversight around them. I describe a layered architecture I call Halo, which combines a thin layer of hard rules (compiled where possible to runtime monitors), a process reward model that can abstain when unsure, and a hierarchical decomposition that letsoversight scale with the depth of a task tree rather than the number of leaf actions. A post-hoc trajectory audit closes the loop. None of the layers is novel on its own; the contribution is the compositional argument–that under stated (and stated honestly) assumptions, layering cheap, fallible checks beats trying to make one layer near-perfect.I derive a compounding-error bound to motivate the design and a much looser layered-failure bound to characterise its benefit. I deliberately avoid empirical claims I cannot back up. I close with an honest list of what the framework does not solve–deceptively-aligned agents, ill-specified user intent, and the LLM-judge regress chief among them–and what I would want to see tested before any of this is taken as more than a working hypothesis.
Anaghashree Nanda (Thu,) studied this question.