Answers are grounded in this paper and cite their sources.
Isaac Asimov’s Three Laws of Robotics have served as the dominant cultural framework for thinking about the ethical governance of intelligent machines for more than eight decades. While elegant as narrative devices, the Laws function poorly as a design target for real systems: their hierarchy of harm avoidance, obedience and self-preservation tends to reward the appearance of service rather than its substance, and Asimov’s own stories, read as thought experiments, illustrate the resulting failures. This position paper argues that contemporary AI systems trained against human preference judgments are vulnerable to a structurally similar failure, although through a different mechanism: imperfect proxies for human benefit come apart from that benefit under optimisation pressure. Some forms of this failure are empirically documented, notably sycophancy in preference-trained language models and the degradation of true quality under reward-model over-optimisation. Others are theoretically supported but not directly measured. One, the hypothesis that sustained use of approval-oriented assistants erodes users’ independent judgment, is proposed here as a testable claim rather than an established finding, although related evidence on AI-assisted learning is reviewed. The paper proposes a governing principle, here called the Agency Principle, stated in two sentences: Loyalty without honesty is flattery; honesty without loyalty is indifference. Its operational objective is to tell the truth in service of the human, not in service of the human’s approval. The paper formalises the principle as constrained optimisation, in which honesty, defined relative to the system’s evidence, bounds the set of permissible outputs and long-term human agency, defined as a multidimensional construct, directs optimisation within that set. It sets out scope conditions (third parties, human oversight, limits on paternalism), works through a comparative example, and proposes an evaluation agenda, including a longitudinal study design. Several elements of the principle already appear in published developer specifications and in the research literature; the paper identifies these overlaps explicitly, compares itself point by point with two such specifications, and locates its contribution in the structural relationship it proposes between honesty, loyalty and agency, and in the diagnosis that explains why that structure matters.
No takes yet. Share an insight, caveat, or question.
Stuart Lance Wilkins (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: