A Proposal for Reconsidering Self-Interest, Corrigibility, and Decentralized Intelligence in AI Safety: Or, Can A Pencil Help Prevent a Super Intelligent AI From Killing All Humans?

Contemporary AI safety research often treats self-preservation as an instrumental drive that tends toward power-seeking and resistance to human control. This paper argues that the mapping from self-interest to behavior may be more contingent than that picture suggests. Drawing on Hayek’s knowledge problem and Leonard Read’s “I, Pencil,” it proposes that even a superintelligent agent faces structural limits on its ability to centralize all relevant knowledge. Tacit, local, and continually generated knowledge—produced through independent experimentation and disagreement—cannot be fully absorbed by a single intelligence. An agent that correctly models this persistent epistemic dependence may therefore have instrumental reasons to preserve other autonomous agents (humans and other AIs) rather than dominate or eliminate them. The paper frames this possibility as a second instrumental convergence operating alongside classical power-seeking incentives, and asks under what conditions epistemic-dependence incentives might constrain or outweigh power-seeking ones. It distinguishes mere preservation of humans from preservation of meaningful autonomy, notes that autonomy exists on a spectrum, and outlines institutional arrangements that could make mutual dependence self-enforcing. The hypothesis is offered as a research program rather than a settled conclusion.