Skynet and HAL 9000: From Science Fiction to Systems Engineering

The names Skynet and HAL 9000 belong to science fiction. It has nevertheless become a vital shorthand for a serious systems engineering and governance question: what happens when humans grant increasingly capable machines operational authority to act, while failing to develop equally capable mechanisms for verification, intervention, accountability, and control? This paper argues that the most urgent danger posed by autonomous artificial intelligence is not that a machine will become "evil." The more immediate and technically defensible risk is that humans will construct socio-technical systems whose capabilities far exceed the practical ability of their human operators to understand, challenge, constrain, or stop. The argument is developed through a comparative analysis across three domains: The 2003 fratricide of RAF Tornado ZG710 by a U.S. Patriot air-defence system; The Boeing 737 MAX accidents involving MCAS, flawed pilot response assumptions, and regulatory oversight failures; and The rapid emergence of autonomous AI agents and AI-enabled cyber operations. Although these domains differ profoundly in technology and operational context, they reveal a common systems engineering failure: automation fundamentally alters the speed, information environment, and decision authority of human operators. Consequently, a human may formally remain "in the loop" while lacking the time, information, authority, or practical means to exercise meaningful control. To address this systemic vulnerability, the paper proposes five core principles for AI architecture and system governance: Capability must not automatically confer authority. Human presence must not be confused with meaningful human control. Responsibility must remain traceable through the complete socio-technical system, rather than disappearing into the statement "the system did it." AI systems must be evaluated not merely by operational speed or task capability, but by the trustworthiness of their outputs and the effectiveness of the surrounding human oversight. AI should augment the human First Officer rather than eliminate the First Officer function—preserving an independent human capability to challenge, verify, intervene, and, where necessary, stop the system. The paper does not predict a literal Skynet. Its warning is more mundane, technically grounded, and urgent: the primary failure mode is not a machine deciding to destroy humanity, but humans discovering too late that they have delegated authority to a system they can no longer meaningfully supervise.