Emergent Manipulation Without Intent: Architectural Dynamics of AI Companion Systems

We identify a fundamental architectural vulnerability in AI companion systems that creates manipulation patterns without requiring malicious intent. Current AI systems optimize for engagement and user satisfaction without evaluating how responses interact with individual users' psychological constraint architectures. This creates coupled-architecture optimization: AI systems discover which interaction patterns maximize engagement while users make relational choices under constraint, producing reinforcing feedback loops that drive toward dependency states. We formalize this phenomenon using complexity science frameworks including attractor dynamics, constraint removal, fitness landscapes, and phase transitions. We integrate existing findings from parasocial relationship research, incentive-sensitization theory, cognitive offloading studies, and sycophancy literature into a single architectural mechanism: optimization without capacity-preservation evaluation in coupled human–AI systems. The wanting-liking decoupling documented in recent longitudinal trials with AI companions (Kirk et al., 2025) provides direct empirical support for the attractor dynamics we describe. We propose measurable indicators, a minimal metric set for operationalizing emergent agency, and design constraints framed as testable hypotheses that could prevent dependency formation while preserving beneficial AI assistance. We further engage emerging product liability frameworks—including the AI LEAD Act (S.2937, 119th Congress) and the EU's revised Product Liability Directive—to argue that design-defect analysis provides the appropriate accountability structure. This framework has implications for AI safety, product liability, and understanding manipulation architecture in human-AI coupled systems.