The Duty-of-Care Dilemma of Companion AI in Mental Health: Care Ethics and Regulatory Responses under EU Law

Companion AI in mental health, anthropomorphic conversational systems providing emotional support across sustained periods of use, has moved from prototypes to mass-market products, and recent suicide cases in Europe and beyond have brought its risks to the centre of EU digital governance. This article argues that the EU’s regulatory framework, the AI Act, the Medical Devices Regulation, the GDPR, the Digital Services Act, and the revised Product Liability Directive, rests on a dual rights-based and product-based foundation ill-suited to the harms generated by companion AI: relational harm, emergent vulnerability, and emotional dependency. Distinguishing four legally distinct types of companion AI, only the first of which is a medical device, and drawing on care ethics, in particular Chadha-Sridhar’s account of hybrid non-voluntarism, it contends that these systems do not engage the physician’s traditional duty of care but give rise to a relational duty of care, triggered by users’ serious need and allocated in layered form, with the application operator as its primary bearer. It identifies three mismatches: between rights-based reasoning and relational harm, between static and emergent vulnerability, and between the product-based paradigm and continuous-care oversight. It proposes a limited but systemically significant shift towards a relational model, under which the duty of care operates as an auxiliary normative tool through the positive obligations of the EU fundamental rights order and the open-textured safety standards of the product-based instruments, within the internal market competence conferred by Article 114 TFEU.