How does Generative AI Affect Patients' Rights?

Photo by Igor Omilaev on Unsplash Abstract Healthcare systems are facing constant changes due to demographic modifications (a rapidly aging population), technological developments, global pandemics, and shifts in social paradigms. These changes are increasingly being analysed through the lens of patients’ rights, which are central in ethical and legal discussions in healthcare. A significant change in healthcare today is the growing use of generative artificial intelligence (AI) in clinical practice. This research analyses the potential risks of the use of generative AI systems to fundamental patients’ rights. With a mixed methodology combining literature review and semi-structured interviews with experts and stakeholders, the study identifies three main areas of risk, each one associated with fundamental values: the right to medical data protection (privacy), the right to equal access to healthcare (justice), and the right to informed consent (autonomy). The report concludes with a discussion of the findings and presents legal and ethical recommendations to promote the benefits of generative AI in healthcare. 1. Introduction The increasing digitalization of healthcare is reshaping how healthcare professionals deal with clinical tasks and patient interactions. This technological shift is accelerated by systemic pressures that healthcare is facing today due to a double aging population and workforce shortages. Generative artificial intelligence (GenAI) has the capacity to help healthcare providers with clinical documentation, decision-making, and patient communication through automated processes. At the same time, the fast integration of GenAI models in healthcare raises ethical and legal concerns. For example, general-purpose AI models are already being used in clinical practice without being subject to high-risk regulatory requirements. This produces regulatory gaps that challenge the protection of fundamental patients’ rights in real-world clinical settings. This report focuses on three main patients’ rights: the right to privacy, the right to equitable access, and the right to informed consent. These rights are represented in bioethical and legal frameworks for the protection of patients. The question guiding this study is the following: How does the use of generative AI in healthcare impact patients’ rights, particularly regarding privacy, justice, and autonomy? While the analysis is framed within the EU context, the concepts and findings remain relevant for broader global discussions. By identifying key risks, such as unauthorized access to health data, limitations of anonymization techniques, algorithmic bias, and digital informed consent, this study contributes to the growing body of research on AI in healthcare and the protection of patients’ rights. 2. Context 2.1. What is Generative AI? Generative artificial intelligence (GenAI) is a broad category of AI that, in addition to recognizing and predicting patterns, can also generate new content such as text, images, and sound, based on input and training data.[1] GenAI differs from traditional AI in two key ways: dynamic context and scale of use. While traditional AI is typically designed for specific contexts and predefined tasks, GenAI has a sort of “flexibility” and “creativity” that allows the model to learn new capabilities that it had never been explicitly trained for, allowing it to adapt to different contexts and uses.[2] In this sense, GenAI is one single tool with multiple uses and applications.[3] Because of this high adaptability, it is harder to interpret the complex learning algorithms of GenAI, which leads to less transparency of the system. Ultimately, when asking a GenAI model to create an outcome, if asked the same thing twice, it will provide inconsistent outcomes due to its probabilistic nature. A specific category of GenAI is large language models (LLMs), which are designed to generate human-like text. These models pertain to the class of natural language processing (NLP), the technology that allows computers to understand and process human language (an example would be Google Translate). LLMs are trained on enormous text datasets that allow the model to self-learn and create text on its own.[4] GenAI has gained significant attention since the release of ChatGPT, a chatbot made publicly available by the American organization OpenAI in 2019. Its ease and free accessibility reached widespread adoption[5] also in healthcare settings.[6] 2.2. Generative AI in Healthcare In healthcare, traditional AI systems are used in several areas. For example, in radiology, they automate the detection and classification of medical images.[7] In emergency departments and intensive care units (ICUs), AI is used as a decision support system. For example, the Pacmed Critical model at Leiden University Medical Centre (UMC) (Netherlands) is a machine learning model that predicts readmission or death after ICU discharge.[8] AI is also used in patient monitoring to track physiological changes and provide predictive analytics: MS Sherpa is an application for multiple sclerosis that uses digital biomarkers to monitor symptom progression and disease activity.[9] GenAI offers new possibilities, mainly aimed at reducing administrative burdens, for instance, through automatically creating clinical documents like discharge letters, referral letters, and clinical notes.[10] For example, the UMC Utrecht (Netherlands) has developed an application that uses General Pre-training Transformer (GPT) to generate draft discharge letters.[11] GenAI is also being used to transcribe and summarize conversations between doctor and patient. “Autoscriber,” at Leiden UMC research department (Netherlands), is a digital scribe system that automatically records, transcribes, and summarizes the clinical encounter.[12] Besides administrative tasks, GenAI can assist with clinical decision-making by creating diagnosis and treatment recommendations based on patient data.[13] It also supports medical research activities like assisting in systematic reviews.[14] GenAI is also used to automatically answer patients’ questions related to their care. For example, at the Elizabeth-Twee Steden Hospital (Netherlands), a chatbot called “Eliza” answers patients’ medical questions.[15] 2.3. Current Use of Generative AI in Healthcare The use of GenAI in healthcare is rapidly increasing, which is changing how healthcare providers manage clinical tasks and patient interactions. Recent empirical studies reveal that more than half of healthcare providers use ChatGPT, or similar general-purpose LLMs, to assist with clinical documentation, patient communication, clinical decision-making, research, and more.[16] These studies also show that despite this widespread use of GenAI, most healthcare providers lack the required knowledge and awareness of the risks of using this tool in general, and specifically for clinical tasks.[17] This lack of comprehension is probably because GenAI has only become popular and widespread recently, which makes it difficult to fully understand and assess the risks and scale of these technologies to society. This gap in understanding GenAI’s risks is reflected in healthcare institutions. For example, a survey on AI use in Dutch hospitals found that GenAI was used in 57 percent of hospitals, with applications such as automatic transcriptions, document summarisation, and text generation.[18] The same study showed critical issues: in only 29 percent of hospitals, it was clear on what frequency AI models are retested, trained, and calibrated to errors such as hallucinations[19] and data drifting.[20] In more than half of the hospitals (52 percent), it is unknown whether, and if so, in what frequency, such practices occur at all, and in 11 percent, AI models are never retrained. Moreover, only 30 percent of hospitals reported having an AI policy describing the frameworks, standards, and guidelines for the use of AI.[21] Another survey found that 76 percent of physicians reported using general-purpose LLMs, like ChatGPT, for clinical decision-making.[22] More than 60 percent of primary care doctors reported using them to check drug interactions; while more than half use them for diagnosis support, nearly half for clinical documentation, and more than 40 percent for treatment planning. Additionally, 70 percent use general-purpose LLMs for patient education and literature search. These findings show a mismatch between the growing use of GenAI in clinical practices and the governance needed to ensure its responsible use. While GenAI has the potential to enhance efficiency and accuracy in clinical tasks, if it is integrated without the necessary knowledge, governance, legal, and ethical oversight, it can lead to harmful consequences to patients, such as data protection violations, automation bias, unclear accountability, healthcare inequality, incorrect clinical decisions, and the spread of misinformation.[23] 2.4. Regulatory Landscape At the European Union (EU) level, efforts to regulate the safe use of AI in healthcare are currently fragmented. This means there is not one regulatory framework solely dedicated to governing the use of AI in healthcare. Instead, different laws cover different parts of the issue, including the European Union AI Act,[24] the General Data Protection Regulation,[25] and the Medical Devices Regulation.[26] 2.4.1. The European Union AI Act In August 2024, the Artificial Intelligence (AI) Act entered into force. The AI Act is an EU regulation that sets rules for the development, introduction to the market, and deployment of AI systems. It adopts a risk-based approach: depending on the application and use of the system, it will fall under low, middle, high, or impermissible risk. The higher the risk, the stricter the regulatory requirements (e.g., risk management, data governance, human oversight).[27] Medical devices like AI diagnostic tools are classified as high-risk systems due to their direct implications for health outcomes. On the contrary, the majority of GenAI systems, like ChatGPT, fall under the category of general-purpose AI systems, which means that they can be classified both as high-risk and low-risk, depending on their application.[28] Therefore, the actual risk of the GenAI system depends on how and where it is used.[29] Large GenAI systems (like ChatGPT, Bard, DALL-E) are considered to pose systemic risks due to their widespread adoption; however, they are not always classified as high-risk applications.[30] This means that, in practice (as previously shown), a healthcare provider can and does use these systems for clinical tasks without the systems being under the requirements of high-risk medical devices. While these tools are fast and have access to vast amounts of data, they are relatively new, freely available, and not specifically designed or trained for medical use. Without the appropriate oversight and awareness, it creates the potential for unacceptable risks to patient care. Moreover, the AI Act is presented as a horizontal regulation, which means that it applies across all sectors and industries rather than focusing on the unique needs, risks, and ethical concerns of the healthcare sector.[31] As argued later, the increasing use of digital healthcare presents new risks to patients’ rights, which will require additional and tailored protections. 2.4.2. Medical Device Regulation The Medical Device Regulation (MDR) is an EU-binding document that governs the use of devices in clinical settings. It is also risk-based, depending on the intended purpose. The MDR provides strict rules for GenAI systems intended for clear medical purposes, such as diagnosis. However, not all applications of GenAI are considered medical devices under the MDR, even when used for clinical tasks.[32] For example, when GenAI is used for facilitating communication between patients and practitioners, summarizing clinical reports, or generating referral letters, it is not defined as a medical purpose; therefore, they do not fall under the MDR regulation. Consequently, if healthcare providers use GenAI for such “non-medical purposes,” there is no regulatory guidance on critical issues like patient privacy and legal responsibilities.[33] GenAI systems are highly adaptable and can be used for many different purposes. Because of this versatility, the MDR and similar regulations based on defined intended purposes face particular challenges. Many GenAI systems, such as ChatGPT, are not specifically designed for medical settings, although healthcare providers use them for clinical tasks. This leads to a regulatory gap: the technology is being used in practice but lacks adequate regulation. This lack of regulation does not ensure the trustworthiness of these tools in clinical settings and poses unacceptable risks to patients’ rights. 2.4.3. General Data Protection Regulation The use of GenAI in healthcare settings often involves dealing with large volumes of sensitive data like medical records, scan images, and lab results. The management of this data is regulated by the General Data Protection Regulation (GDPR), an EU regulatory framework to protect data privacy. The GDPR classifies health data as a special category of sensitive information that requires additional protections. It grants data subjects with specific rights, including the right to informed consent, the right to access the data, the right to rectification, and the right to be forgotten.[34] Patient data falls under this category, and the GDPR provides strong protections, enabling the reinforcement of the principle of medical confidentiality by limiting the use and amount of such data strictly to the purpose of direct care. In practice, this means that a hospital cannot use patient data for training an AI algorithm or share it with an external vendor without obtaining explicit informed consent or meeting a legal exemption. While the GDPR is clear for GenAI systems that are developed by the healthcare organisation itself, it becomes challenging for general-purpose GenAI systems, like ChatGPT, where the influence of the GDPR is less powerful compared to models explicitly designed to process personal data.[35] This creates a regulatory grey area for the use of general-purpose GenAI systems in healthcare settings regarding compliance with sensitive data protection standards. 3. Patients’ Rights Healthcare systems are facing changes constantly (rapidly ageing population, scientific and technological developments, global pandemics, shifts in social paradigms, etc.). These changes are increasingly being analysed through the lens of patients’ rights.[36] A significant change in healthcare today is the growing use of AI in medical tasks. This technological shift is likely to change traditional patients’ rights into what may soon be recognized as digital patients’ rights.[37] The field of patients’ rights lies at the intersection of ethics and health law, bringing together moral imperatives and legal protections. Patients’ rights are a special category of human rights aimed at protecting the dignity of the individual who is in a vulnerable state of illness.[38] Since nearly all humans become patients,[39] and patients are among the most vulnerable groups in society,[40] their rights are uniquely defined and crucially important.[41] The position of the patient is especially vulnerable because of their illness, which can cause insecurity and fear. Moreover, the patient is in an unbalanced position compared to the doctor, who is learned, skilled, and experienced in the topics in which the patient often knows little or nothing about, and still are extremely important for the patient, since their health may depend on them.[42] Besides this information asymmetry, the interaction between patient-practitioner is of a critical and private nature, which leaves the patient to highly depend on the practitioner in order to obtain adequate assistance.[43] This imbalance creates an easy potential for abuse of power (intentional or not) and shows why it is necessary to give special attention to protecting the patient. 3.1. Legal Protection for Patients’ Rights Over the past decades, patients’ rights have been recognized in a variety of different documents (Declarations, Charters, Laws) at the international, regional, and national levels.[44] Examples of these regulatory efforts include: European Convention of Human Rights and Fundamental Freedoms (1950) International Covenant on Civil and Political Rights (1966) A declaration on the promotion of patients’ rights in Europe (WHO, 1994) Declaration of Lisbon of the World Medical Association (1995) Wet op de Geneeskundige Behandelingsovereenkomst (WGBO) (Medical Treatment Agreement Act) (1995) Convention for the Protection of Human Rights and Dignity of the Human Being with regard to the Application of Biology and Medicine (Oviedo Convention) (1997) European Charter of Patient Rights (2002) These documents are crucial to provide a framework to protect the dignity, freedom, self-determination, and respect of patients. However, the fragmented nature of patients’ rights creates a complex landscape that can be challenging for both patients and healthcare providers to navigate. As Herranz notes,[45] these documents are highly diverse and target different audiences; some are universal, others regional or national in scope. While this diversity reflects the growing importance of patients’ rights globally, it also creates a fragmented landscape. Both patients and healthcare providers may find it difficult to understand the specific rights and obligations that apply in their context due to the diverse and spread nature of these rights between jurisdictions.[46] 4. Methodology 4.1. Study Design and Population This qualitative study consisted of semi-structured interviews with key experts and stakeholders. Stakeholder mapping was conducted through a document desk review. This process identified five relevant stakeholder groups: (1) patients, (2) healthcare providers, (3) healthcare organizations, (4) AI & Data experts, and (5) Ethical & Legal experts. Participants were selected based on the following criteria: being 18 years or older, having the capacity to give informed consent, being knowledgeable about the use of AI in healthcare (this criterion did not apply to patient participants), and the ability to communicate in English. Each interview began with a short case study to provide participants with a concrete scenario to consider while answering the questions. Presenting a case study enabled a more focused discussion and helped participants reflect on specific risks.[47] 4.2. Tools Member States of the European Union (EU) do not share a single binding document to protect patients’ rights. Instead, they are diversified in multiple pieces of legislation. Although these rights are widely recognized in the EU, each country applies its own medical regulations depending on its context and traditional norms.[48] However, it is possible to identify a set of fundamental patients’ rights that are widely recognised across all EU Member States.[49] This study focused on three of these fundamental patients’ rights to guide the development of interview questions. These rights were selected based on existing European frameworks (including the European Convention on Human Rights; the Charter of Fundamental Rights of the European Union; and the European Convention on Human Rights and Biomedicine, or the Oviedo Convention), as well as Dutch legislation (Burgerlijk Wetboek Boek). The selected rights include: (1) The right to autonomy & informed consent of the patient: Patients must be able to make informed decisions about their care,[50] (2) The right to privacy & medical data protection: Personal health data must be kept secure and confidential,[51] and (3) The right to access to healthcare & non-discrimination: Care must be accessible to all, regardless of background, and without unfair barriers.[52] This research also draws on the classic bioethical framework proposed by Tom Beauchamp and James Franklin Childress[53] to identify ethical guidelines that can support the responsible use of AI in healthcare and help safeguard those patients’ rights. The principles include: (1) Principle of justice: In healthcare ethics, justice refers to the concept of distributive justice, where all patients must be treated equally. This means every patient should receive the same quality of care (offering a uniform standard of quality) regardless of who they are. Persons with greater levels of need should be entitled to greater healthcare services when there is no discernible direct injury to others with lesser levels of need. In the context of AI, this raises important questions: Is access to AI-driven healthcare tools equitable? Are certain groups being left behind due to cost, location, or bias in algorithms? Justice also requires rejecting discrimination and ensuring that health technologies are available to all who need them. This principle is also public and legislated; (2) Principle of non-maleficence: This principle means “do no harm.” It is rooted in the Hippocratic tradition and updated in modern medicine to include preventing harm from unnecessary medical interventions (quaternary prevention). When applied to AI, it asks: Could the use of AI lead to the misdiagnosis of a patient, reinforce bias, or erode trust in care? If AI tools cause harm through poor design, overreliance, or misuse they can breach this core ethical obligation. It is a principle of the public sphere and non-compliance is punishable by law; (3) Principle of autonomy: Respect for autonomy (“self-rule”) means allowing individuals to make informed decisions about their own care according to their own reasons, values, and motives (rationality). AI tools can either support or undermine this principle. For example, if patients don’t understand how an AI tool was trained (or if it is used without their consent), their autonomy could be undermined. Ensuring transparency and informed consent is crucial to respect this principle; and (4) Principle of beneficence: The duty to do what is in the best interest of the patient. In AI, this might mean improving diagnostic accuracy, supporting clinical decisions, or helping to personalize treatment. This principle is more private in nature; it is a moral obligation rather than a legal one. Still, it is a guiding value in evaluating whether AI truly enhances patient care. If there is a conflict of ethical principles, for instance, when promoting autonomy might risk the harm of a patient, justice and non-maleficence (which are public and legal) are above those of beneficence and autonomy (which are located in the private level).[54] Interview questions were designed to explore how generative AI, such as LLM models used for documentation or decision support, may pose risks to these fundamental patients’ rights. Sample questions include: “What is your opinion on the use of generative AI to assist with clinical notes during patient consultations? How might this use impact patient privacy and informed consent?”; “How do you think this application of generative AI could change the doctor-patient relationship, especially regarding trust and transparency?”; “Do you think the use of generative AI in healthcare could impact equal access to care? If so, how?”; and “Who should be held accountable if the use of generative AI harms a patient?’’ For more information, see Appendix. 4.3. Participants & Recruitment In total, 66 individuals and 12 organizations were invited to partake in this study; 35 did not respond, 17 had other priorities, and one lacked insight on AI in healthcare. 14 individuals participated: one patient, four healthcare professionals, four ethics & legal experts, five AI & data experts, one health attorney, and three medical students (see Table 1). Participants came from different professional backgrounds, as long as their work gave them insights into the use of AI in healthcare (this criterion did not apply to patient participants). Recruitment was done through email and LinkedIn networks. Follow-up reminders were sent seven days after the first invitation. Table 1 – Participants’ characteristics 4.4. Data Collection Interviews were conducted between March 5 and April 7, 2025. Five interviews were conducted face-to-face, and nine were conducted via video calls using Microsoft Teams. Potential bias between the two methods was considered minimal, as video calls also allow for face-to-face interaction and can promote trust.[55] Face-to-face interviews were conducted at a place chosen by participants to ensure comfort and confidentiality. All interviews were semi-structured, with a consistent set of core questions and the flexibility to ask follow-up questions when relevant. This allowed for flexibility, acknowledging that participants have diverse forms of expertise relevant to the research topic. All participants were provided with a case study, interview questions, a participant information sheet, and asked to sign an informed consent form before the interview was conducted. All interviews, with the exception of one, due to the wishes of the participant, were recorded. During the interviews, field notes were taken, capturing the mood, tone, and expressions of the respondents. Field notes also helped to develop follow-up questions. Interviews lasted from 22 to 86 minutes. Five interviews lasted less than 30 minutes (35.7 percent), four lasted between 30 and 45 minutes (28.6 percent), and five lasted more than 45 minutes (35.7 percent). While the same topics were covered in all interviews, the focus varied depending on the participants’ expertise. 4.5. Data Analysis Interview data was analysed using Microsoft Excel. Audio recordings were transcribed manually, and field notes were included. Through the analysis of interviews, textual data was organized, cleaned, and manually coded. Data was classified into five main categories, according to the European Patients’ Rights framework and Bioethical Principles: (1) General Perspectives, (2) Privacy, (3) Autonomy, (4) Justice, and (5) Future Prospects. After coding the data into the general themes, subthemes were identified and coded manually. For example, the subcategories under (2) Privacy include: 2.1) Cloud Storage and Retraining Purposes, 2.2) Unauthorized Data Access, 2.3) Limits of Data Anonymization, and 2.4) Data Rights. Results were synthesised per code while remaining sensitive to links and patterns across the data. After the initial synthesis of results, all interviews were reviewed to identify any overlooked insights. 4.6. Ethical Considerations Under Dutch law, this study did not require approval from an Ethics Review Board.[56] However, the following steps were taken to safeguard ethical conduct: Data was de-identified and reported anonymously. Prior to participation in the study, participants were sent comprehensive information about the study's purpose and procedures via an information sheet and an informed consent form. Verbal informed consent was obtained at the start of the interview. All participants were above 18 years of age. 5. Results This research aimed to identify which patients’ rights are at most risk when using GenAI systems in healthcare. Given the fast digitalization of healthcare, traditional patients’ rights may experience new (and unexpected) risks that must be safeguarded. The study revealed three main areas of risk, each one associated with corresponding fundamental values: medical data protection (privacy), equal access to healthcare (justice), and informed consent (autonomy). Results are organized into the following sections: general perspectives, medical data protection, equal access to healthcare, informed consent, and future perspectives. 5.1. General Perspectives All participants (14/14) considered the use of GenAI for administrative tasks generally more beneficial than risky, pointing out that it would reduce the pressure of healthcare organisations regarding staff shortages and time constraints. While benefits were identified, participants also acknowledged specific concerns associated with this use. For instance, a healthcare provider (P3) mentioned that “AI can be helpful for storing and retrieving patient information quickly. However, there are critical concerns, including loss of human empathy, depersonalized care, vulnerability to cyberattacks, and malfunction.” In contrast, when discussing the use of GenAI for clinical reasoning, such as diagnostics, participants considered patients’ rights to be at a greater risk. Eight participants expressed reluctance regarding this use due to the complexity and contextuality of clinical decisions. A healthcare provider (P2) explained: “A diagnosis is never based on one result, but on a combination of contextual factors, so I think AI cannot fully operate there.” Moreover, “AI will be fed with massive datasets and can be very accurate at specific tasks, but it might miss important details that a practitioner would see, since human minds are more varied and flexible,” an ethical and legal expert (P11) remarked. As a result, participants considered patients’ rights to be at different levels of risk depending on the specific use of the GenAI, consistently mentioning clinical reasoning to be at the greatest risk. 5.2. The Right to Medical Data Protection (Privacy) Most participants (12/14) identified the right to keep medical data private (confidentiality) as the most threatened patient right by the use of GenAI, both for administrative tasks and clinical reasoning. Four main problems were recognized: (1) cloud storage and retraining purposes, (2) unauthorized access to