A Privacy-Preserving and Attack-Aware AI Approach for High-Risk Healthcare Systems Under the EU AI Act

Artificial intelligence (AI) has significantly driven advancement in the healthcare field by enabling the integration of highly advanced algorithms to improve diagnostics, patient surveillance, and treatment planning. Nonetheless, dependence on sensitive health data and automated decision-making exposes such systems to escalating risks of privacy breaches and is under rigorous regulatory oversight. In particular, the EU AI Act classifies AI uses pertaining to healthcare as “high-risk”, thus requiring the application of strict provisions related to transparency, safety, and privacy. This paper presents a comprehensive overview of the diverse privacy attacks that can target machine learning (ML)-based healthcare systems, including data-centric and model-centric attacks. We then propose a novel privacy-preserving architecture that integrates federated learning with secure computation protocols to minimally expose data while ensuring strong model performance. We outline an ongoing monitoring mechanism compliant with EU AI Act specifications and GDPR standards to further improve trust and compliance. We further elaborate on an independent adaptive algorithm that automatically tunes the level of cryptographic protection based on contextual factors like risk severity, computational capacity, and regulatory environment. This research aims to serve as a blueprint for designing trustworthy, high-risk AI systems in healthcare under emerging regulations by providing an in-depth review of ML-specific privacy threats and proposing a holistic technical solution.