Evaluating GenAI for automated EU AI Act compliance against human experts

The growing adoption of Artificial Intelligence (AI) in critical sectors raises increasing concerns about regulatory compliance. The European Union Artificial Intelligence Act (EU AI Act) represents the first comprehensive legal framework specifically designed to govern AI systems, classifying them according to their associated risk levels. This paper presents an AI-based compliance checker developed to support the assessment of AI systems under the EU AI Act. The tool is designed to help developers understand legal obligations and identify potential compliance risks through structured, contextualized analysis. We describe the system’s architecture, which combines a generative language model with legal document retrieval techniques to interpret regulatory requirements. The tool was tested on a range of real-world use cases, including algorithmic hiring, medical diagnostics, smart city traffic management, and facial recognition in retail environments. Evaluation was conducted through both automated analysis and expert review. The results confirm the system’s effectiveness in understanding and reconstructing core regulatory obligations, particularly in assessing risk levels and suggesting improvements. While the tool performed strongly in central legal areas, some limitations emerged in handling highly technical or cross-regulatory content. Expert feedback highlighted its utility as a pre-audit tool, emphasizing its depth of legal reasoning, transparency, and contextual adaptability.