AI Act in Medicine: Legal Framework for High-Risk Systems

In 2023, the European regulator approved an AI system for emergency department triage support. A year later, the same type of system was added to a list of cases where AI incorrectly deprioritized patients with atypical clinical presentations. Both scenarios are possible simultaneously: AI in medicine works and fails—often on the same data, for the same reasons. The AI Act does not ban these applications. It defines the conditions under which they can operate responsibly.

The AI Act classifies systems based on potential harm, not industry. But the list of high-risk systems in Annex III directly targets healthcare:

• Point 5a: AI systems intended to support clinical decisions (diagnostics, treatment planning, medical imaging analysis),
• Point 5b: AI systems managing access to healthcare or resource allocation (prioritization, triage),
• Point 5c: AI systems monitoring patient health in real time.

For each of these applications, the full set of high-risk obligations applies. A chatbot at a hospital reception that only provides opening hours—limited risk. The same chatbot collecting symptoms and suggesting a department—falls under point 5b and triggers technical documentation, validation, human oversight, and incident logging.

The AI Act specifies obligations in Chapter III. For medical systems, these boil down to four pillars:

Each of these pillars is not a checklist item—it’s a process requirement. Missing logs means no way to demonstrate compliance in case of an audit or adverse event.

The most common question from hospitals and clinics is: "How do we explain to a doctor why the AI made that decision?" This question now has legal weight.

Art. 13 of the AI Act requires high-risk systems to be designed so their operation is sufficiently transparent for users to interpret results and use them correctly. In practice, this means:

• The doctor must receive not just the result, but justification (which image features, which clinical parameters influenced the decision),
• The justification must be understandable to a human without knowledge of the model’s architecture,
• The system must indicate confidence limits—when its recommendation is less reliable.

In our work with explainable AI (XAI) systems, we distinguish three levels: local explanation (why this decision for this patient), global explanation (how the model works in general), and audit trail (what it did over time). The AI Act requires at least the first and third. Methodologies like SHAP, LIME, or attention maps are implementation tools—none are required by name, but any can be used to demonstrate compliance.

The AI Act and GDPR are two separate regimes that overlap almost entirely in medicine. Patient data is health data—a special category under Art. 9 GDPR, requiring a clear legal basis (consent or public interest) and a significantly higher level of protection.

For every AI system operating on medical data, a Data Protection Impact Assessment (DPIA) must be considered. GDPR requires it when processing may pose a high risk to individuals’ rights—and AI deciding on diagnostics or treatment almost always meets this threshold.

Key principles for designing the data layer in medical AI systems:

• Minimization: The model should operate on the smallest possible dataset—not the full medical history if a fragment suffices,
• Pseudonymization before training: Patient identifiers must not appear in the training set or prompts,
• PII masking before any external API calls—sensitive data may not leave the hospital’s infrastructure,
• Right to erasure: If the model was trained on a patient’s data who withdraws consent, a technical method to remove that data must exist (machine unlearning or retraining without the subset).

A hospital implementing an outsourced AI system remains the data controller—it does not transfer this responsibility to the provider.

Art. 14 of the AI Act states that human oversight must be designed into the system—not added as an option. In a clinical context, this means a specific workflow architecture:

The doctor (or other authorized clinician) should be able to:

1. Review the justification before making a decision based on the AI’s recommendation,
2. Reject the recommendation without losing access to the system or needing to provide a reason,
3. Halt the system if there are doubts about its operation.

In agent-based architectures (systems that execute sequences of actions, e.g., treatment planning, ordering tests, updating documentation), a human-gate—requiring human confirmation before any irreversible action—is a direct implementation of Art. 14. The system can suggest, but it cannot independently execute irreversible actions. We detail this mechanism in the article on AI agent safety.

This does not limit automation. It limits it where errors could harm the patient.

High-risk AI systems in medicine are most often also medical devices under the Medical Device Regulation (MDR 2017/745) or In Vitro Diagnostics Regulation (IVDR). Both regulations must be met simultaneously—the AI Act does not replace MDR.

A notified body certifying a medical device must now consider the AI Act as part of the technical assessment. In practice:

• AI technical documentation (required by the AI Act) becomes part of the MDR certification files,
• The post-market monitoring plan (Art. 72 AI Act) must be synchronized with the Post-Market Surveillance Plan required by MDR,
• The adverse event log must be consistent across both regimes.

Manufacturers who already have MDR certification (class IIa or higher) for a system with an AI component should treat the AI Act as an extension of existing obligations, not as a new, independent project. A general overview of obligations for companies implementing AI—outside the medical sector—is covered in the article AI Act and GDPR in 2026.

One of the less obvious consequences of the AI Act in medicine: the requirement that training data be representative and free from discriminatory biases (Art. 10). In clinical practice, this is a problem the industry has known for years, but it is now a legal obligation.

Classic example: cardiovascular risk prediction algorithms trained primarily on cohorts of middle-aged Western men. Applied to women, older adults, or other ethnic groups, they may systematically underestimate risk. This is a statistical error, but if it leads to delayed treatment—it’s an adverse event for which the AI system administrator is liable.

Identifying and documenting potential biases is part of the technical documentation. If the training dataset has demographic limitations, these must be explicitly described along with recommendations for the populations the system is validated for. The AI readiness assessment tool helps identify gaps before the project enters the regulatory documentation phase.

Describe a use case for an AI system in a medical or healthcare context, and the model will help preliminarily assess the risk category under the AI Act, identify relevant articles, and pinpoint key obligations. This is a starting point for discussions with a lawyer and the responsible entity, not a substitute (playground: data masked, zero retention):

No. A chatbot limited to organizational information (hours, addresses, administrative procedures) is a limited-risk system. It must meet transparency requirements—the user must know they are interacting with AI. The boundary shifts when the chatbot starts collecting symptoms, suggesting diagnoses, or directing to departments: then it falls under Annex III and becomes a high-risk system with all associated obligations.

Both. The AI Act distinguishes between the provider (who developed and placed the system on the market) and the deploying entity (the hospital using it). The provider is responsible for technical compliance and documentation. The deploying entity is responsible for proper use according to instructions, oversight, and real-world monitoring. A hospital purchasing an off-the-shelf system does not offload its data controller obligations under GDPR onto the provider.

Not in the sense of clicking "OK" for every suggestion—that would be impractical. It requires that the clinician can intervene, reject the result, or halt the system. In practice, this is designed as: the AI’s result visible alongside clinical data, the ability to reject with a note, and a block on executing irreversible actions (e.g., automatic test ordering) without explicit confirmation. The scope of oversight is proportional to the stakes of the decision.

Yes, but the technical documentation must include a description of the training population and its demographic limitations. If the model was validated only on Polish patients and is deployed in Germany, an assessment is needed to determine if the cohorts are sufficiently similar. The AI Act requires systems to function correctly under foreseeable usage conditions—population differences may be such a risk condition.

At minimum: session identifier (without PII), model version, type of recommendation, whether the clinician accepted or rejected it, response time, model confidence score, and timestamp. For adverse events, the log must allow reconstruction of the decision context—without this, demonstrating that the system functioned correctly or that the error was due to usage (not the model) is impossible. Log retention time must comply with legal requirements for medical documentation in the given country.