AI Automated Decision-Making and Human Oversight

AI Automated Decision-Making and Human Oversight

Overview

GDPR Article 22 grants data subjects the right not to be subject to decisions based solely on automated processing, including profiling, which produce legal or similarly significant effects. The EU AI Act Art. 14 supplements this with specific human oversight design requirements for high-risk AI systems. Together, these provisions require organisations to identify when AI systems make consequential decisions, ensure meaningful human intervention where required, provide explainable decision logic, and offer effective contestation mechanisms. This skill provides the complete framework for Art. 22 compliance and AI Act human oversight implementation.

Art. 22 Scope and Applicability

Three Cumulative Conditions

Art. 22(1) is triggered only when all three conditions are met:

"Solely Automated" — EDPB Interpretation

The EDPB Guidelines 06/2020 on automated decision-making clarify:

• Solely automated means no meaningful human involvement in the decision process

• A human who merely confirms an AI recommendation without genuine assessment is not providing meaningful intervention

• Meaningful human intervention requires:

  • The reviewer has authority and competence to change the decision
  • The reviewer has access to all relevant information
  • Sufficient time is allocated for genuine consideration
  • The reviewer routinely exercises independent judgment (not just confirming AI output)
  • Override capability is actually used in practice

• Not solely automated when:

  • A qualified human genuinely reviews the AI output as one input among several
  • The human applies independent judgment and makes the final decision
  • The human has the capability and actually exercises discretion to deviate from AI recommendations

"Legal or Similarly Significant Effects"

Effects that are not similarly significant (per EDPB):

• Personalised advertising (unless it reinforces prejudices about vulnerable groups)
• Generic recommendation systems (unless they determine access to information)
• Spam filtering (unless it systematically blocks important communications)

Art. 22 Exceptions and Safeguards

Art. 22(2) — Permitted Automated Decisions

Art. 22(3) — Mandatory Safeguards

When an Art. 22(2) exception is relied upon, the controller must implement at least:

1. Right to obtain human intervention: A qualified person reviews the automated decision
2. Right to express point of view: Data subject can present additional information or context
3. Right to contest: Formal mechanism to challenge the automated decision with review by a different decision-maker

Art. 22(4) — Special Category Data Restriction

Automated decisions based on Art. 9 special category data are only permitted under:

• Art. 9(2)(a) — Explicit consent, OR
• Art. 9(2)(g) — Substantial public interest based on Union or Member State law

In both cases, suitable measures to safeguard data subject rights must be in place.

Human Oversight Design — AI Act Art. 14

Art. 14 Requirements for High-Risk AI

High-risk AI systems must be designed and developed so that they can be effectively overseen by natural persons during use:

Human Oversight Levels

Meaningful Human Intervention Criteria

A human review qualifies as "meaningful intervention" when all criteria are met:

Contestation and Appeal Mechanism

Design Requirements

Contestation Workflow

1. Data subject notified of automated decision with explanation
2. Data subject submits contestation with reasons and any supporting evidence
3. Contestation assigned to qualified human reviewer (different from original oversight)
4. Reviewer assesses: AI inputs, AI output, explanation, data subject's arguments
5. Reviewer makes independent decision: uphold, modify, or overturn
6. Outcome communicated to data subject with reasons
7. If upheld: data subject informed of further options (DPA complaint, judicial remedy)
8. Record maintained in rights exercise register

AI-Specific Decision Categories

Credit and Financial Decisions

• AI application: Credit scoring, loan approval, insurance pricing, fraud detection
• Art. 22 trigger: Yes — determines access to financial services
• Required: Meaningful human review before denial; explanation of key scoring factors; contestation with independent review
• Enforcement: AEPD fined CaixaBank EUR 6M for automated credit decisions without adequate safeguards

Employment Decisions

• AI application: CV screening, candidate ranking, performance scoring, termination prediction
• Art. 22 trigger: Yes — significantly affects employment and livelihood
• Required: Human hiring manager makes final decision with genuine authority to deviate from AI ranking; applicants informed of AI use; rejected candidates can request explanation
• Enforcement: Italian DPA fined Deliveroo EUR 2.5M for algorithmic worker management without Art. 22 safeguards

Healthcare Decisions

• AI application: Diagnosis assistance, treatment recommendations, triage, risk scoring
• Art. 22 trigger: Yes if AI determines care pathway — usually mitigated by physician oversight
• Required: Physician makes final clinical decision; AI functions as decision support; patient informed of AI role
• Special category: Health data — Art. 22(4) applies

Public Administration

• AI application: Benefit eligibility, fraud detection, risk assessment, resource allocation
• Art. 22 trigger: Yes — determines access to public services and benefits
• Required: Art. 22(2)(b) legal basis required; suitable measures mandated by law; transparency about algorithmic criteria
• Enforcement: Dutch court struck down SyRI fraud detection system for lack of transparency and proportionality

Profiling Assessment

GDPR Definition of Profiling (Art. 4(4))

Any form of automated processing to evaluate personal aspects relating to a natural person, in particular to analyse or predict:

• Work performance
• Economic situation
• Health
• Personal preferences
• Interests
• Reliability
• Behaviour
• Location
• Movements

AI Profiling Risk Assessment

Enforcement Precedents

• AEPD v. CaixaBank (PS/00421/2020, 2021): EUR 6M fine for automated credit decision-making without adequate Art. 22 safeguards, explanation, or contestation mechanism.
• Italian DPA v. Deliveroo (2021): EUR 2.5M fine for algorithmic management of delivery riders — Art. 22 applied to automated work allocation and performance scoring.
• Dutch Court v. SyRI (2020): Algorithmic fraud detection system struck down — automated profiling of citizens without proportionality, transparency, or adequate safeguards.
• Hungarian DPA v. Bank (2019): Fine for automated credit denial without providing meaningful information about decision logic or contestation mechanism.
• French Conseil d'Etat v. Parcoursup (2019): Court upheld use of algorithm for university admissions only because meaningful human review was genuinely conducted for each application.
• Austrian DPA v. CRIF (2023): Credit scoring company — violation of Art. 15(1)(h) for failing to provide meaningful information about automated scoring logic.

Integration Points

• ai-transparency-reqs: Explanation requirements feed into transparency framework
• ai-dpia: Human oversight assessment is DPIA Phase 5 component
• ai-data-subject-rights: Right to explanation and contestation are rights exercise procedures
• ai-deployment-checklist: Art. 22 compliance is pre-deployment validation item
• ai-bias-special-category: Bias in automated decisions creates Art. 22 harm