Conducting AI System Privacy Assessment

Conducting AI System Privacy Assessment

Overview

AI systems that process personal data require a combined privacy and conformity assessment addressing both GDPR obligations and the EU AI Act (Regulation 2024/1689). This skill integrates the GDPR Art. 35 DPIA framework with AI-specific risk assessment, encompassing training data lawfulness, Art. 22 automated decision-making implications, algorithmic fairness, and the NIST AI Risk Management Framework MAP function. The assessment methodology draws from the EDPB-EDPS Joint Opinion 5/2021 on the AI Act proposal and subsequent EDPB Guidelines 06/2025 on AI and data protection.

Legal Framework

GDPR Provisions Applicable to AI

EU AI Act (Regulation 2024/1689) Risk Classification

EDPB-EDPS Joint Opinion 5/2021

Key recommendations from the Joint Opinion on the AI Act proposal:

1. Prohibition of remote biometric identification in public spaces — supported without exceptions
2. AI-specific DPIA — the EDPB recommended mandatory DPIAs for all high-risk AI systems, not limited to Art. 35 triggers
3. Right to explanation — Art. 22(3) right to obtain human intervention and contest decisions should apply to all AI-assisted decisions, not only fully automated ones
4. Training data governance — controllers must ensure lawfulness of personal data used for AI training throughout the model lifecycle, including when training data is subsequently deleted
5. Purpose limitation for AI training — repurposing personal data for AI model training requires a compatibility assessment under Art. 6(4) or a separate lawful basis

Assessment Methodology

Phase 1: AI System Classification (Week 1)

1.1 AI Act Risk Classification

Determine the AI system's risk category under the AI Act:

1. Check Art. 5 prohibited practices list — if the system falls within a prohibition, it cannot be deployed.
2. Check Annex III high-risk categories:
   • Biometric identification and categorisation of natural persons
   • Management and operation of critical infrastructure
   • Education and vocational training (admissions, assessment, proctoring)
   • Employment, worker management, and access to self-employment (recruitment, task allocation, performance evaluation, termination)
   • Access to and enjoyment of essential private and public services (credit scoring, insurance pricing, emergency services dispatch)
   • Law enforcement (crime risk assessment, polygraph, evidence evaluation)
   • Migration, asylum, and border control
   • Administration of justice and democratic processes
3. Check Art. 50 transparency obligations for limited-risk AI.
4. If not high-risk or limited-risk, classify as minimal risk.

1.2 GDPR Processing Assessment

1. Identify all personal data processed at each AI lifecycle stage: data collection, data preparation, model training, model validation, inference/deployment, output use.
2. For each stage, identify the lawful basis under Art. 6(1).
3. If special category data is involved, identify the Art. 9(2) condition.
4. Assess whether Art. 22 applies (solely automated decisions with legal or significant effect).

Phase 2: Training Data Lawfulness (Week 2)

2.1 Data Collection Lawfulness

2.2 Data Quality and Bias Assessment

2.3 Training Data Retention

• Art. 5(1)(e) storage limitation applies to training data.
• Once training is complete, assess whether original training data must be retained or can be deleted.
• If training data is deleted, document how data subject rights (access, erasure) will be facilitated regarding data embedded in the trained model.
• Consider whether the trained model itself constitutes personal data (if individual data points can be extracted through model inversion attacks).

Phase 3: Art. 22 Automated Decision-Making Assessment (Week 3)

3.1 Art. 22(1) Applicability Test

Does the AI system make decisions about individuals?
├─ NO → Art. 22 does not apply.
└─ YES → Continue.
    │
    Are decisions based solely on automated processing?
    ├─ NO → Art. 22(1) does not apply, but Art. 13-14 transparency still applies.
    │   (Note: "meaningful human involvement" must be genuine, not rubber-stamping.)
    └─ YES → Continue.
        │
        Do decisions produce legal effects or similarly significantly affect the individual?
        ├─ NO → Art. 22(1) does not apply.
        └─ YES → Art. 22(1) applies. The individual has the right not to be subject
            to the decision unless an Art. 22(2) exception applies.

3.2 Art. 22(2) Exceptions

3.3 Art. 22(3) Safeguards

Even where an Art. 22(2) exception applies, the controller must implement suitable measures including:

• Right to obtain human intervention
• Right to express the data subject's point of view
• Right to contest the decision
• Meaningful information about the logic involved (not the full algorithm, but sufficient to understand the factors and their relative weight)

Phase 4: Algorithmic Bias and Fairness Assessment (Week 3-4)

4.1 Fairness Metrics

4.2 Bias Testing Protocol

1. Define protected characteristics relevant to the jurisdiction and use case (race, gender, age, disability, religion, sexual orientation per Equality Act 2010 / EU Charter Art. 21).
2. Partition test dataset by protected characteristics.
3. Calculate fairness metrics for each group.
4. Identify disparate impact: if the selection rate for any protected group is less than 80% of the rate for the group with the highest rate (four-fifths rule from US EEOC, widely adopted as a starting benchmark).
5. If disparate impact is detected, assess whether the disparity is justified by a legitimate, non-discriminatory factor.
6. Document bias testing methodology, results, and remediation actions.

Phase 5: NIST AI RMF MAP Function Integration (Week 4)

The NIST AI Risk Management Framework (AI RMF 1.0, January 2023) MAP function identifies the context, capabilities, and potential impacts of AI systems. Integrate the following MAP subcategories:

Phase 6: Combined Risk Assessment and Reporting (Week 5-6)

1. Consolidate GDPR DPIA risks, AI Act compliance gaps, and NIST AI RMF findings.
2. For each risk, assess likelihood and severity using the DPIA risk matrix.
3. Identify mitigation measures combining privacy controls and AI governance controls.
4. Calculate residual risk levels.
5. If high-risk AI under the AI Act, prepare conformity assessment documentation.
6. Submit to DPO for Art. 35(2) advice.
7. If residual risk remains high, initiate Art. 36 prior consultation.

AI-Specific Privacy Risks

Common Assessment Deficiencies

1. Treating AI training as a single processing operation: Each stage (collection, preparation, training, inference) is a distinct processing operation requiring its own lawful basis assessment.
2. Relying on anonymisation claims without verification: Data claimed to be anonymised may be re-identifiable through model inversion or linkage attacks.
3. Inadequate Art. 22 human involvement: Rubber-stamp human review of AI decisions does not constitute meaningful human involvement.
4. Ignoring purpose limitation for training data: Using customer data collected for service delivery to train AI models without compatibility assessment.
5. No bias testing on protected characteristics: Fairness assessment limited to overall accuracy without demographic disaggregation.
6. Static assessment: AI systems evolve through retraining; the assessment must be updated when the model is retrained or the deployment context changes.

Enforcement Precedents

• Italian Garante vs Clearview AI (2022): EUR 20 million fine for scraping biometric data from the internet to train facial recognition AI without lawful basis, consent, or DPIA.
• Italian Garante vs Deliveroo (2021): EUR 2.5 million fine for algorithmic management of food delivery riders constituting Art. 22 automated decision-making without adequate safeguards. The algorithm assigned reputation scores and allocated delivery shifts without meaningful human review.
• Hungarian DPA vs Budapest Bank (2023): Fine for automated credit scoring system that used postcode as a proxy for ethnicity without bias impact assessment.
• CNIL vs Clearview AI (2022): EUR 20 million fine; CNIL ordered deletion of all data of French residents and prohibited further collection.
• Dutch DPA vs Tax Authority (2020): EUR 3.7 million fine for algorithmic profiling system (SyRI) that disproportionately targeted ethnic minorities in fraud detection, violating ECHR Art. 8 and Art. 14.