10  Auditing AI systems in practice

In this chapter we will cover: - Checklist for auditing AI systems from a data protection perspective - Focus on technology and not on legal compliance (work should be done with dpo or other legal experts) - Evaluating AI systems doing procurement

This chapter provides a collection of checklists and good practices gathered from international standards, peer reviewed publications, and work in progress in the broad community of artificial intelligence and privacy experts. The checklists are using the following tags to identify the audience and the type of checklist.

NoteTarget Audience Tags

Since checklists can be useful for different experts working on an AI system, these tags define the audience for the checklist.

Tag Name Explanation
AI Governance Responsible for overseeing AI policies and regulations from inception to decommissioning.
Data Engineers Handle the preparation, management, and optimization of data pipelines in the early stages of development.
ML Engineers Focused on model development, training, and optimization during the development phase.
Cybersecurity Experts Involved in securing AI infrastructure and systems from development through deployment and monitoring.
Data Privacy Officers Ensure data protection throughout the AI lifecycle, from inception through monitoring and decommissioning.
Cross-functional Teams Involved throughout the lifecycle, supporting collaboration between different roles.
NoteType of Checklist Tags

These tags define the type and scope of the checklists.

Tag Name Explanation
Ethics Considerations for fairness, transparency, and responsibility, applied throughout the entire lifecycle from inception onward.
Compliance Ensuring adherence to legal, regulatory, and industry standards, from the design phase through decommissioning.
Data Management Covering data handling, governance, and storage, primarily during data preparation and development but extending through the entire lifecycle.
System Security Focused on securing systems and software during development, deployment, and monitoring.
Data Privacy Ensuring the protection of personal data across all stages, especially during development and deployment.
Monitoring Continuous assessment of AI system performance and security during deployment and production.
Auditability Maintaining traceability and accountability throughout the lifecycle, especially during deployment, monitoring, and decommissioning.

10.1 AI Impact Assesment

NoteChecklist card
Name: AI Impact Assessment
Audience: AI Governance, Data Privacy Officers, Cross-functional Teams
Type: Compliance, Auditability, Ethics
Reference: ISO/IEC DIS 42005 AI system impact assessment, ref1
Comment: Under development

This is currently at a draft stage; the checklist outlines a structured process for assessing AI systems.

10.2 PLOT4ai

NoteChecklist card
Name: PLOT4ai
Audience: AI Governance, Data Engineers, ML Engineers, Cybersecurity Experts, Data Privacy Officers, Cross-functional Teams
Type: Compliance, Auditability, Ethics, Data Privacy
Reference: PLOT4ai
Comment: All questions can be accessed in machine-readable format at PLOT4ai Github respository

The PLOT4ai (Privacy, Lawfulness, Oversight, and Trust for AI) project provides a structured and practical framework for identifying and mitigating risks in the development and deployment of AI systems. It supports developers, researchers, and organizations in building AI responsibly by offering tools for threat modeling, risk assessment, and compliance with legal and ethical standards. Central to the project is a curated set of “threat cards,” which guide users in recognizing potential harms and implementing safeguards throughout the AI lifecycle. PLOT4ai promotes responsible AI innovation grounded in principles of data protection, transparency, and accountability.

Threat Card Categories and Usefulness:
The PLOT4ai threat cards (86 cards) are organized into several key categories that reflect different areas of concern in AI systems:

  • Privacy: Includes threats like data leakage, re-identification, and inference attacks; useful for identifying risks to personal data and ensuring GDPR compliance.
  • Security: Covers attacks like adversarial examples, model inversion, and data poisoning; essential for securing AI models and infrastructure.
  • Fairness and Bias: Highlights issues like discriminatory outcomes and representation bias; helps promote equity and inclusiveness in AI decisions.
  • Transparency and Explainability: Focuses on challenges in understanding model behavior and outputs; supports trustworthy and interpretable AI.
  • Accountability and Governance: Emphasizes documentation, oversight, and role clarity; useful for aligning with legal frameworks and organizational responsibilities.
  • Misuse and Abuse: Points to risks of dual use, malicious repurposing, and unintended consequences; aids in evaluating and preventing harm from system misuse.

These categories are useful in AI development, auditing, and teaching to promote a culture of risk awareness and proactive mitigation across technical, ethical, and legal dimensions.

To see the full list of cards, click the box below.

ID Categories Question Recommendation Sources
1 Technique & Processes Is the task or assignment completely clear? * Clearly define the problem and outcome you are optimizing for.
* Assess if your AI system will be well-suited for this purpose.
* Always discuss if there are alternative ways to solve the problem.
* Define success! Working with individuals who may be directly affected can help you identify an appropriate way to measure success.
* Make sure there is a stakeholder involved (product owner for instance) with enough knowledge of the business and a clear vision about what the model needs to do.
* Did you try analytics first? In this context analytics could also offer inspiring views that can help you decide on the next steps. They can be a good source of information and are sometimes enough to solve the problem without the need of AI/ML.
2 Technique & Processes Can we assure that the data that we need is complete and trustworthy? * Verify the data sources:
* Is there information missing within the dataset?
* Are all the necessary classes represented?
* Does the data belong to the correct time frame and geographical coverage?
* Evaluate which extra data you need to collect/receive.
* Carefully consider representation schemes, especially in cases of text, video, APIs, and sensors. Text representation schemes are not all the same. If your system is counting on ASCII and it gets Unicode, will your system recognize the incorrect encoding? Source: BerryVilleiML
3 Technique & Processes Can the data be representative of the different groups/populations? * Who is covered and who is underrepresented?
* Prevent disparate impact: when the output of a member of a minority group is disparate compared to representation of the group. Consider measuring the accuracy from minority classes too instead of measuring only the total accuracy. Adjusting the weighting factors to avoid disparate impact can result in positive discrimination which has also its own issues: disparate treatment.
* One approach to addressing the problem of class imbalance is to randomly resample the training dataset. This technique can help to rebalance the class distribution when classes are under or over represented:
- random oversampling (i.e. duplicating samples from the minority class)
- random undersampling (i.e. deleting samples from the majority class)
* There are trade-offs when determining an AI system’s metrics for success. It is important to balance performance metrics against the risk of negatively impacting vulnerable populations.
* When using techniques like statistical generalisation is important to know your data well, and get familiarised with who is and who is not represented in the samples. Check the samples for expectations that can be easily verified. For example, if half the population is known to be female, then you can check if approximately half the sample is female.		 Related to disparate impact;
AI Fairness - Explanation of Disparate Impact Remover;
Mitigating Bias in AI/ML Models with Disparate Impact Analysis;
Certifying and removing disparate impact;
Avoiding Disparate Impact with Counterfactual Distributions;
;
Related to random resampling;
Oversampling and Undersampling;
Random Oversampling and Undersampling for Imbalanced Classification;
;
Related to Statistical Generalization;
Generalization in quantitative and qualitative research: Myths and strategies;
Generalizing Statistical Results to the Entire Population
4 Technique & Processes Have we identified all the important stakeholders needed in this phase of the project? * Identify and involve on time the people that you need during the whole life cycle of the AI system. This will avoid unnecessary rework and frustrations.
* Identifying who’s responsible for making the decisions and how much control they have over the decision-making process allows for a more evident tracking of responsibility in the AI’s development process.
5 Technique & Processes Does the model need to be explainable for the users or affected persons? * Evaluate the type of models that you could use to solve the problem as specified in your task.
* Consider what the impact is if certain black box models cannot be used and interpretability tools do not offer sufficient results. You might need to evaluate a possible change in strategy.
* Data scientists can evaluate the impact from a technical perspective and discuss this with the rest of stakeholders. The decision keeps being a team effort.		 Explainable Artificial Intelligence (XAI);
LIME;
Why Should I Trust You? Explaining the Predictions of Any Classifier;
SHAP and LIME: An Evaluation of Discriminative Power in Credit Risk;
Explainable AI;
Explainable AI - The TAILOR Handbook of Trustworthy AI
6 Technique & Processes Are we preventing Data Leakage? * Avoid using proxies for the outcome variable as a feature.
* Do not use the entire data set for imputations, data-based transformations or feature selection.
* Avoid doing standard k-fold cross-validation when you have temporal data.
* Avoid using data that happened before model training time but is not available until later. This is common where there is delay in data collection.
* Do not use data in the training set based on information from the future: if X happened after Y, you shouldn’t build a model that uses X to predict Y.		 Leakage in data mining: formulation, detection, and avoidance;
The Treachery of Leakage;
Top 10 ways your Machine Learning models may have leakage;
Leakage and the Reproducibility Crisis in ML-based Science
7 Technique & Processes Are we preventing Concept and Data Drift? * Select an appropriate drift detection algorithm and apply it separately to labels, model’s predictions and data features.
* Incorporate monitoring mechanisms to detect potential errors early.		 Data Drift vs. Concept Drift;
Characterizing Concept Drift;
Inferring Concept Drift Without Labeled Data;
Automatic Learning to Detect Concept Drift;
From concept drift to model degradation: An overview on performance-aware drift detectors;
Learning under Concept Drift: A Review;
Detect data drift (preview) on datasets
8 Technique & Processes Once our model is running, can we keep feeding it data? * Consider how the model will keep learning. Design a strategy to prevent issues with the next steps.
* Imagine you planned to feed your model with input obtained by mining surveys and it appears these surveys contain a lot of free text fields. To prepare that data and avoid issues (bias, inaccuracies, etc) you might need extra time. Consider these type of scenarios that could impact the whole life cycle of your product!		 Text Mining in Survey Data
9 Technique & Processes Is human intervention necessary to oversee the automatic decision making (ADM) process of the AI system? It is important that people are available for this role and that they receive specific training on how to exercise oversight. The training should teach them how to perform the oversight without being biased by the decision of the AI system (automation bias). Automation Bias;
The Flaws of Policies Requiring Human Oversight of Government Algorithms;
The False Comfort of Human Oversight as an Antidote to AI Harm
10 Technique & Processes, Security Could the channels that we will use to collect real-time input data fail? * If you are collecting/receiving data from sensors, consider estimating the impact it could have on your model if any of the sensors fail and your input data gets interrupted or corrupted.
* Sensor blinding attacks are one example of a risk faced by poorly designed input gathering systems. Note that consistent feature identification related to sensors is likely to require human calibration. Source: BerryVilleiML
11 Technique & Processes When datasets from external sources are updated, can we receive and process the new data on time? Not only do you need to be able to trust the sources but you also need to design a process in which data is prepared on time to be used in the model and where you can timely consider the impact it could have in the output of the model, especially when this could have a negative impact on the users. This process can be designed once you know how often changes in the data can be expected and how big the changes are.
12 Technique & Processes Can we confirm the legitimacy of the data sources that we need? * Develop a robust understanding of your relevant data feeds, flows and structures such that if any changes occur to the model data inputs, you can assess any potential impact on model performance. In case of third party AI systems contact your vendor to ask for this information.
* If you are using synthetic data you should know how it was created and the properties it has. Also keep in mind that synthetic data might not be the answer to all your privacy related problems; synthetic data does not always provide a better trade-off between privacy and utility than traditional anonymisation techniques.
* Do you need to share models and combine them? The usage of Model Cards and Datasheets can help providing the source information.		 Providing Assurance and Scrutability on Shared Data and Machine Learning Models with Verifiable Credentials;
Synthetic Data – Anonymisation Groundhog Day;
Model Cards;
Model Cards for Model Reporting;
Datasheets for Datasets
13 Technique & Processes Do we have enough dedicated resources to monitor the algorithm? * Put a well-defined process in place to monitor if the AI system is meeting the intended goals.
* Define failsafe fallback plans to address AI system errors of whatever origin and put governance procedures in place to trigger them.
* Put measure in places to continuously assess the quality of the output data: e.g. check that predictions scores are within expected ranges; anomaly detection in output and reassign input data leading to the detected anomaly.
* Does the data measure what you need to measure? You could get measurement errors if data is not correctly labelled.
14 Technique & Processes Can we collect all the data that we need for the purpose of the algorithm? In the early phases of the project (as soon as the task becomes more clear), start considering which raw data and types of datasets you might need. You might not have the definitive answer until you have tested the model, but it will already help to avoid extra delays and surprises. You might have to involve your legal and financial department. Remember that this is a team effort.
15 Accessibility Can our system’s user interface be used by those with special needs or disabilities? * Implement Universal Design principles during every step of the planning and development process. This is not only important for web interfaces but also when AI systems/robots assist individuals.
* Test the accessibility of your design with different users (also with disabilities).		 A Proposal of Accessibility Guidelines for Human-Robot Interaction;
ISO/IEC 40500:2012 Information technology — W3C Web Content Accessibility Guidelines (WCAG) 2.0;
ISO/IEC GUIDE 71:2001 Guidelines for standards developers to address the needs of older persons and persons with disabilities;
ISO 9241-171:2008(en) Ergonomics of human-system interaction;
Mandate 376 Standards EU
16 Accessibility, Non-compliance Do we need to offer a redress mechanism to the users? * Think about implementing mechanisms to effectively detect and rectify wrong decisions made by your system.
* Provide a mechanism to ignore or dismiss undesirable features or services.
* Wrong decisions could also have an impact on people that have not been the target of the data collection (data spillovers). Consider designing a way to offer all affected people the opportunity to contest the decisions of your system and request remedy or compensation. This mechanism should be easily accessible and it implies that you would need to have internally implemented a process where redress can be effectibily executed. This also has impact on the resources/skills needed to fulfil this process.
* Consider this a necessary step to ensure responsibility and accountability.		 EU guidelines on ethics in artificial intelligence: Context and implementation
17 Accessibility, Technique & Processes Do we need to implement an age gate to use our product? * Clearly specify in the user instructions for which age group the application is built. Labels or symbols can be very helpful.
* Consider which design is more appropriate based on your use case, and consider the possible risks associated with your design choice, and the mitigating measures you can implement to reduce that risk. Document the rest risks that you want to accept.
* Test the accessibility of your design with different age groups.		 Evolution in Age-Verification Applications
18 Accessibility, Non-compliance If users need to provide consent, can we make the required information easily available? * As part of privacy compliance you need to provide clear information about the processing and the logic of the algorithm. This information should be easily readable and accessible. During the design phase consider when and how you are going to provide this information. Especially in robots using AI this could be a challenge.
* Comply with accessibility rules.		 A Proposal of Accessibility Guidelines for Human-Robot Interaction;
ISO/IEC 40500:2012 Information technology — W3C Web Content Accessibility Guidelines (WCAG) 2.0;
ISO/IEC GUIDE 71:2001 Guidelines for standards developers to address the needs of older persons and persons with disabilities;
ISO 9241-171:2008(en) Ergonomics of human-system interaction;
Mandate 376 Standards EU
19 Accessibility, Unawareness Could the user perceive the message from the AI system in a different way than intended? * Understanding who is going to interact with the AI system can help to make the interaction more effective. Identify your different user groups.
* Involve communication experts and do enough user testing to reduce the gap between the intended and the perceived meaning.		 The Who in Explainable AI: How AI Background Shapes Perceptions of AI Explanations
20 Accessibility, Safety Could the learning curve of the product be an issue? * You can also provide assistance, appropriate training material and disclaimers to users on how to adequately use the system.
* The words and language used in the interface, the complexity and lack of accessibility of some features could exclude people from using the application. Consider making changes in the design of the product where necessary.
* Consider this also when children are future users.
21 Identifiability & Linkability Can the data used to feed the model be linked to individuals? * Unique identifiers might be included in the training set when you want to be able to link the results to individuals. Consider using pseudo-identifiers or other techniques that can help you protect personal data.
* Document the measures you are taking to protect the data. Consider if your measures are necessary and proportional.
22 Identifiability & Linkability Could actions be incorrectly attributed to an individual or group? * Evaluate the possible consequences of inaccuracies of your AI system and implement measures to prevent these errors from happening: avoiding bias and discrimination during the life cycle of the model, ensuring the quality of the input data, implementing a strict human oversight process, ways to double check the results with extra evidence, implementing safety and redress mechanisms, etc.
* Assess the impact on the different human rights of the individual.
* Consider not to implement such a system if you cannot mitigate the risks.
23 Identifiability & Linkability Could we be revealing information that a person has not chosen to share? * Be careful when making data public that you think is anonymised. Location data and routes can sometimes be de-anonymised (e.g. users of a running app disclosing location by showing heatmap).
* It is also important to offer privacy by default: offer the privacy settings by default at the maximum protection level. Let the users change the settings after having offered them clear information about the consequences of reducing the privacy levels.
24 Security Do we need to red-team/pen test the AI system? Include the time you might need for a pen test in your project planning. Sometimes this can take weeks: you might have to hire an external party, agree on the scope, sign the corresponding agreements and even plan a retest. Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
25 Security Are our APIs securely implemented? * Check how do you handle time and state and how is authentication implemented in your APIs.
* Make sure that sensitive information such us API calls secrets are not sent in your commands.
* Implement encryption at rest and in transit (TLS) and test often your APIs for vulnerabilities.		 OWASP API Security Project;
BerryVilleiML;
Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
26 Security Is our data storage protected? * Implement access control rules.
* Verify the security of the authentication mechanism (and the system as a whole).
* Consider the risk when utilizing public/external data sources.		 Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
27 Security If our AI system uses randomness, is the source of randomness properly protected? Use of cryptographic randomness sources is encouraged. When it comes to machine learning (ML), setting weights and thresholds “randomly” must be done with care. Many pseudo-random number generators (PRNG) are not suitable for use. PRNG loops can really damage system behaviour during learning. Cryptographic randomness directly intersects with ML when it comes to differential privacy. Using the wrong sort of random number generator can lead to subtle security problems.
Source: BerryVilleiML		 Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
28 Security Is our model suited for processing confidential information? When selecting the algorithm perform analyses and test to rule out algorithmic leakage. Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
ICO - How should we assess security and data minimisation in AI?;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
29 Security Can our AI system scale in performance from data input to data output? * Find out what the rate would be of expected data arrival to your model and perform tests in a similar environment with similar amount of data input.
* Implement measures to make your model scalable.		 Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
30 Security Are we protected from insider threats? * Implement on and off boarding procedures to help guarantee the trustworthiness of your internal and external employees.
* Enforce separation of duties and least privilege principle.
* Enforce the usage of managed devices with appropriate policies and protective software.
* Awareness training.
* Implement strict access control and audit trail mechanisms.		 Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
31 Security Are we protected against model sabotage? * Implement security measures to protect your models against sabotage.
* Assess the security profile of third party tooling and providers.
* Consider implementing a disaster recovery plan with mitigation measures for this type of attack.		 Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
32 Security, Safety Could there be possible malicious use, misuse or inappropriate use of our AI system? * Threat model your system: anticipate vulnerabilities and look for ways to hijack and weaponize your system for malicious activity.
* Conduct red team exercises.		 MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
33 Security Could environmental phenomena or natural disasters have a negative impact on our AI system? Implement a disaster discovery plan considering different scenarios, impact, Recovery Time Objective (RTO), Recovery Point Objective (RPO) and mitigation measures. Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
34 Security Are we protected from perturbation attacks? Reactive/Defensive Detection Actions:
* Implement a minimum time threshold between calls to the API providing classification results. This slows down multi-step attack testing by increasing the overall amount of time required to find a success perturbation.

Proactive/Protective Actions:
* Develop a new network architecture that increases adversarial robustness by performing feature denoising.
* Train with known adversarial samples to build resilience and robustness against malicious inputs.
* Invest in developing monotonic classification with selection of monotonic features. This ensures that the adversary will not be able to evade the classifier by simply padding features from the negative class.
* Feature squeezing can be used to harden DNN models by detecting adversarial examples.

Response Actions:
* Issue alerts on classification results with high variance between classifiers, especially when from a single user or small group of users.

Source: Microsoft, Threat Modelling AI/ML Systems and Dependencies.		 Microsoft, Threat Modelling AI/ML Systems and Dependencies;
Adversarially Robust Malware Detection Using Monotonic Classification;
Reinforcing Adversarial Robustness using Model Confidence Induced by Adversarial Training;
Attribution-driven Causal Analysis for Detection of Adversarial Examples;
Feature Denoising for Improving Adversarial Robustness;
Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
35 Security Are we protected from poisoning attacks? * Define anomaly sensors to look at data distribution on day to day basis and alert on variations.
* Measure training data variation on daily basis, telemetry for skew/drift.
* Input validation, both sanitization and integrity checking.

Source: Microsoft, Threat Modelling AI/ML Systems and Dependencies.

* Implement measures against insider threats.		 Microsoft, Threat Modelling AI/ML Systems and Dependencies;
Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
Robustness Techniques & Toolkits for Applied AI;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
36 Security Are we protected from model inversion attacks? * Interfaces to models trained with sensitive data need strong access control.
* Implement rate-limiting on the queries allowed by the model.
* Implement gates between users/callers and the actual model by performing input validation on all proposed queries, rejecting anything not meeting the model’s definition of input correctness and returning only the minimum amount of information needed to be useful.

Source: Microsoft, Threat Modelling AI/ML Systems and Dependencies.		 Microsoft, Threat Modelling AI/ML Systems and Dependencies;
Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
37 Security Are we protected from membership inference attacks? * Some research papers indicate Differential Privacy would be an effective mitigation. Check for more information Threat Modeling AI/ML Systems and Dependencies.
* The usage of neuron dropout and model stacking can be effective mitigations to an extent. Using neuron dropout not only increases resilience of a neural net to this attack, but also increases model performance.

Source: Microsoft, Threat Modelling AI/ML Systems and Dependencies.		 Microsoft, Threat Modelling AI/ML Systems and Dependencies;
Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
38 Security Are we protected from model stealing attacks? * Minimize or obfuscate the details returned in prediction APIs while still maintaining their usefulness to honest applications.
* Define a well-formed query for your model inputs and only return results in response to completed, well-formed inputs matching that format.
* Return rounded confidence values. Most legitimate callers do not need multiple decimal places of precision.

Source: Microsoft, Threat Modelling AI/ML Systems and Dependencies.		 Microsoft, Threat Modelling AI/ML Systems and Dependencies;
Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
39 Security Are we protected from reprogramming deep neural nets attacks? * Configure a strong client-server mutual authentication and access control to model interfaces.
* Takedown of the offending accounts.
* Identify and enforce a service-level agreement for your APIs. Determine the acceptable time-to-fix for an issue once reported and ensure the issue no longer repros once SLA expires.

Source: Microsoft, Threat Modelling AI/ML Systems and Dependencies.		 Microsoft, Threat Modelling AI/ML Systems and Dependencies;
Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
40 Security Are we protected from adversarial example? These attacks manifest themselves because issues in the machine learning layer were not mitigated. As with any other software, the layer below the target can always be attacked through traditional vectors. Because of this, traditional security practices are more important than ever, especially with the layer of unmitigated vulnerabilities (the data/algo layer) being used between AI and traditional software.
Source: Microsoft, Threat Modelling AI/ML Systems and Dependencies.		 Microsoft, Threat Modelling AI/ML Systems and Dependencies;
Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
41 Security Are we protected from malicious AI/ML providers who could recover training data? * Research papers demonstrating the viability of this attack indicate Homomorphic Encryption could be an effective mitigation. Check for more information Threat Modeling AI/ML Systems and Dependencies
* Train all sensitive models in-house.
* Catalog training data or ensure it comes from a trusted third party with strong security practices.
* Threat model the interaction between the MLaaS provider and your own systems.

Source: Microsoft, Threat Modelling AI/ML Systems and Dependencies.		 Microsoft, Threat Modelling AI/ML Systems and Dependencies;
Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
42 Security Are we protected from attacks to the AI/ML Supply Chain? * Minimize 3rd-party dependencies for models and data where possible.
* Incorporate these dependencies into your threat modeling process.
* Leverage strong authentication, access control and encryption between 1st/3rd-party systems.

Source: Microsoft, Threat Modelling AI/ML Systems and Dependencies.
* Perform integrity checks where possible to detect tampering.		 Microsoft, Threat Modelling AI/ML Systems and Dependencies;
Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
43 Security Are we protected from exploits on software dependencies of our AI/ML systems? Work with your security team to follow applicable Security Development Lifecycle/Operational Security Assurance best practices. Source: Microsoft, Threat Modelling AI/ML Systems and Dependencies. Microsoft, Threat Modelling AI/ML Systems and Dependencies;
Securing Machine Learning Algorithms, ENISA;
STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets;
Stride-ML Threat Model;
MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems
44 Safety In case of system failure, could users be adversely impacted? * Implement some kind of stop button or procedure to safely abort an operation when needed.
* Establish a detection and response mechanism for undesirable adverse effects on individuals.
* Define criticality levels of the possible consequences of faults/misuse of the AI system: what type of harm could be caused to the individuals, environment or organisations?
45 Safety Could our AI system have an adverse impact on the environment? * Establish mechanisms to evaluate the environmental impact of your AI system; for example, the amount of energy used and carbon emissions.
* Implement measures to reduce the environmental impact of the AI system throughout its lifecycle.		 UNSDGs United Nations Sustainable Development goals
46 Safety Could our model be deployed in a different context? * Use different data for testing and training. Make sure diversity is reflected in the data. Specify your training approach and statistical method. Explore the different environments and contexts and make sure your model is trained with the expected different data sources. This also applies to reinforcement learning.
* Are you considering enough aspects in the environment? Did you forget any environmental variable that could be harmful? Could limited sampling due to high costs be an issue? Document this risk and look for support in your organisation. The organisation is accountable and responsible for the mitigation or acceptance of this risk. And hopefully you get extra budget assigned.
* Consider applying techniques such as cultural effective challenge; this is a technique for creating an environment where technology developers can actively participate in questioning the AI process. This better translates the social context into the design process by involving more people and can prevent issues associated with target leakage where the AI system trains on data that prepares it for an alternative job other than the one it was initially intended to complete.		 Information about cultural effective challenge: A Proposal for Identifying and Managing Bias in Artificial Intelligence
47 Safety Could the AI system become persuasive causing harm to the individual? * Signals of susceptibility coming from a robot or computer could have an impact on the willingness of humans to cooperate or take advice from it.
* It is important to consider and test this possible scenario when your AI system is interacting with humans and some type of collaboration/cooperation in expected.		 The role of reciprocity in human-robot social influence;
Reciprocity in Human-Robot Interaction;
Social robots and the risks to reciprocity
48 Safety Could our RL agents develop strategies that could have undesired negative side effects on the environment? AI systems don’t share our understanding of the world. It is not sufficient to formulate the objective as “complete task X”; the designer also needs to specify the safety criteria under which the task is to be completed. A better strategy could be to define a budget for how much the AI system is allowed to impact the environment. This would help to minimize the unintended impact, without neutralizing the AI system.

Another approach would be training the agent to recognize harmful side effects so that it can avoid actions leading to such side effects. In that case, the agent would be trained for two tasks: the original task that is specified by the objective function and the task of recognizing side effects.
The AI system would still need to undergo extensive testing and critical evaluation before deployment in real life settings.
Source: OpenAI		 Concrete Problems in AI Safety;
Concrete AI Safety Problems
49 Safety Could our RL agents “hack” their reward functions? One possible approach to mitigating this problem would be to have a “reward agent” whose only task is to mark if the rewards given to the learning agent are valid or not. The reward agent ensures that the learning agent (robot for instance) does not exploit the system, but rather, completes the desired objective. For example: a “reward agent” could be trained by the human designer to check if a room has been properly cleaned by the cleaning robot. If the cleaning robot shuts off its visual sensors to avoid seeing garbage and claims a high reward, the “reward agent” would mark the reward as invalid because the room is not clean. The designer can then look into the rewards marked as “invalid” and make necessary changes in the objective function to fix the loophole. Source: OpenAI Concrete Problems in AI Safety;
Concrete AI Safety Problems
50 Safety Can we provide human resources to supervise and give feedback every time the RL agent performs an action? One promising research direction to tackle this problem is semi-supervised learning, where the agent is still evaluated on all the actions (or tasks), but receives rewards only for a small sample of those actions (or tasks).

Another promising research direction is hierarchical reinforcement learning, where a hierarchy is established between different learning agents. There could be a supervisor agent/robot whose task is to assign some work to another agent/robot and provide it with feedback and rewards.
Source: OpenAI		 Concrete Problems in AI Safety;
Concrete AI Safety Problems
51 Safety Can our AI/ML system be robust to changes in the data distribution? One promising research direction focuses on identifying when the agent has encountered a new scenario so that it recognizes that it is more likely to make mistakes. While this does not solve the underlying problem of preparing AI systems for unforeseen circumstances, it helps in detecting the problem before mistakes happen. Another direction of research emphasizes transferring knowledge from familiar scenarios to new scenarios safely. Source: OpenAI Concrete Problems in AI Safety;
Concrete AI Safety Problems
52 Safety Can our RL agents learn about their environment without causing harm or catastrophic actions? One approach to reduce harm is to optimize the performance of the learning agent in the worst case scenario. When designing the objective function, the designer should not assume that the agent will always operate under optimal conditions. Some explicit reward signal may be added to ensure that the agent does not perform some catastrophic action, even if that leads to more limited actions in the optimal conditions. Source: OpenAI Concrete Problems in AI Safety;
Concrete AI Safety Problems
53 Unawareness, Accessibility Do we need to inform users that they are interacting with an AI system? In cases of interactive AI systems (e.g., chatbots, robots) you should inform the users that they are interacting with an AI system instead of a human. This information should be received at the beginning of the interaction.
54 Unawareness, Accessibility Can we provide the necessary information to the users about possible impacts, benefits and potential risks? * Provide clear information about how and why an AI-assisted decision was made and which personal data was used to train and test the model.
* The model you choose should be at the right level of interpretability for your use case and for the impact it will have on the decision recipient. If you use a black box model make sure the supplementary explanation techniques you use provide a reliable and accurate representation of the systems behaviour. Source: UK ICO
* Communicate the benefits, the technical limitations and potential risks of the AI system to users, such as its level of accuracy and/or error rates.
* Ask your users (with a survey for instance) if they understand the decisions that your product makes.
55 Unawareness, Safety Can users anticipate the actions of the AI system? * Consider this as part of the GDPR transparency principle.
* Users should be aware of what the AI system can do.
* Clear Information should be provided on time and made accessible following accessibility design principles.		 GDPR transparency principle
56 Ethics & Human Rights, Technique & Processes Bias & Discrimination: could there be groups who might be disproportionately affected by the outcomes of the AI system? * Consider the different types of users and contexts where your product is going to be used.
* Consider the impact of diversity of backgrounds, cultures, and other important different attributes when selecting your input data, features and when testing the output.
* Assess the risk of possible unfairness towards individuals or communities to avoid discriminating minority groups.
* The disadvantage to people depends on the kind of harm, severity of the harm and significance (how many people are put at a disadvantage compared to another group of people). Statistical assessments on group differences are an important tool to assess unfair and discriminatory uses of AI.
* Design with empathy, diversity and respect in mind.		 Why Fairness Cannot Be Automated: Bridging the Gap Between EU Non-Discrimination Law and AI;
The Fairness Handbook
57 Ethics & Human Rights Can we expect mostly positive reactions from the users or individuals? * Consider the different types of users and contexts your product is going to be used.
* Consider diversity of backgrounds, cultures, and many other important different attributes.
* Do enough user testing, like FUPs - Friendly User Pilots.
* Design with empathy, diversity and respect in mind.
* Assess the risk of possible unfairness towards individuals or communities to avoid discriminating minority groups and also to prevent a bad reputation for your organisation.
58 Ethics & Human Rights Could the AI system have an impact on human work? * Pave the way for the introduction of the AI system in your organisation by informing and consulting with future impacted workers and their representatives (e.g. trade unions, work councils) in advance.
* Adopt measures to ensure that the impact of the AI system on human work is well understood.
* Ensure that workers understand how the AI system operates, which capabilities it has and does not have. Provide workers with the necessary safety instructions (e.g. when using machine-robots).
* If you are a third party provider of this type of systems, provide information related to this possible risk to your customers. This information should be easily accessible and understandable.
59 Ethics & Human Rights Could the AI system have an adverse impact on society at large? * Consider if your product could be used or misused for this purposes. Maybe it is not possible in the way it currently is but it could be possible with adaptations.
* Evaluate the possible scenarios and think what role you want to play based on the responsibility and accountability principle.
* How can you prevent something like that from happening?
* Does your organisation agree with such a use of the technology?
* Have you evaluated what the possible impact could be for society and the world you live in?
60 Ethics & Human Rights Could the AI system limit the right to be heard? It is very important that your product complies with the key EU requirements for achieving a trustworthy AI:
* human agency and oversight
* robustness and safety
* privacy and data governance
* transparency
* diversity, non-discrimination and fairness
* societal and environmental well-being
* accountability

Remember that there are other human rights that could be affected by your product. Check the other rights in the Charter of Fundamental Rights:		 Ethics guidelines for trustworthy AI;
Operational Guidance on taking account of Fundamental Rights in Commission Impact Assessments;
Artificial Intelligence and Fundamental Rights
61 Ethics & Human Rights Could the system have a big impact on decisions regarding the right to life? It is very important that your product complies with the key EU requirements for achieving a trustworthy AI:
* human agency and oversight
* robustness and safety
* privacy and data governance
* transparency
* diversity, non-discrimination and fairness
* societal and environmental well-being
* accountability

Remember that there are other human rights that could be affected by your product. Check the other rights in the Charter of Fundamental Rights:		 Ethics guidelines for trustworthy AI;
Operational Guidance on taking account of Fundamental Rights in Commission Impact Assessments;
Artificial Intelligence and Fundamental Rights
62 Ethics & Human Rights Could the AI system affect the freedom of expression of its users? It is very important that your product complies with the key EU requirements for achieving a trustworthy AI:
* human agency and oversight
* robustness and safety
* privacy and data governance
* transparency
* diversity, non-discrimination and fairness
* societal and environmental well-being
* accountability

Remember that there are other human rights that could be affected by your product. Check the other rights in the Charter of Fundamental Rights:		 Ethics guidelines for trustworthy AI;
Operational Guidance on taking account of Fundamental Rights in Commission Impact Assessments;
Artificial Intelligence and Fundamental Rights
63 Ethics & Human Rights Could the AI system affect the freedom of its users? It is very important that your product complies with the key EU requirements for achieving a trustworthy AI:
* human agency and oversight
* robustness and safety
* privacy and data governance
* transparency
* diversity, non-discrimination and fairness
* societal and environmental well-being
* accountability

Remember that there are other human rights that could be affected by your product. Check the other rights in the Charter of Fundamental Rights:		 Ethics guidelines for trustworthy AI;
Operational Guidance on taking account of Fundamental Rights in Commission Impact Assessments;
Artificial Intelligence and Fundamental Rights
64 Ethics & Human Rights Could the AI system affect the right to a fair hearing? It is very important that your product complies with the key EU requirements for achieving a trustworthy AI:
* human agency and oversight
* robustness and safety
* privacy and data governance
* transparency
* diversity, non-discrimination and fairness
* societal and environmental well-being
* accountability

Remember that there are other human rights that could be affected by your product. Check the other rights in the Charter of Fundamental Rights:		 Ethics guidelines for trustworthy AI;
Operational Guidance on taking account of Fundamental Rights in Commission Impact Assessments;
Artificial Intelligence and Fundamental Rights
65 Ethics & Human Rights Could children be part of our users’ group? * Check if an age verification mechanism is necessary.
* Pay attention to the way of communication in the product but also in your privacy policy.
* Implement policies to ensure the safety of children when using or being exposed to your products.
* Implement procedures to assess and monitor the usage of your product, this can help you identify any dangers (mental, moral or physical) to children’s health and safety.
* Label your product properly and provide the right instructions for the children’s safety.
* Monitor possible inappropriate usage of your products to abuse, exploit or harm children.
* Implement a responsible marketing and advertising policy that prohibits harmful and unethical advertising related to children.		 Convention on the Rights of the Child
66 Ethics & Human Rights Can our AI system represent different norms and values without creating ambiguity? * Consider designing with value alignment, what means that you want to ensure consideration of existing values and sensitivity to a wide range of cultural norms and values.
* Make sure that when you test the product you include a large diversity in type of users.
* Think carefully about what diversity means in the context where the product is going to be used.
* Remember that this is a team effort and not an individual decision!		 Value alignment;
Online Ethics Canvas;
AI Values and Alignment
67 Ethics & Human Rights Could our AI system not be representing current social needs and social context? Make sure that you are using correct, complete, accurate and current data. Also make sure that you have sufficient data to represent all possible contexts that you might need.
68 Ethics & Human Rights Could our AI system have an impact denying access to jobs, housing, insurance, benefits or education? It is very important that your product complies with the key EU requirements for achieving a trustworthy AI:
* human agency and oversight
* robustness and safety
* privacy and data governance
* transparency
* diversity, non-discrimination and fairness
* societal and environmental well-being
* accountability

Remember that there are other human rights that could be affected by your product. Check the other rights in the Charter of Fundamental Rights:		 Ethics guidelines for trustworthy AI;
Operational Guidance on taking account of Fundamental Rights in Commission Impact Assessments;
Artificial Intelligence and Fundamental Rights
69 Ethics & Human Rights Could our AI system affect human autonomy by interfering with the user’s decision-making process in an unintended and undesirable way? * Consider the possibility of your product affecting the behaviour and the freedom of choice of individuals.
* Test the system with enough and varied groups of users.
* Consult with experts; this is a team effort and it is very important that adverse impacts to individuals are prevented.		 Dispositional and Situational Attributions of Human Versus Robot Behaviour;
Understanding Human Over-Reliance on Technology
70 Ethics & Human Rights Does the labelling of our training data respect the dignity and well-being of the labour force involved? Verify the sources of your datasets and who has been responsible for the labelling process. Does your organisation support this unfair practices? Think in ways to help prevent this. The exploited labor behind AI
71 Ethics & Human Rights Are we going to collect and use behavioural data to feed the AI system? * Consider the way you label certain behaviours and the consequences it could have on the final output and eventually on the individuals. How do you decide which behaviours are good or bad?
* Consider diversity of opinion and possible ethical considerations.
* Consider if you will be able to collect enough information to decide which behaviours you are aiming for and which behaviours you are trying to avoid.
72 Ethics & Human Rights Could our AI system automatically label or categorize people? * It is important that you check the output of your model, not only in isolation but also when this is linked to other information. Think in different possible scenarios that could affect the individuals. Is your output categorizing people or helping to categorize them? In which way? What could be the impact?
* Think about ways to prevent adverse impact to the individual: provide information to the user, consider changing the design (maybe using different features or attributes?), consider ways to prevent misuse of your output, consider not to release the product to the market.
73 Non-compliance, Technique & Processes Is data minimisation possible? * Sometimes data minimisation can be achieved by using less features and training data that is of good quality. However it is not always possible to predict which data elements are relevant to the objective of the system.
* Consider to start training the model with less data, observe the learning curve and add more data if necessary, thereby justifying why it was necessary.
* The usage of a large amount of data could be compensated by using pseudonymisation techniques, or techniques like perturbation, differential privacy in pre-processing, use of synthetic data and federated learning.
* Try to select the right amount of features with the help of experts to avoid Curse of dimensionality (which means that errors increase with an increase in the number of features)		 pag. 13 Artificial Intelligence and Data Protection How the GDPR Regulates AI;
Data Minimization for GDPR Compliance in Machine Learning Models: Methods like the one proposed in this paper can inspire you to find a way to mitigate the accuracy risk. They show how to reduce the amount of personal data needed to perform predictions, by removing or generalizing some of the input features.;
The answer to this post also contains information about this problems in different models: Does Dimensionality curse effect some models more than others?;
Towards Breaking the Curse of Dimensionality for High-Dimensional Privacy
74 Non-compliance, Technique & Processes Could we be processing sensitive data? * If you need to use special categories of data as defined in the GDPR art. 9, then you need to check if you have the right lawful basis to do this.
* Applying techniques like anonymisation might still not justify the fact that you first need to process the original data. Check with your privacy/legal experts.
* Prevent proxies that could infer sensitive data (especially from vulnerable populations).
* Check if historical data/practices might bias your data.
* Identify and remove features that are correlated to sensitive characteristics.
* Use available methods to test for fairness with respect to different affected groups.		 AI Fairness 360;
What-if Tool: Playing with AI Fairness;
Hunting for Discriminatory Proxies in Linear Regression Models;
Differential Privacy Blog Series
75 Non-compliance Do we have a lawful basis for processing the personal data? In the case of the GDPR you need to be able to apply one of the six available legal grounds for processing the data (art. 6). Check with your privacy expert, not being able to apply one of the legal grounds could bring the project in danger.

Take into account, that also other laws besides the GDPR could be applicable.		 Lawful basis for processing;
Artificial Intelligence and Data Protection How the GDPR Regulates AI
76 Non-compliance Is the creation of the AI system proportional to the intended goal? * Proportionality requires that advantages due to limiting the right are not outweighed by the disadvantages to exercise the right. In other words, the limitation on the right must be justified.
* Safeguards accompanying a measure can support the justification of a measure. A pre-condition is that the measure is adequate to achieve the envisaged objective.
* In addition, when assessing the processing of personal data, proportionality requires that only that personal data which is adequate and relevant for the purposes of the processing is collected and processed. Source: EDPS		 Proportionality test: EDPS Guidelines on assessing the proportionality of measures that limit the fundamental rights to privacy and to the protection of personal data;
;
Assess the possible impact on human rights: Charter of Fundamental Rights of the European Union
77 Non-compliance Can we comply with the purpose limitation principle? Check with your privacy officer what the original purpose of the data was and if there are any possible constraints.
78 Non-compliance Can we comply with all the applicable GDPR data subjects’ rights? * Complying with these provisions from the GDPR (art. 15-21) could have an impact on the design of your product. What if users withdraw their consent? Do you need to delete their data used to train the model? What if you cannot identify the users in the datasets anymore? And what information should the users have access to?
* Consider all these possible scenarios and involve your privacy experts early in the design phase.
79 Non-compliance Have we considered the need to start with a data protection impact assessment (DPIA)? * This threat modeling library can help you to assess possible risks.
* Remember that a DPIA is not a piece of paper that needs to be done once the product is in production. The DPIA starts in the design phase by finding and assessing risks, documenting them and taking the necessary actions to create a responsible product from day one until it is finalized.
* Consider the time and resources that you might need for the execution of a DPIA, as it could have some impact on your project deadlines.
80 Non-compliance Do we use third party providers and are we processing data from children or other type of vulnerable people? * Check which data your third party applications are collecting and if you have the right agreements in place.
* Sometimes you can change the configuration of a tool to avoid sending certain data, or you can protect that data with pseudonymisation/anonymisation techniques.
* Consider stop using some of your third party providers, evaluate also the impact it could have on your organisation.
81 Non-compliance Do we need to use metadata to feed our model? * Make sure you are allowed to use this data.
* Verify the data sources.
* Consider using anonymisation techniques.
82 Non-compliance Will our AI system make automatic decisions without human intervention? * Check with your privacy expert if your processing falls under art. 22 GDPR or under the exceptions. Human oversight could be a way to mitigate certain risks for individuals. Discuss this with your legal advisors and the rest of the team.
* Article 22(3) also provides individuals with a right to obtain human intervention in decisions made by AI and the right to contest the decision.
* Implement specific oversight and control measures to oversee (and audit) the self-learning or autonomous nature of the AI system.
* Remember that transparency, human agency, oversight and accountability are key principles for trustworthy AI.		 Ethics guidelines for trustworthy AI
83 Non-compliance Could our dataset have copyright or other legal restrictions? * Consider if you also need to claim ownership or give credits to creators.
* Think about trademarks, copyrights in databases or training data, patents, license agreements that could be part of the dataset, library or module that you are using.
* Legal ownership of digital data can sometimes be complex and uncertain so get the proper legal advise here.
84 Non-compliance, Security Are we planning to use a third party AI tool? If personal data is involved, review which ones are your responsibilities (look into art. 24 and 28 GDPR).

You can also start by checking:
* That you have the right agreements in place with the third party provider.
* That the origin and data lineage of their datasets are verified.
* How their models are fed; do they anonymise the data?
* How you have assessed their security, ethical handling of data, quality process and ways to prevent bias and discrimination in their AI system.
* That you have informed users accordingly.
85 Non-compliance Could we have geolocation restrictions for implementing our AI system in other countries? There is no AI international regulatory environment and there are more and more new regulations that are being enforced in different countries. Keep up to date!
86 Non-compliance, Technique & Processes Can we comply with the storage limitation principle? * Personal data must not be stored longer than necessary for the intended purpose. (art.5 e GDPR). In order to comply with this principle it is important to have a clear overview of the data flow during the life cycle of the model.
* You might receive raw data that you need to transform. Check what are you doing with this data and all the different types of input files you might be receiving/collecting.
* Check if you need to store that data for quality and auditing purposes.
* Check where are you going to store the data from the data preparation, the training and test sets, the outputs, the processed outputs (when they are merged or linked to other information), metrics, etc.
* How long should all this data be stored? What type of deletion process can you put in place? And who will be responsible for the retention and deletion of this data?
* Implement the right retention schedules when applicable. In case you might still need a big part of the data in order to feed the model, consider anonymising the data.
* Deleting data from a trained model can be challenging to carry out (short of retraining the model from scratch from a dataset with the deleted data removed, but that is expensive and often infeasible). Note that through the learning process, input data are always encoded in some way in the model itself during training. That means the internal representation developed by the model during learning (say, thresholds and weights) may end up being legally encumbered as well. Source: BerryvilleiML

10.3 BayLDA AI Checklist

NoteChecklist card
Name: BayLDA AI Checklist
Audience: AI Governance, Data Engineers, ML Engineers, Cybersecurity Experts, Data Privacy Officers, Cross-functional Teams
Type: Compliance, Auditability, Data Privacy, Ethics, Data Management, System Security, Monitoring
Reference: BayLDA AI Checklist
Comment: Checklist based on the Bavarian Data Protection Authority (BayLDA), aligned with GDPR requirements and EU guidelines on Trustworthy AI.

The BayLDA AI Checklist by the Bavarian Data Protection Authority provides structured guidelines for ensuring GDPR compliance throughout the AI systems lifecycle. The checklist is structured around four main areas:

  1. Classification of AI systems:
    • Clarifies the AI technologies in use, such as large language models, machine learning approaches, or rule-based systems.
    • Considers hosting environments, preprocessing, post-processing, and filtering mechanisms for input and output data.
  2. Training of AI Models:
    • Specifies and documents AI architectures and training procedures.
    • Evaluates the use of personal, pseudonymised, anonymised, or synthetic data, ensuring clear legal bases (particularly for sensitive data).
    • Requires comprehensive documentation of data sources and quality assurance measures to mitigate bias.
    • Establishes the necessity of conducting Data Protection Impact Assessments (DPIA) under GDPR (Art. 35) and emphasizes maintaining data subject rights, including access, rectification, deletion, and data portability (Arts. 15-21 GDPR).
  3. Risk Assessment of AI Systems:
    • Encourages creating and regularly updating a structured risk model based on key data protection and ethical principles from the EU Guidelines for Trustworthy AI.
    • Core principles include fairness (avoiding discrimination), autonomy and control (human oversight), transparency (explainability and accountability), reliability (accuracy and robustness against manipulation), security (protection against unauthorized access and manipulation), and data privacy compliance.
  4. Operational Deployment of AI Applications:
    • Outlines clear documentation and legal basis requirements for deploying AI applications, particularly in scenarios involving cloud providers or third-party services.
    • Stresses the importance of continuous monitoring and logging of AI systems, comprehensive testing prior to deployment, and systematic handling of data subject requests and rights.
    • Advocates integration of AI system usage within organizational training and awareness programs, ensuring transparency and compliance throughout ongoing operations.

Here’s the checklist card and summary for the provided checklist, titled “AI and Data Protection Risk Toolkit” (ICO - UK GDPR aligned):

10.4 AI and Data Protection Risk Toolkit (ICO)

NoteChecklist card
Name: AI and Data Protection Risk Toolkit (ICO)
Audience: AI Governance, Data Privacy Officers, ML Engineers, Cybersecurity Experts, Cross-functional Teams
Type: Compliance, Auditability, Data Privacy, Monitoring, Ethics
Reference: ICO AI and Data Protection Risk Toolkit
Comment: Checklist structured according to AI lifecycle stages, aligned explicitly with UK GDPR Articles and ICO guidelines.

The AI and Data Protection Risk Toolkit provided by the UK’s Information Commissioner’s Office (ICO) is a comprehensive framework designed to assist organizations in identifying, assessing, and mitigating data protection risks associated with AI systems. The checklist is structured according to the AI lifecycle stages, systematically addressing data protection risks:

  • Business Requirements and Design
    • Emphasizes accountability, requiring Data Protection Impact Assessments (DPIAs) to ensure thorough identification and mitigation of risks to individual rights and freedoms.
  • Risk Areas and Controls
    • Risk statements linked to GDPR provisions, accompanied by control objectives and practical steps.
    • Controls include DPIAs, consulting domain experts, and engaging with affected individuals or their representatives.
  • Assessment and Documentation
    • Requires documentation of inherent and residual risks, controls implemented, their ownership, and current status, including planned completion dates for risk mitigation actions.
  • Guidance and Best Practices
    • Provides direct references to detailed ICO guidance documents for implementing specific GDPR compliance measures, ensuring organizations have authoritative resources for further clarification and detailed actions.

Here’s your checklist card and summary for the provided checklist titled “AI Auditing Checklist” (European Data Protection Board - Support Pool of Experts Programme):

10.5 AI Auditing Checklist (EDPB’s SPE Programme)

NoteChecklist card
Name: EDPB AI Auditing Checklist (SPE Programme)
Audience: AI Governance, Data Privacy Officers, ML Engineers, Cybersecurity Experts, Cross-functional Teams
Type: Compliance, Auditability, Data Privacy, Ethics, Monitoring
Reference: EDPB AI Auditing Checklist (SPE)
Comment: Checklist developed under the EDPB Support Pool of Experts Programme, specifically aligned with GDPR and focused on bias, fairness, transparency, and accountability of AI systems.

The AI Auditing Checklist, developed within the European Data Protection Board’s (EDPB) Support Pool of Experts Programme (SPE), provides a structured methodology to audit algorithmic systems comprehensively. It emphasizes socio-technical auditing, focusing explicitly on the fairness, bias, transparency, accountability, and overall social impacts of AI systems, particularly those based on machine learning algorithms. The checklist presents five auditing activities which end up forming the audit report:

  1. Model Card:
    • Documentation about AI models, including purposes, data characteristics, methodologies, bias metrics, and redress mechanisms.
    • Accountability through detailed documentation aligned with GDPR articles, facilitating clarity and transparency from the outset.
  2. System Map:
    • Maps the interactions between AI algorithms, technical systems, and decision-making processes.
    • Clarifies responsibilities and identifies transparency requirements, particularly regarding AI component identification, management roles, documentation, traceability, and explainability.
  3. Moments and sources of bias:
    • Defines and categorizes moments (from data collection to decision-making) and sources of bias (historical bias, aggregation bias, selection bias, etc.).
    • Emphasizes thorough bias evaluation, documentation, and active mitigation strategies at every stage (pre-processing, in-processing, post-processing).
  4. Bias testing:
    • Suggests explicit bias testing strategies to ensure fairness across protected groups (race, gender, socio-economic status, etc.) through defined statistical fairness metrics.
  5. Adversarial Audits:
    • Adversarial auditing to uncover hidden biases or unintended behaviors by actively testing systems in real-life scenarios.