11 Advanced cases with AI systems and data protection
In this last chapter we cover a few more advanced cases. The content of this chapter is useful to explore more special scenarios: the teacher could assign these as essays topics or small literature research tasks.
11.1 Collection of cases by the EDPS, November 2024
The EDPS released an excellent report (European Data Protection Supervisor 2024) on six emerging trends that combine novel AI applications with data protection issues: retrieval-augmented generation, on-device AI, machine unlearning, multimodal AI, scalable oversight, neurosymbolic AI.
11.1.1 Retrieval-Augmented Generation (RAG)
Retrieval-Augmented Generation (RAG) combines the strengths of generative AI with one or more external knowledge bases. For example, a system might generate a detailed medical report using a large language model by combining its internal capabilities with external verified medical databases. This improves the accuracy and relevance of outputs, limiting the risks of hallucinated content in the RAG output. However, the integration of external data introduces risks related to data protection. If the external source of knowledge contains sensitive information (e.g. personal data), it could be retrieved inadvertently e.g. via prompt injection attacks techniques increasing the risk of data leakage in the AI responses. Understanding and safeguarding how external data is integrated with the AI system is important to mitigate such risks. A good reference on the topic is Zeng et al. (2024).
Definition:
The Model Context Protocol (MCP) is a relatively new open protocol designed to enable AI systems (typically generative AI systems based on large language models) to dynamically interact with external tools, resources, APIs, and data in a modular and interoperable way. It allows AI agents to autonomously discover, invoke, and operate tools, without manual API integration. For example, an AI agent integrated with MCP can autonomously fetch live stock prices using a financial API, summarize the results, and notify the user via email. Another example is a code assistant like Cursor, which can use MCP to interact with the version control system of the project, test frameworks, and deployment tools directly from the development environment, allowing developers to execute complex workflows with simple natural language commands.
As the reader can already imagine, with so much power and so much automation, security or data protection risks are just behind the corner. For example, if someone sets up a fake MCP server (installer spoofing) or tool with a misleading name (name collision), an AI agent might accidentally send personal data or private records to the wrong place. This can happen without the user knowing, especially if the tools look trustworthy. Because MCP allows AI to automatically connect with many different services, a poorly secured tool can greatly increase the risks of a data breach. For more considerations on security and privacy aspects of MCP, see Hou et al. (2025).
11.1.2 On-device AI
On-device AI performs processing directly on a user’s device rather than relying on remote servers. This setup is beneficial because it reduces response times and enhances privacy by keeping data local. For example, voice assistants like Apple’s Siri could process certain commands without sending data to the cloud. On-device AI however can still carry risks to data protection, especially when it is never fully clear if other operations around the AI system are sending data elsewhere. Sensitive data stored locally might be accessed without the user’s awareness, or systems may create profiles of user behavior derived from the data about what apps are used, when, and how. These profiles could then be shared with third parties, potentially violating data minimization (only collecting what is strictly necessary) and purpose limitation (using data only for specific, agreed purposes) principles. Proper safeguards are necessary to ensure that locally processed data is not exploited or misused.
11.1.3 Machine unlearning
Machine unlearning is a method where AI systems “forget” specific data upon request. This aligns with rights granted under data protection laws like the GDPR’s right to erasure, which allows individuals to request the removal of their data. For instance, a machine learning model trained on medical data could unlearn contributions from a specific patient when they request it. However, unlearning comes with challenges. If data from certain groups is removed disproportionately, it could introduce biases into the AI system, leading to unfair treatment, inaccurate predictions, and eventually catastrophic forgetting. Moreover, membership inference attacks techniques con be exploited on the model with forgotten data, to reveal which data points were erased, compromising privacy. Machine unlearning is a very active field of research, for an introductory review on the topic see Zhang et al. (2023).
11.1.4 Multimodal AI
Multimodal AI systems combines different types of data, like text, images, and sound, to make more informed decisions. A self-driving car, for instance, uses visual data from cameras, audio signals, and spatial data from sensors to navigate safely. While this integration is powerful, it can also raise privacy concerns. Biometric data, like facial images or voice recordings, are often part of multimodal inputs, and mishandling this sensitive information could lead to serious data protection violations. For example, if a system misuses facial recognition data, it might unfairly discriminate against certain individuals or violate laws governing sensitive personal data. Additionally, multimodal systems often require large datasets, increasing the chances of data breaches or misuse during collection and processing.
11.1.5 Scalable Oversight
Scalable oversight refers to using tools and feedback loops to ensure AI systems remain aligned with human values and safety standards, even as they grow in complexity. For instance, feedback mechanisms can teach AI systems to avoid generating harmful content or making unethical recommendations. While scalable oversight improves AI reliability, it can create data retention issues (retaining identifiable information longer than necessary). Poorly designed oversight systems might also fail to detect nuanced risks, such as subtle biases in model outputs or errors affecting minority groups.
11.1.6 Neuro-Symbolic AI
Neuro-symbolic AI combines the pattern recognition abilities of neural networks with the structured reasoning of symbolic systems. This approach can make AI systems more interpretable and accurate. For instance, a neuro-symbolic system in healthcare could use neural networks to identify potential diseases from medical images and symbolic reasoning to explain its diagnosis logically. However, the reliance on structured reasoning introduces risks when these systems are trained on sensitive datasets, such as medical or financial records. If reasoning rules are improperly derived or poorly managed, they might make unfair decisions or inadvertently expose sensitive information. Automated decision-making based on flawed logic could violate data protection laws, particularly if individuals are not informed or cannot challenge the decisions. Transparency in how reasoning is applied and safeguards to protect sensitive data are crucial to address these risks.
11.2 Named Entity Recognition systems
A report commissioned by the Support Pool of Experts of the EDPB covered the possible privacy risks associated to named entity recognition (NER, Barberá (2023a)).
NER is a method used in Natural Language Processing (NLP) to identify specific entities, such as names, organizations, or locations, within a text and classify them into predefined categories. It is widely used across sectors such as healthcare, legal analysis, and customer support. NER systems can employ lexicon-based, rule-based, or machine learning-based methods, with modern systems often relying on machine learning due to their adaptability.
The report identifies significant privacy risks associated with NER technology. Key concerns include the processing of sensitive data (e.g., medical records or criminal data), which could lead to serious harm if mishandled. Risks also arise from large-scale data processing, where breaches could have amplified effects. Additionally, issues like low-quality input data or insufficient security measures could lead to incorrect decisions, adversely affecting individuals’ rights. These risks further increase when NER solutions involve cloud-based third-party services, as they may store data for extended periods, share it without proper consent, or process it in jurisdictions lacking adequate data protection laws.
11.3 Optical Character Recognition
Another report commissioned by the Support Pool of Experts of the EDPB covered the possible privacy risks associated to Optical Character Recognition (OCR, Barberá (2023b)).
OCR is a technology that extract text from images, scanned documents, or physical records into machine-readable formats. Modern OCR systems can use machine learning (ML) and deep learning (DL) models to handle structured, semi-structured, and unstructured documents. These systems often employ stages such as detection, localization, and segmentation to identify and extract text accurately. OCR technology can also present risks to data protection and privacy. Such technology might be used with sensitive data (e.g health data) or in connection with large-amount ot data, further increasing the changes of unauthorised access, or data breaches. OCR can also produce wrong outputs, like misrecognition of text, which may adversely impact individuals if used in automated decision-making processes.
11.4 Summary
We have covered a few advanced cases, and some very new directions of AI systems, towards more agentic systems that can automate possibly everything. We are basically done with this book, in the next and final chapter we will do a final summary and where to go from here.