3 The AI development lifecycle and cybersecurity
After completing chapter you will:
- Learn about the different stages of the AI Systems lifecycle.
- Understand the basics of Machine Learning Operations, the data and computational flows in AI systems
- Familiarize with relevant ISO/IEC standards and the types of cybersecurity threats with AI systems
In this chapter we start with the more practical approach in learning how to develop and implement AI systems that are trained with or that are processing personal data. When developing or deploying AI systems there are a series of steps to take into account. Such steps have been formalised into the AI system lifecycle. It is important to learn the AI lifecycle as defined in international standards, as they ensure a level of quality from the inception of the idea to the final AI product. The different stages of the AI lifecycle go hand-in-hand with requirements from data governance. In this chapter we will cover some existing standards relate
3.1 The AI lifecycle
The AI system lifecycle defined by ISO 5338 consists of stages that guide the development, deployment, and maintenance of AI systems, ensuring they are built, used, and monitored responsibly. Each stage focuses on ensuring ethical, transparent, and robust AI practices. The lifecycle includes stages from problem definition and data collection through model development, deployment, monitoring, and retirement. Emphasis is placed on compliance with legal and ethical standards, particularly regarding personal data handling, bias mitigation, and ongoing evaluation to prevent unintended consequences.
| AI Lifecycle Stage | Description |
|---|---|
| Inception | Define the objectives, scope, and requirements for the AI system, including stakeholder needs, ethical considerations, and regulatory requirements. |
| Design & Development | Architect and build the AI model, selecting algorithms, designing workflows, and preparing data to meet intended functionality and performance goals. |
| Verification & Validation | Test and evaluate the AI model to ensure it meets predefined standards and aligns with intended objectives, addressing performance, bias, and reliability. |
| Deployment | Implement the validated model in a production environment, ensuring integration with other systems and adhering to operational and security requirements. |
| Operation & Monitoring | Actively oversee the AI system in its operational environment, monitoring for issues like model drift, bias, or performance changes that may impact outcomes. |
| Continuous Validation | Regularly assess and validate the AI model’s performance and behavior, ensuring it continues to meet the necessary standards and objectives over time. |
| Re-evaluation | Periodically review the AI system’s relevance, assessing if updates or modifications are needed due to changes in data patterns, requirements, or external factors. |
| Retirement | Decommission the AI system or components when they are no longer effective, documenting processes and ensuring secure handling of related data and system resources. |
3.2 Mapping the AI lifecycle to MLops
The AI system lifecycle, as defined by ISO 5338, provides a robust, structured approach to ensure AI development meets high standards of quality, ethics, and compliance. However, the shift to an MLOps approach, specifically as described in the end-to-end MLOps architecture by Kreuzberger, Kühl, and Hirschl (2023), is critical for maintaining operational efficiency, continuous improvement, and adaptability in real-world deployment.
MLOps, or Machine Learning Operations, improves and extends the ISO lifecycle by automating and orchestrating each stage of the AI system lifecycle within a unified framework. This approach can deployment cycles faster and more reproducibile, with reliable monitoring for reacting to any type of security threats. MLOps merges the principles of DevOps with machine learning-specific requirements, creating a well-defined path from inception to model retirement that is reliable. MLops supports cross-functional collaboration and mitigates risks associated with manual interventions, making it a preferred approach in AI system deployment and management.
The figure above visually shows a simplified MLOps pipeline matched to the ISO 5338 standard. The MLOps diagram also adds important elements related to security and data protection highlighted in blue. The initial data processing and model deployment (Experimentation Zone) happens in closed infrastructures (highlighted in green and purple) while model fine-tuning, serving, and live monitoring is in production infrastructures (highlighted in yellow) and potentially more prone to external attacks.
The Secure MLOps (secMLOps) paradigm extends traditional MLOps by embedding security principles directly into each stage of the AI lifecycle. This approach, as described by Zhang and Jaskolka (2022), integrates security considerations like confidentiality, integrity, and availability throughout the machine learning pipeline, aligning closely with the principle of privacy by design. In secMLOps, each role—from data engineer to MLOps engineer—operates with explicit security and privacy responsibilities, ensuring that risks are proactively managed from data ingestion to model deployment.
This continuous security focus, following the People, Processes, Technology, Governance, and Compliance (PPTGC) framework, highlights that security is not a single step but an embedded process across the lifecycle. By ensuring comprehensive monitoring, secure CI/CD pipelines, and ongoing threat modeling, secMLOps provides a structured, secure operational environment. With this paradigm we ensure the robustness of AI systems, making secMLOps a practical and scalable approach for secure and trustworthy AI deployment in diverse, production-oriented environments.
3.3 The components of secure AI development and deployment
In this section we expand the MLOps development/deployment/monitoring pipeline with details related to privacy by design, data protection in practice, and privacy enhancing technologies. Chapter 4-9 will go deeper into the details of the various stages of the MLOps pipeline, but here it is important that it is clear what each stage does, especially considering the fact that we are developing an AI model/system using personal data.
To get an overview of Figure 3.3 we need to understand the three stages:
Data engineering stage: this is all about preparing the data before training, re-using data from previous training models, and of course ensuring the quality of the data. This section will be covered in detail in Chapter 4 and 5. Briefly, everything starts with the acquisition of the raw data. The raw data is then pre-processed to apply various privacy enhancing techniques (“data preprocessing” stage in the figure) and the pre-processed data is obtained. The data is not yet ready to be used for training an AI: for example salient features from the data could be extracted (e.g. instead of processing raw images, one might want to process image features like brightness, sharpness, etc), or then the data could be “augmented” (for example a medical image is accompanied by a mask-image to identifies which pixels in the image are those of interest). The output of this stage provides the “features and augmented data” which can then be used for training the AI model. At this stage the data is split in three parts: training, test, and validation sets. These three sets are then passed to the next stage of “data experimentation”. It is important to also notice that other data (e.g. data from AI system users, or more specialised data for fine tuning) can become training/test/validation data, especially when the AI model is trained in various stages like in the case of Large Language Models (LLMs).
Data experimentation stage: this is about development and refining parameters before creating the model ready for production. This section will be covered in Chapters 6-9.Briefly, the training data is used to train the ML model, and the validation data and test data are used to improve (validation) and measure (test) the performance of the model. At this stage various machine learning techniques can be tested to obtain good ML models prototypes as well as software that is used in the experimentation stage.
Production stage: this is when the final model is trained and pushed to registry and then deployed. This section will be covered in Chapters 6-9. Briefly, the software developed in the experimentation stage can now be reused with the training data to train the final version of the model that will be used in production. The AI model is then deployed into an AI system that will accept input data from the user (e.g. in the form of prompts or queries, the input can also come from another system) and produce the output data. Filtering can be added at this stage to ensure the alignment of the AI system to avoid unwanted inputs or prohibited outputs. The deployed AI system is monitored accordingly and the monitoring data can contribute to the fine-tuning of the AI model.
Reflect on the similarity of the two approaches and try to also highlight the differences. This is a discussion exercise that the instructor can lead in the classroom. As homework the learners can search the internet and explore existing MLOps solutions and share their findings in the class.
3.4 Cybersecurity threats and the AI lifecycle
We conclude this introductory module with some definitions of the possible threats that AI systems can face. The reference for this section is the OWASP AI exchange OWASP (2024).
AI systems can bring unique cybersecurity risks due to their large scale data processing and complex nature of the systems involved. Unlike traditional systems, AI systems and AI models are particularly susceptible to threats that exploit both the learning processes and the data these models consume. Adversaries may want to manipulate or misuse AI systems through poisoning data, or by bypassing model defenses to extract sensitive information. Protecting AI systems thus requires a layered approach, addressing each phase of the AI system/MLOps lifecycle—from development and training to deployment and runtime operations.
3.4.1 Types of AI Cybersecurity Threats
Here we define some of the most common security threats that AI systems or AI models might suffer.
There are four types of threats:
- Threats through use: Attacks that occur when users interact with the AI system, often aiming to deceive or mislead the model. Common examples include:
- Evasion attacks: Manipulating input to trick the model into incorrect predictions or classifications.
- Inference attacks: Extracting sensitive data from model outputs, such as through membership inference or model inversion.
- Development-time threats: Threats that arise during the data preparation, model training, or fine-tuning phases, which can compromise model integrity before deployment. Key threats include:
- Data poisoning: Introducing malicious or biased data to skew model outcomes.
- Model poisoning: Directly altering the model’s parameters or training process to create harmful behaviors.
- Runtime application security threats: Attacks targeting the AI model or application after deployment, often intending to manipulate, disrupt, or compromise the model’s performance or data integrity:
- Model reprogramming (poisoning at runtime): Altering a deployed model’s behavior, possibly through adversarial inputs.
- Output integrity and security: Ensuring the model’s output does not unintentionally leak sensitive data or contain vulnerabilities.
- Conventional security threats: Traditional cybersecurity risks that also impact AI systems, particularly through exposed infrastructure or supply chains. AI systems can be deployed on cloud infrastructures and will suffer similar types of attacks than other API (Application Program Interface) applications. Examples include:
- Data breaches: Unauthorized access to model data or user inputs, compromising confidentiality.
- Supply chain attacks: Infiltrating third-party components within the AI system’s supply chain to introduce vulnerabilities.
- Denial of service: Overloading the model with requests to hinder functionality or availability.
Later in this course, we will cover how to test AI models by simulating these attacks. We will also look at different types of monitoring that can be set up to detect and reduce these threats as the AI model is used. These methods will help the learners understand how to keep AI systems secure.
Figure Figure 3.4 is a simplified version of the detailed MLOps schematic from figure Figure 3.3. As an exercise, identify the attack surface – the number of all possible points where a malicious user might attack your MLOps workflow – in all various stages of the MLOps workflow.
3.5 Other relevant standards
While this book focuses on foundational principles of AI, data protection, and cybersecurity, there are several other standards that play a crucial role in the broader field of AI governance and cybersecurity.
For instance, ISO/IEC 42001, the international standard for AI management systems, establishes guidelines for managing risks and ensuring the reliability of AI systems throughout their lifecycle. Another relevant standard, ISO 22989, outlines principles and concepts for AI, providing a framework for AI terminology, trustworthiness, and quality measures. These standards, while important, go beyond the purpose of this book and they will not be covered.
Additionally, the European Union is actively advancing AI and cybersecurity standards through frameworks like the European Cybersecurity Certification Framework and implementing guidelines based on the AI Act. These aim to ensure that high-risk AI systems comply with strict safety, transparency, and accountability requirements.
While in this curriculum we focus on the case of training AI models using personal data, you might not necessarily need to train a model from scratch: you can develop an AI system (and be an AI system provider according to the AI Act) simply by integrating existing AI models into your applications and data workflows.
Here a few possible scenarios:
You fine-tune an existing model: fine-tuning is a ML technique so that starting from the weights of an existing pre-trained model, the learning stage is continued with new data (the fine-tuning dataset). Techniques such as LoRa () make it possible to modify the model weights so that the AI model’s responses are better reflecting the patterns learned in the fine-tuned dataset. In this scenario, you still need to prepare the personal data to create the fine-tuning dataset, but most likely the training stages are simplified.
You process personal data without embedding it into the AI model: in this scenario the personal data that you are processing is not going to be used to actually train the AI model, however it can augment the knowledge of the model with techniques like Retrieval Augmented Generation (RAG). So for example a query for your AI system might first search for a record in a database and then the record along with your query are passed to the AI model for inference.
More hybrid systems might actually live together and process personal data with and without the use of AI models. Furthermore multiple models might actually be deployed in parallel (for example “Mixture of Experts” architectures with LLMs). We will not cover these scenarios, but they tend to be very popular with various machine learning applications (see Huyen 2022 especially Chapter 7).
3.6 Summary
In this chapter we explored the AI lifecycle as defined by ISO 5338 and its alignment with MLOps frameworks. Understanding these stages, from inception to decommissioning, provides a structured approach to mapping all the components of AI system development, ensuring security, data protection, and responsible AI practices. In the next chapters we will break down the lifecycle into its multiple stages and consider which tools and techniques are useful at each stage.
| Question | Options |
|---|---|
| 1. What is the primary purpose of the AI lifecycle as defined by ISO 5338? | 1) To reduce computational costs. 2) To guide the development, deployment, and maintenance of AI systems responsibly. 3) To ensure faster model training. 4) To simplify the AI development process. |
| 2. Which stage of the AI lifecycle focuses on defining objectives, stakeholder needs, and regulatory requirements? | 1) Design & Development 2) Verification & Validation 3) Inception 4) Operation & Monitoring |
| 3. What is the main focus during the deployment stage of the AI lifecycle? | 1) Developing algorithms. 2) Testing model accuracy. 3) Integrating the model into a production environment. 4) Retiring the model. |
| 4. How does MLOps improve the AI lifecycle? | 1) By reducing the need for secure pipelines. 2) By automating and orchestrating the lifecycle stages within a unified framework. 3) By replacing the need for data governance. 4) By focusing solely on model training. |
| 5. What is the goal of the secMLOps paradigm? | 1) To focus on model performance exclusively. 2) To embed security principles into each stage of the AI lifecycle. 3) To minimize the need for monitoring. 4) To replace MLOps with a simpler alternative. |
| 6. During which stage of the MLOps pipeline is data augmented or features extracted? | 1) Data experimentation 2) Data engineering 3) Production 4) Model deployment |
| 7. What type of threat involves manipulating input to mislead an AI model into incorrect predictions? | 1) Data poisoning 2) Evasion attack 3) Model inversion 4) Supply chain attack |
| 8. Which of the following is an example of a runtime application security threat in AI systems? | 1) Data poisoning 2) Output integrity issues 3) Denial of service 4) Model reprogramming |
| 9. What is the primary focus of ISO/IEC 42001? | 1) To standardize AI terminology. 2) To establish guidelines for managing risks in AI systems throughout their lifecycle. 3) To address cybersecurity in cloud systems. 4) To simplify AI model development. |
| 10. What is the main cybersecurity concern with AI models in production? | 1) Ensuring algorithm efficiency. 2) Protecting against runtime attacks such as model reprogramming. 3) Reducing training time. 4) Simplifying model monitoring. |
Click to reveal solutions
Answer: 2) To guide the development, deployment, and maintenance of AI systems responsibly.
Explanation: The ISO 5338 AI lifecycle ensures ethical, transparent, and robust AI practices.
Answer: 3) Inception.
Explanation: The inception stage focuses on defining the goals, scope, and requirements of the AI system.
Answer: 3) Integrating the model into a production environment.
Explanation: Deployment involves implementing the validated model and ensuring its integration.
Answer: 2) By automating and orchestrating the lifecycle stages within a unified framework.
Explanation: MLOps ensures efficiency and reliability through automation and orchestration.
Answer: 2) To embed security principles into each stage of the AI lifecycle.
Explanation: SecMLOps focuses on integrating security into every phase of the AI lifecycle.
Answer: 2) Data engineering.
Explanation: The data engineering stage involves pre-processing, augmentation, and feature extraction.
Answer: 2) Evasion attack.
Explanation: Evasion attacks manipulate input to mislead the AI model.
Answer: 4) Model reprogramming.
Explanation: Runtime threats include attacks like model reprogramming to alter behavior.
Answer: 2) To establish guidelines for managing risks in AI systems throughout their lifecycle.
Explanation: ISO/IEC 42001 provides comprehensive risk management guidelines for AI.
Answer: 2) Protecting against runtime attacks such as model reprogramming.
Explanation: Runtime attacks pose significant risks to AI models in production environments.