Training data pipeline automation in MLOps - Step-by-Step Execution

Process Flow - Training data pipeline automation

Start Pipeline

↓

Extract Data

↓

Transform Data

↓

Load Data

↓

Trigger Model Training

↓

Monitor & Log

↓

End Pipeline

The pipeline starts by extracting data, then transforms it, loads it for training, triggers model training, and finally monitors the process.

Execution Sample

MLOps

def run_pipeline():
    data = extract_data()
    clean_data = transform_data(data)
    load_data(clean_data)
    trigger_training()
    log_status('Pipeline completed')

This code runs a simple training data pipeline automating extraction, transformation, loading, training trigger, and logging.

Process Table

Step	Action	Input	Output	Status
1	extract_data()	None	raw_data.csv	Success
2	transform_data(raw_data.csv)	raw_data.csv	clean_data.csv	Success
3	load_data(clean_data.csv)	clean_data.csv	Data loaded to storage	Success
4	trigger_training()	Data loaded	Training job started	Success
5	log_status('Pipeline completed')	Training job started	Logged completion	Success
6	End	N/A	Pipeline finished	Pipeline completed successfully

💡 Pipeline ends after logging completion successfully.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
data	None	raw_data.csv	raw_data.csv	raw_data.csv	raw_data.csv	raw_data.csv
clean_data	None	None	clean_data.csv	clean_data.csv	clean_data.csv	clean_data.csv
status	None	Success	Success	Success	Success	Pipeline completed successfully

Key Moments - 3 Insights

Why do we transform data after extracting it?

Because raw data often has errors or unwanted parts; transforming cleans and prepares it for training, as shown in step 2 of the execution_table.

What happens if loading data fails?

The pipeline would stop or retry; in our table, step 3 shows 'Success', meaning data was loaded correctly, allowing training to start.

Why do we log status at the end?

Logging confirms the pipeline finished and helps track issues later, as seen in step 5 where 'Pipeline completed' is logged.

Visual Quiz - 3 Questions

Look at the execution_table, what is the output of step 2?

A clean_data.csv

B raw_data.csv

C Training job started

D Data loaded to storage

💡 Hint

Check the 'Output' column in row for step 2 in execution_table.

At which step does the training job start?

A Step 3

B Step 4

C Step 5

D Step 2

💡 Hint

Look at the 'Action' and 'Output' columns in execution_table to find when training starts.

If transform_data fails, what would happen to the variable 'clean_data'?

A It would remain None

B It would have raw_data.csv

C It would be set to 'Training job started'

D It would be 'Pipeline completed successfully'

💡 Hint

Refer to variable_tracker and consider what happens if step 2 fails.

Concept Snapshot

Training data pipeline automation:
- Extract raw data
- Transform data to clean it
- Load data for training
- Trigger model training
- Log pipeline status
Automates data prep and training start for ML models.

Full Transcript

This visual execution shows a training data pipeline automation. It starts by extracting raw data, then transforms it to clean and prepare it. Next, it loads the clean data into storage. After loading, it triggers the model training job. Finally, it logs the completion status. Variables like 'data' and 'clean_data' change as the pipeline progresses. Key moments include why transformation is needed, the importance of loading success, and logging at the end. The quiz tests understanding of outputs and pipeline flow.

Practice

1. What is the main benefit of automating a training data pipeline in machine learning?

easy

A. It saves time and reduces human errors during data preparation.

B. It makes the model training faster by using GPUs.

C. It increases the size of the training dataset automatically.

D. It guarantees 100% accuracy of the machine learning model.

Solution

1. Step 1: Understand the purpose of automation in data pipelines

   Automation helps by handling repetitive tasks consistently without manual intervention.

2. Step 2: Identify the key benefits of automation

   Automation saves time and reduces errors that happen when humans prepare data manually.

3. Final Answer:

   It saves time and reduces human errors during data preparation. -> Option A

4. Quick Check:

   Automation = saves time and reduces errors [OK]

Hint: Automation mainly saves time and avoids mistakes [OK]

Common Mistakes:

• Thinking automation speeds up model training directly
• Assuming automation increases dataset size automatically
• Believing automation guarantees perfect model accuracy

2. Which of the following is the correct Python syntax to define a simple function that automates a data cleaning step?

easy

A. clean_data(data) => data.dropna()

B. def clean_data(data):\n return data.dropna()

C. def clean_data(data):\nreturn data.dropna()

D. function clean_data(data) { return data.dropna() }

Solution

1. Step 1: Identify correct Python function syntax

   Python functions start with 'def', followed by name and parameters, then indented body.

2. Step 2: Check indentation and syntax correctness

   def clean_data(data):\n return data.dropna() uses correct indentation and syntax; others use wrong language syntax or missing indentation.

3. Final Answer:

   def clean_data(data):\n return data.dropna() -> Option B

4. Quick Check:

   Python function syntax = def + indent + return [OK]

Hint: Python functions need 'def' and proper indentation [OK]

Common Mistakes:

• Using JavaScript syntax in Python
• Missing indentation after function definition
• Using arrow functions which are not Python syntax

3. Consider this Python code snippet automating a data pipeline step:

def normalize(data):
    mean = data.mean()
    std = data.std()
    return (data - mean) / std

import pandas as pd
sample = pd.Series([10, 20, 30])
result = normalize(sample)
print(result.round(2))

What is the printed output?

medium

A. [ -1.0, 0.0, 1.0 ]

B. [ -1.22, 0.00, 1.22 ]

C. [ 10, 20, 30 ]

D. [ 0.0, 0.0, 0.0 ]

Solution

1. Step 1: Calculate mean and standard deviation of the sample

   Mean = (10+20+30)/3 = 20; Std deviation = 10 (pandas std() uses ddof=1 by default).

2. Step 2: Normalize each value and round to 2 decimals

   (10-20)/10 = -1.0, (20-20)/10=0.0, (30-20)/10 = 1.0

3. Final Answer:

   [ -1.0, 0.0, 1.0 ] -> Option A

4. Quick Check:

   Normalization = (value-mean)/std [OK]

Hint: Normalize by subtracting mean and dividing by std [OK]

Common Mistakes:

• Confusing standard deviation with variance
• Not rounding output
• Returning original data instead of normalized

4. You have this code snippet for automating data loading:

def load_data(file_path):
    data = pd.read_csv(file_path)
    return data

# Usage
dataset = load_data('data.csv')
print(dataset.head())

But it throws an error: NameError: name 'pd' is not defined. How do you fix it?

medium

A. Remove the function and read CSV directly.

B. Change 'pd.read_csv' to 'csv.read'.

C. Add 'import pandas as pd' at the top of the script.

D. Rename 'file_path' to 'filepath' in the function.

Solution

1. Step 1: Understand the error message

   NameError means 'pd' is not recognized because pandas was not imported.

2. Step 2: Fix by importing pandas with alias 'pd'

   Add 'import pandas as pd' at the top so 'pd.read_csv' works correctly.

3. Final Answer:

   Add 'import pandas as pd' at the top of the script. -> Option C

4. Quick Check:

   Import pandas as pd to use pd.read_csv [OK]

Hint: Always import pandas as pd before using pd functions [OK]

Common Mistakes:

• Changing function parameter names without reason
• Assuming csv module replaces pandas read_csv
• Removing function instead of fixing import

5. You want to automate a training data pipeline that:
1. Loads CSV data,
2. Cleans missing values,
3. Normalizes numeric columns,
4. Saves the processed data.

Which tool or approach best supports scheduling and monitoring this pipeline automatically?

hard

A. Using Excel macros to clean and normalize data.

B. Writing a single Python script and running it manually each time.

C. Training the model directly without data preprocessing.

D. Using Apache Airflow to create and schedule pipeline tasks.

Solution

1. Step 1: Identify requirements for automation and monitoring

   We need a tool that schedules tasks and tracks their success or failure.

2. Step 2: Evaluate options for pipeline automation

   Apache Airflow is designed for scheduling, monitoring, and managing workflows automatically.

3. Final Answer:

   Using Apache Airflow to create and schedule pipeline tasks. -> Option D

4. Quick Check:

   Airflow = scheduling + monitoring pipelines [OK]

Hint: Use Airflow for automated scheduling and monitoring [OK]

Common Mistakes:

• Running scripts manually instead of automating
• Using Excel which lacks automation for pipelines
• Skipping data preprocessing before training