Training data pipeline automation in MLOps - Step-by-Step Execution

Process Flow - Training data pipeline automation

Start Pipeline

↓

Extract Data

↓

Transform Data

↓

Load Data

↓

Trigger Model Training

↓

Monitor & Log

↓

End Pipeline

The pipeline starts by extracting data, then transforms it, loads it for training, triggers model training, and finally monitors the process.

Execution Sample

MLOps

def run_pipeline():
    data = extract_data()
    clean_data = transform_data(data)
    load_data(clean_data)
    trigger_training()
    log_status('Pipeline completed')

This code runs a simple training data pipeline automating extraction, transformation, loading, training trigger, and logging.

Process Table

Step	Action	Input	Output	Status
1	extract_data()	None	raw_data.csv	Success
2	transform_data(raw_data.csv)	raw_data.csv	clean_data.csv	Success
3	load_data(clean_data.csv)	clean_data.csv	Data loaded to storage	Success
4	trigger_training()	Data loaded	Training job started	Success
5	log_status('Pipeline completed')	Training job started	Logged completion	Success
6	End	N/A	Pipeline finished	Pipeline completed successfully

💡 Pipeline ends after logging completion successfully.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
data	None	raw_data.csv	raw_data.csv	raw_data.csv	raw_data.csv	raw_data.csv
clean_data	None	None	clean_data.csv	clean_data.csv	clean_data.csv	clean_data.csv
status	None	Success	Success	Success	Success	Pipeline completed successfully

Key Moments - 3 Insights

Why do we transform data after extracting it?

Because raw data often has errors or unwanted parts; transforming cleans and prepares it for training, as shown in step 2 of the execution_table.

What happens if loading data fails?

The pipeline would stop or retry; in our table, step 3 shows 'Success', meaning data was loaded correctly, allowing training to start.

Why do we log status at the end?

Logging confirms the pipeline finished and helps track issues later, as seen in step 5 where 'Pipeline completed' is logged.

Visual Quiz - 3 Questions

Look at the execution_table, what is the output of step 2?

A clean_data.csv

B raw_data.csv

C Training job started

D Data loaded to storage

💡 Hint

Check the 'Output' column in row for step 2 in execution_table.

At which step does the training job start?

A Step 3

B Step 4

C Step 5

D Step 2

💡 Hint

Look at the 'Action' and 'Output' columns in execution_table to find when training starts.

If transform_data fails, what would happen to the variable 'clean_data'?

A It would remain None

B It would have raw_data.csv

C It would be set to 'Training job started'

D It would be 'Pipeline completed successfully'

💡 Hint

Refer to variable_tracker and consider what happens if step 2 fails.

Concept Snapshot

Training data pipeline automation:
- Extract raw data
- Transform data to clean it
- Load data for training
- Trigger model training
- Log pipeline status
Automates data prep and training start for ML models.

Full Transcript

This visual execution shows a training data pipeline automation. It starts by extracting raw data, then transforms it to clean and prepare it. Next, it loads the clean data into storage. After loading, it triggers the model training job. Finally, it logs the completion status. Variables like 'data' and 'clean_data' change as the pipeline progresses. Key moments include why transformation is needed, the importance of loading success, and logging at the end. The quiz tests understanding of outputs and pipeline flow.