Pipeline components and DAGs in MLOps - Step-by-Step Execution

Process Flow - Pipeline components and DAGs

Define pipeline components

↓

Arrange components as tasks

↓

Create DAG with dependencies

↓

Schedule and run pipeline

↓

Monitor task execution and results

↓

Complete pipeline run

This flow shows how pipeline components become tasks arranged in a DAG, which is then scheduled and executed step-by-step.

Execution Sample

MLOps

task1 = load_data()
task2 = preprocess(task1)
task3 = train_model(task2)
task4 = evaluate(task3)
pipeline = DAG([task1, task2, task3, task4])
pipeline.run()

This code defines four tasks as pipeline components, arranges them in a DAG with dependencies, and runs the pipeline.

Process Table

Step	Task	Status Before	Action	Status After	Output
1	load_data	Not started	Start task	Completed	Raw data loaded
2	preprocess	Waiting for load_data	Start after load_data	Completed	Data cleaned
3	train_model	Waiting for preprocess	Start after preprocess	Completed	Model trained
4	evaluate	Waiting for train_model	Start after train_model	Completed	Model evaluated
5	pipeline	Running	All tasks completed	Completed	Pipeline run successful

💡 All tasks completed in order respecting dependencies, pipeline run ends successfully

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
task1_status	Not started	Completed	Completed	Completed	Completed	Completed
task2_status	Not started	Not started	Completed	Completed	Completed	Completed
task3_status	Not started	Not started	Not started	Completed	Completed	Completed
task4_status	Not started	Not started	Not started	Not started	Completed	Completed
pipeline_status	Running	Running	Running	Running	Running	Completed

Key Moments - 3 Insights

Why does 'preprocess' wait before starting?

'preprocess' depends on 'load_data' completing first, as shown in execution_table row 2 where status before is 'Waiting for load_data'. This ensures data is ready before cleaning.

Can tasks run in any order?

No, tasks run in order defined by dependencies in the DAG. For example, 'train_model' waits for 'preprocess' to finish (row 3). This prevents errors from missing inputs.

What happens if a task fails?

If a task fails, downstream tasks depending on it won't start. This is not shown here but is a key safety feature of DAGs to avoid running tasks with missing data.

Visual Quiz - 3 Questions

Look at the execution_table, what is the status of 'train_model' before step 3?

A Not started

B Waiting for preprocess

C Completed

D Running

💡 Hint

Check the 'Status Before' column for 'train_model' at step 3 in execution_table

At which step does the pipeline status change to 'Completed'?

A Step 4

B Step 3

C Step 5

D Step 2

💡 Hint

Look at the 'pipeline' row in execution_table and find when status changes to 'Completed'

If 'load_data' fails, what happens to 'preprocess'?

A It waits indefinitely

B It runs anyway

C It starts immediately

D It skips and pipeline completes

💡 Hint

Refer to key_moments explanation about task dependencies and waiting

Concept Snapshot

Pipeline components are tasks arranged in a Directed Acyclic Graph (DAG).
Each task depends on outputs of previous tasks.
The DAG ensures tasks run in order respecting dependencies.
Pipeline runs by executing tasks step-by-step.
Failures stop downstream tasks to keep data safe.

Full Transcript

This visual execution shows how pipeline components become tasks arranged in a DAG. The code example defines four tasks: load_data, preprocess, train_model, and evaluate. Each task waits for its dependencies to complete before starting. The execution table traces each step, showing task statuses before and after running. Variables track task states changing from 'Not started' to 'Completed'. Key moments clarify why tasks wait for others and how the DAG controls execution order. The quiz tests understanding of task statuses and pipeline completion. The snapshot summarizes the core idea: pipelines use DAGs to run tasks in order safely.