Pipeline components and DAGs in MLOps - Mini Project: Build & Apply

Building a Simple MLOps Pipeline with Components and DAGs

📖 Scenario: You are working as a data engineer in a team that builds machine learning pipelines. Your task is to create a simple pipeline that has components for data loading, data preprocessing, and model training. These components will be connected in a Directed Acyclic Graph (DAG) to define the order of execution.This project will help you understand how pipeline components and DAGs work in MLOps.

🎯 Goal: Build a simple MLOps pipeline using Python dictionaries to represent components and a list to represent the DAG order. You will create components for data loading, preprocessing, and training, then connect them in a DAG, and finally print the execution order.

📋 What You'll Learn

Create a dictionary called components with keys 'load_data', 'preprocess_data', and 'train_model' each having a string description as value.

Create a list called dag that defines the execution order of the components as 'load_data', 'preprocess_data', 'train_model'.

Use a for loop to iterate over the dag list and print the component name and its description from the components dictionary.

💡 Why This Matters

🌍 Real World

In real MLOps, pipelines are built with components representing tasks like data loading, preprocessing, and training. These tasks are connected in a DAG to control the order of execution.

💼 Career

Understanding pipeline components and DAGs is essential for roles like MLOps engineer, data engineer, and machine learning engineer to automate and manage ML workflows efficiently.

1

Create pipeline components dictionary

Create a dictionary called components with these exact entries: 'load_data': 'Load raw data from source', 'preprocess_data': 'Clean and transform data', and 'train_model': 'Train ML model on processed data'.

MLOps

# Create the components dictionary with exact keys and descriptions
# Your code here

Hint

Use curly braces {} to create a dictionary. Each key is a string like 'load_data' and each value is a string description.

2

Define the DAG execution order

Create a list called dag with the exact order of component names: 'load_data', 'preprocess_data', 'train_model'.

MLOps

components = {
    'load_data': 'Load raw data from source',
    'preprocess_data': 'Clean and transform data',
    'train_model': 'Train ML model on processed data'
}

# Create the dag list with the exact component names in order
# Your code here

Hint

Use square brackets [] to create a list. Put the component names as strings in the correct order.

3

Iterate over DAG and print component info

Use a for loop with variable component to iterate over the dag list. Inside the loop, get the description from components[component] and print the component name and description in this format: "Component: {component}, Description: {description}".

MLOps

components = {
    'load_data': 'Load raw data from source',
    'preprocess_data': 'Clean and transform data',
    'train_model': 'Train ML model on processed data'
}

dag = ['load_data', 'preprocess_data', 'train_model']

# Use a for loop to print each component and its description
# Your code here

Hint

Use a for loop to go through each item in dag. Use f-strings to format the print output.

4

Print the pipeline execution order

Write a print statement to display the text exactly: "Pipeline execution order completed."

MLOps

components = {
    'load_data': 'Load raw data from source',
    'preprocess_data': 'Clean and transform data',
    'train_model': 'Train ML model on processed data'
}

dag = ['load_data', 'preprocess_data', 'train_model']

for component in dag:
    description = components[component]
    print(f"Component: {component}, Description: {description}")

# Print the completion message below
# Your code here

Hint

Use print("Pipeline execution order completed.") exactly to show the final message.

Practice

1. What does a Directed Acyclic Graph (DAG) represent in an MLOps pipeline?

easy

A. Tasks and their dependencies without any cycles

B. A loop of tasks that repeat indefinitely

C. Random tasks executed in parallel without order

D. Only the final output of a pipeline

Solution

1. Step 1: Understand DAG structure

   A DAG is a graph with nodes and edges where edges show dependencies and no cycles exist.

2. Step 2: Relate DAG to pipeline tasks

   In MLOps, tasks are nodes and dependencies are edges, ensuring tasks run in order without loops.

3. Final Answer:

   Tasks and their dependencies without any cycles -> Option A

4. Quick Check:

   DAG = tasks + dependencies without loops [OK]

Hint: DAG means no loops, just tasks linked in order [OK]

Common Mistakes:

• Thinking DAG allows loops
• Confusing DAG with random task order
• Assuming DAG only shows final output

2. Which of the following is the correct syntax to define a simple DAG in Apache Airflow?

easy

A. dag = DAG('my_dag', interval='daily')

B. dag = DAG('my_dag' schedule='daily')

C. dag = DAG('my_dag', schedule='everyday')

D. dag = DAG('my_dag', schedule_interval='@daily')

Solution

1. Step 1: Check Airflow DAG syntax

   The DAG constructor requires a name and a schedule_interval parameter for timing.

2. Step 2: Validate options

   dag = DAG('my_dag', schedule_interval='@daily') uses correct parameter 'schedule_interval' with valid value '@daily'. Others use wrong parameter names or values.

3. Final Answer:

   dag = DAG('my_dag', schedule_interval='@daily') -> Option D

4. Quick Check:

   Correct DAG syntax uses schedule_interval [OK]

Hint: Use schedule_interval='@daily' for daily DAGs [OK]

Common Mistakes:

• Using 'schedule' instead of 'schedule_interval'
• Wrong interval value formats
• Missing commas between parameters

3. Given this Airflow DAG snippet, what is the order of task execution?

task1 = DummyOperator(task_id='task1', dag=dag)
task2 = DummyOperator(task_id='task2', dag=dag)
task3 = DummyOperator(task_id='task3', dag=dag)
task1 >> task2 >> task3

medium

A. task3, then task2, then task1

B. task1, then task2, then task3

C. task2, then task1, then task3

D. All tasks run in parallel

Solution

1. Step 1: Analyze task dependencies

   The '>>' operator sets order: task1 before task2, task2 before task3.

2. Step 2: Determine execution sequence

   Tasks run in sequence: task1 first, then task2, then task3.

3. Final Answer:

   task1, then task2, then task3 -> Option B

4. Quick Check:

   task1 >> task2 >> task3 means sequential order [OK]

Hint: >> means run left task before right task [OK]

Common Mistakes:

• Assuming tasks run in reverse order
• Thinking tasks run in parallel
• Ignoring the '>>' operator meaning

4. You wrote this DAG code but get an error: TypeError: 'DAG' object is not iterable. What is the likely cause?

with DAG('example_dag', schedule_interval='@daily') as dag:
    task1 = DummyOperator(task_id='task1')
    task2 = DummyOperator(task_id='task2')
    task1 >> task2

for task in dag:
    print(task.task_id)

medium

A. DAG object is not iterable, so 'for task in dag' causes error

B. DummyOperator requires a 'dag' parameter outside the context

C. Missing import for DummyOperator

D. schedule_interval '@daily' is invalid

Solution

1. Step 1: Identify error cause

   The error says 'DAG' object is not iterable, likely from trying to loop over dag object.

2. Step 2: Understand DAG iterability

   DAG objects in Airflow are not iterable directly; looping over them causes this error.

3. Final Answer:

   DAG object is not iterable, so 'for task in dag' causes error -> Option A

4. Quick Check:

   DAG is not iterable; use dag.tasks list instead [OK]

Hint: DAG is not iterable; use dag.tasks to loop [OK]

Common Mistakes:

• Trying to loop directly over DAG object
• Assuming DummyOperator needs dag param outside context
• Misreading error as import issue

5. You want to create a pipeline where task A runs first, then tasks B and C run in parallel, and finally task D runs after both B and C finish. Which DAG structure correctly represents this?

hard

A. [A, B] >> C >> D

B. A >> B >> C >> D

C. A >> [B, C] >> D

D. A >> D >> [B, C]

Solution

1. Step 1: Understand task order requirements

   Task A runs first, then B and C run at the same time, then D runs after both finish.

2. Step 2: Translate to DAG syntax

   Using Airflow syntax, 'A >> [B, C] >> D' means A before B and C in parallel, then D after both.

3. Final Answer:

   A >> [B, C] >> D -> Option C

4. Quick Check:

   Parallel tasks in list brackets between sequential tasks [OK]

Hint: Use brackets [] for parallel tasks in DAG [OK]

Common Mistakes:

• Placing tasks in wrong order
• Not using brackets for parallel tasks
• Assuming linear order for all tasks