Pipeline components and DAGs in MLOps - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding DAG structure in ML pipelines

In an ML pipeline, a Directed Acyclic Graph (DAG) defines the order of tasks. Which statement best describes a DAG?

A A graph where tasks are connected with no cycles, ensuring a clear execution order.

B A graph where tasks can loop back to previous tasks forming cycles.

C A graph that allows tasks to run in any order without dependencies.

D A graph that only contains a single task with no connections.

💡 Hint

Think about what 'acyclic' means in the graph context.

A DAG is a graph with directed edges and no cycles, so tasks run in a defined order without loops.

💻 Command Output

intermediate

Output of a pipeline task execution command

Given the following command to run an ML pipeline task:

mlflow run . -P entry_point=train_model

What is the expected output if the task runs successfully?

A Run completed successfully with metrics logged and model saved.

B ERROR: Entry point 'train_model' not found in the project.

C SyntaxError: invalid syntax in command line.

D Task skipped due to unmet dependencies.

💡 Hint

Consider what happens when a valid entry point is executed in MLflow.

If the entry point exists and runs without error, MLflow logs metrics and saves the model, indicating success.

❓ Configuration

advanced

Correct DAG YAML configuration snippet

Which YAML snippet correctly defines a DAG with three tasks where preprocess runs before train, and train runs before evaluate?

A

tasks:
  preprocess:
    depends_on: [train, evaluate]
  train:
    depends_on: []
  evaluate:
    depends_on: [preprocess]

B

tasks:
  preprocess:
    depends_on: [train]
  train:
    depends_on: [evaluate]
  evaluate:
    depends_on: []

C

tasks:
  preprocess:
    depends_on: [evaluate]
  train:
    depends_on: []
  evaluate:
    depends_on: [train]

D

tasks:
  preprocess:
    depends_on: []
  train:
    depends_on: [preprocess]
  evaluate:
    depends_on: [train]

💡 Hint

Check the order of dependencies to ensure no cycles and correct sequence.

Option D correctly sets dependencies so preprocess runs first, then train, then evaluate.

❓ Troubleshoot

advanced

Identifying cause of pipeline task failure

You run a pipeline and the evaluate task fails with the error: FileNotFoundError: model.pkl not found. Which is the most likely cause?

A The <code>evaluate</code> task has a syntax error in its script.

B The <code>train</code> task did not run or failed to save the model file.

C The pipeline DAG has a cycle causing infinite loops.

D The <code>preprocess</code> task deleted the model file by mistake.

💡 Hint

Think about which task creates the model file and what happens if it is missing.

The evaluate task depends on the model file created by train. If train fails or is skipped, the file won't exist.

🔀 Workflow

expert

Determining pipeline execution order from DAG

Given the following DAG dependencies:

tasks:
  data_ingest:
    depends_on: []
  feature_engineering:
    depends_on: [data_ingest]
  model_training:
    depends_on: [feature_engineering]
  model_validation:
    depends_on: [model_training]
  deployment:
    depends_on: [model_validation, feature_engineering]

What is the correct order of task execution?

A data_ingest → feature_engineering → model_training → deployment → model_validation

B data_ingest → feature_engineering → deployment → model_training → model_validation

C data_ingest → feature_engineering → model_training → model_validation → deployment

D data_ingest → model_training → feature_engineering → model_validation → deployment

💡 Hint

Remember that a task can only run after all its dependencies have completed.

Deployment depends on both model_validation and feature_engineering, so it runs last after those complete.

Practice

1. What does a Directed Acyclic Graph (DAG) represent in an MLOps pipeline?

easy

A. Tasks and their dependencies without any cycles

B. A loop of tasks that repeat indefinitely

C. Random tasks executed in parallel without order

D. Only the final output of a pipeline

Solution

1. Step 1: Understand DAG structure

   A DAG is a graph with nodes and edges where edges show dependencies and no cycles exist.

2. Step 2: Relate DAG to pipeline tasks

   In MLOps, tasks are nodes and dependencies are edges, ensuring tasks run in order without loops.

3. Final Answer:

   Tasks and their dependencies without any cycles -> Option A

4. Quick Check:

   DAG = tasks + dependencies without loops [OK]

Hint: DAG means no loops, just tasks linked in order [OK]

Common Mistakes:

• Thinking DAG allows loops
• Confusing DAG with random task order
• Assuming DAG only shows final output

2. Which of the following is the correct syntax to define a simple DAG in Apache Airflow?

easy

A. dag = DAG('my_dag', interval='daily')

B. dag = DAG('my_dag' schedule='daily')

C. dag = DAG('my_dag', schedule='everyday')

D. dag = DAG('my_dag', schedule_interval='@daily')

Solution

1. Step 1: Check Airflow DAG syntax

   The DAG constructor requires a name and a schedule_interval parameter for timing.

2. Step 2: Validate options

   dag = DAG('my_dag', schedule_interval='@daily') uses correct parameter 'schedule_interval' with valid value '@daily'. Others use wrong parameter names or values.

3. Final Answer:

   dag = DAG('my_dag', schedule_interval='@daily') -> Option D

4. Quick Check:

   Correct DAG syntax uses schedule_interval [OK]

Hint: Use schedule_interval='@daily' for daily DAGs [OK]

Common Mistakes:

• Using 'schedule' instead of 'schedule_interval'
• Wrong interval value formats
• Missing commas between parameters

3. Given this Airflow DAG snippet, what is the order of task execution?

task1 = DummyOperator(task_id='task1', dag=dag)
task2 = DummyOperator(task_id='task2', dag=dag)
task3 = DummyOperator(task_id='task3', dag=dag)
task1 >> task2 >> task3

medium

A. task3, then task2, then task1

B. task1, then task2, then task3

C. task2, then task1, then task3

D. All tasks run in parallel

Solution

1. Step 1: Analyze task dependencies

   The '>>' operator sets order: task1 before task2, task2 before task3.

2. Step 2: Determine execution sequence

   Tasks run in sequence: task1 first, then task2, then task3.

3. Final Answer:

   task1, then task2, then task3 -> Option B

4. Quick Check:

   task1 >> task2 >> task3 means sequential order [OK]

Hint: >> means run left task before right task [OK]

Common Mistakes:

• Assuming tasks run in reverse order
• Thinking tasks run in parallel
• Ignoring the '>>' operator meaning

4. You wrote this DAG code but get an error: TypeError: 'DAG' object is not iterable. What is the likely cause?

with DAG('example_dag', schedule_interval='@daily') as dag:
    task1 = DummyOperator(task_id='task1')
    task2 = DummyOperator(task_id='task2')
    task1 >> task2

for task in dag:
    print(task.task_id)

medium

A. DAG object is not iterable, so 'for task in dag' causes error

B. DummyOperator requires a 'dag' parameter outside the context

C. Missing import for DummyOperator

D. schedule_interval '@daily' is invalid

Solution

1. Step 1: Identify error cause

   The error says 'DAG' object is not iterable, likely from trying to loop over dag object.

2. Step 2: Understand DAG iterability

   DAG objects in Airflow are not iterable directly; looping over them causes this error.

3. Final Answer:

   DAG object is not iterable, so 'for task in dag' causes error -> Option A

4. Quick Check:

   DAG is not iterable; use dag.tasks list instead [OK]

Hint: DAG is not iterable; use dag.tasks to loop [OK]

Common Mistakes:

• Trying to loop directly over DAG object
• Assuming DummyOperator needs dag param outside context
• Misreading error as import issue

5. You want to create a pipeline where task A runs first, then tasks B and C run in parallel, and finally task D runs after both B and C finish. Which DAG structure correctly represents this?

hard

A. [A, B] >> C >> D

B. A >> B >> C >> D

C. A >> [B, C] >> D

D. A >> D >> [B, C]

Solution

1. Step 1: Understand task order requirements

   Task A runs first, then B and C run at the same time, then D runs after both finish.

2. Step 2: Translate to DAG syntax

   Using Airflow syntax, 'A >> [B, C] >> D' means A before B and C in parallel, then D after both.

3. Final Answer:

   A >> [B, C] >> D -> Option C

4. Quick Check:

   Parallel tasks in list brackets between sequential tasks [OK]

Hint: Use brackets [] for parallel tasks in DAG [OK]

Common Mistakes:

• Placing tasks in wrong order
• Not using brackets for parallel tasks
• Assuming linear order for all tasks