Pipeline scheduling and triggers in MLOps - Mini Project: Build & Apply

Pipeline scheduling and triggers

📖 Scenario: You work as a machine learning engineer. You want to automate your ML pipeline to run regularly and also trigger it when new data arrives.This helps save time and keeps your models updated without manual work.

🎯 Goal: Build a simple Python script that simulates scheduling an ML pipeline to run every day at a fixed time and also triggers immediately when new data is detected.You will create a schedule, a trigger condition, and then run the pipeline accordingly.

📋 What You'll Learn

Create a variable called pipeline_name with the value 'daily_ml_pipeline'

Create a variable called schedule_time with the value '02:00' representing 2 AM daily

Create a variable called new_data_arrived set to True or False to simulate data arrival

Write an if statement that triggers the pipeline if new_data_arrived is True

Write an else statement that prints the scheduled run time of the pipeline

💡 Why This Matters

🌍 Real World

Automating ML pipelines saves time and ensures models stay updated by running at set times or when new data arrives.

💼 Career

Understanding pipeline scheduling and triggers is essential for ML engineers and DevOps professionals working with automated workflows.

1

Set up pipeline name

Create a variable called pipeline_name and set it to the string 'daily_ml_pipeline'.

MLOps

# Create the pipeline_name variable
# Your code here

Need a hint?

Use the assignment operator = to set the variable.

2

Add schedule time variable

Create a variable called schedule_time and set it to the string '02:00' representing 2 AM daily.

MLOps

pipeline_name = 'daily_ml_pipeline'
# Create the schedule_time variable
# Your code here

Need a hint?

Use a string to represent the time in 24-hour format.

3

Simulate new data arrival

Create a variable called new_data_arrived and set it to True to simulate that new data has arrived.

MLOps

pipeline_name = 'daily_ml_pipeline'
schedule_time = '02:00'
# Create the new_data_arrived variable
# Your code here

Need a hint?

Use the boolean value True to indicate data arrival.

4

Trigger pipeline based on data arrival or schedule

Write an if statement that checks if new_data_arrived is True. If yes, print "Triggering {pipeline_name} due to new data." using an f-string. Otherwise, print "Scheduled run of {pipeline_name} at {schedule_time}." using an f-string.

MLOps

pipeline_name = 'daily_ml_pipeline'
schedule_time = '02:00'
new_data_arrived = True
# Add if-else to trigger pipeline or show schedule
# Your code here

Need a hint?

Use if new_data_arrived: and f-strings for printing.