Pipeline scheduling and triggers in MLOps - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a pipeline trigger in MLOps?

A pipeline trigger is an event or condition that starts the execution of a machine learning pipeline automatically, such as a code commit, a schedule, or a data update.

beginner

Explain the difference between manual and scheduled pipeline triggers.

Manual triggers require a user to start the pipeline by clicking a button or running a command. Scheduled triggers start the pipeline automatically at set times or intervals without user intervention.

intermediate

What is a cron expression in pipeline scheduling?

A cron expression is a string that defines the schedule for running a pipeline using time and date fields, like minutes, hours, day of month, month, and day of week.

intermediate

How can data changes trigger a pipeline run?

Data changes can trigger a pipeline when the system monitors data sources and starts the pipeline automatically if new data arrives or existing data is updated.

beginner

Why is pipeline scheduling important in MLOps?

Scheduling ensures pipelines run regularly to keep models updated with fresh data, improving accuracy and automating repetitive tasks without manual effort.

Which of the following is NOT a common pipeline trigger?

A Manual start by user

B Scheduled time using cron

C Random time intervals

D Data update event

Random time intervals are not a standard trigger method; triggers are usually manual, scheduled, or event-based.

What does a cron expression define in pipeline scheduling?

A The time and frequency to run the pipeline

B The user permissions for the pipeline

C The data source for the pipeline

D The pipeline's output format

A cron expression specifies when and how often the pipeline should run.

How can pipelines be triggered by data changes?

A By monitoring data sources for updates

B By manually checking data files

C By running pipelines on fixed schedules only

D By restarting the server

Monitoring data sources allows automatic triggering when new or updated data is detected.

Which trigger type requires user interaction to start the pipeline?

A Scheduled trigger

B Manual trigger

C Data trigger

D Event trigger

Manual triggers need a user to start the pipeline explicitly.

Why automate pipeline scheduling in MLOps?

A To increase manual checks

B To make pipelines run slower

C To avoid using data

D To reduce manual work and keep models updated

Automation saves time and ensures models use fresh data regularly.

Describe how pipeline triggers work and list three common types.

Think about what starts a pipeline automatically or by user action.

Explain the role of scheduling in MLOps pipelines and how cron expressions help.

Consider why pipelines need to run regularly without manual effort.

Practice

1. What is the main purpose of pipeline scheduling in MLOps?

easy

A. To store pipeline logs for debugging

B. To manually start pipelines whenever needed

C. To run tasks automatically at specific times without manual intervention

D. To create new machine learning models from scratch

Solution

1. Step 1: Understand pipeline scheduling

   Pipeline scheduling is designed to run tasks automatically at set times, like daily or hourly, without needing a person to start them.

2. Step 2: Compare options

   Only To run tasks automatically at specific times without manual intervention describes automatic running at specific times. Other options describe manual actions or unrelated tasks.

3. Final Answer:

   To run tasks automatically at specific times without manual intervention -> Option C

4. Quick Check:

   Pipeline scheduling = automatic timed runs [OK]

Hint: Scheduling means automatic runs at set times [OK]

Common Mistakes:

• Confusing scheduling with manual triggering
• Thinking scheduling stores logs
• Assuming scheduling creates models directly

2. Which of the following is a correct cron expression to schedule a pipeline to run every day at 3 AM?

easy

A. 3 0 * * *

B. 0 3 * * *

C. * 3 * * *

D. 0 0 3 * * *

Solution

1. Step 1: Understand cron format

   Cron syntax is: minute hour day month weekday. To run at 3 AM daily, minute=0, hour=3, day/month/weekday=any (*).

2. Step 2: Match expression

   0 3 * * * "0 3 * * *" means minute 0, hour 3, every day. Others have wrong order or extra fields.

3. Final Answer:

   0 3 * * * -> Option B

4. Quick Check:

   Minute=0, Hour=3 daily = 0 3 * * * [OK]

Hint: Cron: minute hour day month weekday; 3 AM is '0 3 * * *' [OK]

Common Mistakes:

• Swapping hour and minute fields
• Adding extra fields in cron
• Using '*' in wrong positions

3. Given this pipeline trigger configuration snippet:

{
  "trigger": {
    "event": "data_arrival",
    "filter": {
      "file_type": "csv"
    }
  }
}

What happens when a new JSON file arrives in the data folder?

medium

A. The pipeline does not run because the file type is not CSV

B. The pipeline runs because any new file triggers it

C. The pipeline runs only if the JSON file is large

D. The pipeline runs but ignores the file type

Solution

1. Step 1: Analyze trigger filter

   The trigger listens for 'data_arrival' events but only runs if the file type is 'csv'.

2. Step 2: Apply to JSON file

   A JSON file does not match the 'csv' filter, so the pipeline will not run.

3. Final Answer:

   The pipeline does not run because the file type is not CSV -> Option A

4. Quick Check:

   Filter file_type=csv blocks JSON files [OK]

Hint: Triggers with filters run only on matching events [OK]

Common Mistakes:

• Ignoring filter conditions
• Assuming any file triggers pipeline
• Confusing event type with file type

4. You wrote this cron expression to schedule a pipeline every hour:

60 * * * *

Why does the pipeline never run?

medium

A. Because the hour field is missing

B. Because cron requires seconds field

C. Because the asterisks are misplaced

D. Because 60 is not a valid minute value in cron syntax

Solution

1. Step 1: Check minute field validity

   Cron minute values must be 0-59. '60' is invalid and causes no runs.

2. Step 2: Confirm other fields

   The hour and other fields are correct as '*', meaning every hour/day. The error is only the minute value.

3. Final Answer:

   Because 60 is not a valid minute value in cron syntax -> Option D

4. Quick Check:

   Minute must be 0-59; 60 is invalid [OK]

Hint: Minutes in cron go 0-59, never 60 [OK]

Common Mistakes:

• Using 60 as minute value
• Thinking cron needs seconds field
• Misplacing asterisks

5. You want a pipeline to run automatically when new data arrives and also every Sunday at midnight. Which setup correctly combines scheduling and event triggers?

hard

A. Use a cron schedule '0 0 * * 0' and an event trigger for 'data_arrival' together

B. Use only a cron schedule '0 0 * * 0' because event triggers conflict with schedules

C. Use only an event trigger for 'data_arrival' and manually run on Sundays

D. Use a cron schedule '0 0 * * 7' and ignore event triggers

Solution

1. Step 1: Understand combined triggers

   Pipelines can have both cron schedules and event triggers to run on different conditions.

2. Step 2: Verify cron expression for Sunday midnight

   '0 0 * * 0' runs at midnight on Sundays (0 or 7 can represent Sunday, but 0 is standard).

3. Step 3: Confirm event trigger for data arrival

   Adding an event trigger for 'data_arrival' ensures pipeline runs when new data arrives.

4. Final Answer:

   Use a cron schedule '0 0 * * 0' and an event trigger for 'data_arrival' together -> Option A

5. Quick Check:

   Combine cron and event triggers for full automation [OK]

Hint: Combine cron and event triggers for multiple run conditions [OK]

Common Mistakes:

• Thinking schedules and triggers cannot coexist
• Using wrong cron day for Sunday
• Ignoring event triggers for data arrival