Pipeline scheduling and triggers in MLOps - Step-by-Step Execution

Process Flow - Pipeline scheduling and triggers

Define Pipeline

↓

Set Schedule or Trigger

↓

Wait for Trigger Event

↓

Time-based Schedule?

Yes→Run Pipeline

↓

Complete

↓

Event-based Trigger?

Yes→Run Pipeline

↓

Complete

This flow shows how a pipeline is defined, then scheduled or triggered by time or events, leading to pipeline execution and completion.

Execution Sample

MLOps

pipeline:
  name: example-pipeline
  schedule:
    cron: '0 6 * * *'
  triggers:
    - event: data_arrival

Defines a pipeline named 'example-pipeline' that runs daily at 6 AM or when new data arrives.

Process Table

Step	Event	Condition Checked	Action Taken	Pipeline State
1	Pipeline Defined	N/A	Pipeline configuration saved	Idle
2	Time Tick at 5:59 AM	Is current time 6:00 AM? No	Wait	Idle
3	Time Tick at 6:00 AM	Is current time 6:00 AM? Yes	Start pipeline run	Running
4	Pipeline Running	Pipeline tasks executing	Process data	Running
5	Pipeline Completed	All tasks done	Mark pipeline complete	Completed
6	Event: data_arrival at 7:00 AM	Is event trigger enabled? Yes	Start pipeline run	Running
7	Pipeline Running	Pipeline tasks executing	Process new data	Running
8	Pipeline Completed	All tasks done	Mark pipeline complete	Completed
9	Time Tick at 6:00 AM next day	Is current time 6:00 AM? Yes	Start pipeline run	Running

💡 Pipeline runs triggered by schedule or event, then completes; waits for next trigger.

Status Tracker

Variable	Start	After Step 3	After Step 5	After Step 6	After Step 8	After Step 9
pipeline_state	Idle	Running	Completed	Running	Completed	Running
current_time	N/A	6:00 AM	N/A	7:00 AM	N/A	6:00 AM next day
event_triggered	No	No	No	Yes	No	No

Key Moments - 3 Insights

Why doesn't the pipeline run at 5:59 AM even though time is close to 6:00 AM?

Because the schedule condition checks for exact time '6:00 AM' (see execution_table step 2 and 3). The pipeline only starts when the condition is exactly met.

How can the pipeline run twice in one day?

The pipeline can run once by schedule (6:00 AM) and again by event trigger (data arrival at 7:00 AM), as shown in steps 3 and 6.

What happens if the event trigger is disabled?

If event triggers are disabled, the pipeline will only run on schedule, so steps like 6 and 7 would not start a run.

Visual Quiz - 3 Questions

Look at the execution_table at step 3, what is the pipeline_state?

A Idle

B Running

C Completed

D Waiting

💡 Hint

Check the 'Pipeline State' column at step 3 in the execution_table.

At which step does the pipeline run because of an event trigger?

A Step 3

B Step 9

C Step 6

D Step 2

💡 Hint

Look for the row where 'Event: data_arrival' occurs in the execution_table.

If the schedule cron is changed to '0 7 * * *', when will the pipeline run instead of step 3?

A 7:00 AM

B 6:00 AM

C 5:59 AM

D Immediately

💡 Hint

Refer to the 'current_time' variable in variable_tracker and the schedule condition in execution_table step 3.

Concept Snapshot

Pipeline Scheduling and Triggers:
- Define pipeline with name and triggers
- Schedule uses cron syntax for time-based runs
- Triggers can be event-based (e.g., data arrival)
- Pipeline waits for schedule or event
- On trigger, pipeline runs and completes
- Can have multiple triggers for flexibility

Full Transcript

Pipeline scheduling and triggers let you run your pipeline automatically. First, you define the pipeline and set when it should run. This can be at a specific time using a schedule or when something happens, like new data arriving. The system waits for these triggers. When the time matches the schedule or the event happens, the pipeline starts running. After it finishes, it waits again for the next trigger. This way, your pipeline runs without you needing to start it manually.