SLA misses and notifications in Apache Airflow - Step-by-Step Execution

Process Flow - SLA misses and notifications

DAG runs task

↓

Check if task finishes before SLA time

Yes↓

No action

↓

Send notification email

↓

Log SLA miss for review

The DAG runs a task and checks if it finishes before the SLA deadline. If it misses, an SLA miss event triggers a notification and logging.

Execution Sample

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.utils.dates import days_ago
from datetime import timedelta

default_args = {
    'sla': timedelta(minutes=5),
    'email': ['[e-mail]'],
    'email_on_failure': True
}
with DAG('sla_example', default_args=default_args, start_date=days_ago(1)) as dag:
    task = BashOperator(task_id='task1', bash_command='sleep 600')

This DAG defines a task with a 5-minute SLA and email notification on SLA miss.

Process Table

Step	Task Status	Elapsed Time	SLA Time	SLA Miss?	Action Taken	Notification Sent
1	Task started	0s	5m	No	None	No
2	Task running	3m	5m	No	None	No
3	Task running	5m	5m	No	None	No
4	Task finished	10m	5m	Yes	SLA miss event triggered	Yes
5	Notification sent	-	-	-	Email sent to [e-mail]	Yes
6	SLA miss logged	-	-	-	Logged in Airflow UI	Yes
7	End	-	-	-	No further action	No

💡 Task finished after 10 minutes, exceeding SLA of 5 minutes, triggering SLA miss and notification.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
task_status	Not started	Started	Running	Running	Finished	Finished
elapsed_time	0s	0s	3m	5m	10m	10m
sla_miss	False	False	False	False	True	True
notification_sent	False	False	False	False	True	True

Key Moments - 2 Insights

Why does the SLA miss only trigger after the task finishes, not during the run?

The SLA miss event triggers only after the task finishes and the elapsed time is compared to the SLA. See execution_table step 4 where the task finishes at 10 minutes, exceeding the 5-minute SLA.

Does Airflow send notifications automatically on SLA miss?

Yes, if email_on_failure or email_on_retry is set and email addresses are configured, Airflow sends notifications automatically on SLA miss events as shown in execution_table step 5.

Visual Quiz - 3 Questions

Look at the execution table, at which step does the SLA miss get detected?

A Step 3

B Step 4

C Step 2

D Step 5

💡 Hint

Check the 'SLA Miss?' column in the execution_table rows.

According to the variable tracker, what is the value of 'notification_sent' after step 3?

A True

B Undefined

C False

D Error

💡 Hint

Look at the 'notification_sent' row and the column 'After Step 3' in variable_tracker.

If the task finished in 4 minutes instead of 10, how would the SLA miss column change in the execution table?

A It would be 'No' at step 4

B It would be 'Yes' at step 4

C It would be 'Yes' at step 3

D It would be 'No' at step 2

💡 Hint

Compare elapsed time with SLA time in execution_table rows.

Concept Snapshot

SLA misses in Airflow occur when a task exceeds its defined SLA time.
Airflow detects SLA misses after task completion.
Notifications (e.g., emails) can be sent automatically on SLA miss.
Configure SLA and email in DAG default_args.
SLA miss events are logged for monitoring.

Full Transcript

This visual execution shows how Airflow handles SLA misses and notifications. The DAG runs a task with a 5-minute SLA. The task runs longer (10 minutes), so after it finishes, Airflow detects the SLA miss and triggers an event. This event sends an email notification and logs the miss in the Airflow UI. Variables like task status, elapsed time, SLA miss flag, and notification sent flag change step-by-step. Key moments clarify why SLA miss triggers after task completion and how notifications are sent. Quiz questions test understanding of when SLA miss is detected and notification status.

Practice

1. What does setting an sla parameter on an Airflow task do?

easy

A. It defines a time limit; if the task runs longer, it triggers an SLA miss.

B. It sets the maximum retries for the task.

C. It schedules the task to run at a specific time.

D. It disables the task from running automatically.

Solution

1. Step 1: Understand the purpose of sla in Airflow tasks

   The sla parameter sets a time limit for task execution.

2. Step 2: Identify what happens when the limit is exceeded

   If the task runs longer than the sla time, Airflow marks it as an SLA miss.

3. Final Answer:

   It defines a time limit; if the task runs longer, it triggers an SLA miss. -> Option A

4. Quick Check:

   sla sets time limit = C [OK]

Hint: Remember SLA means max allowed time for task [OK]

Common Mistakes:

• Confusing SLA with retry count
• Thinking SLA schedules task start time
• Assuming SLA disables tasks

2. Which of the following is the correct way to define an SLA of 30 minutes on a task in Airflow?

easy

A. sla='30 minutes'

B. sla=30

C. sla=timedelta(minutes=30)

D. sla='00:30:00'

Solution

1. Step 1: Recall the correct data type for sla

   The sla parameter expects a datetime.timedelta object.

2. Step 2: Identify the correct syntax for 30 minutes timedelta

   timedelta(minutes=30) correctly creates a 30-minute duration.

3. Final Answer:

   sla=timedelta(minutes=30) -> Option C

4. Quick Check:

   sla uses timedelta object = B [OK]

Hint: Use timedelta for time durations in Airflow [OK]

Common Mistakes:

• Using string instead of timedelta
• Passing integer without timedelta
• Using incorrect string format

3. Given this DAG snippet, what will happen if task1 runs longer than 10 minutes?

from datetime import timedelta
from airflow import DAG
from airflow.operators.dummy import DummyOperator
from airflow.utils.dates import days_ago

def sla_miss_callback(dag, task_list, blocking_task_list, slas, session=None):
    print('SLA missed!')

dag = DAG('sla_example', start_date=days_ago(1))

task1 = DummyOperator(
    task_id='task1',
    dag=dag,
    sla=timedelta(minutes=10)
)

dag.sla_miss_callback = sla_miss_callback

medium

A. The sla_miss_callback function will print 'SLA missed!' when the SLA is missed.

B. Nothing happens because DummyOperator ignores SLA.

C. The task will automatically retry until it finishes within 10 minutes.

D. Airflow will fail the DAG immediately.

Solution

1. Step 1: Understand SLA miss callback usage

   The DAG-level sla_miss_callback is called when any task misses its SLA.

2. Step 2: Identify what the callback does

   The callback prints 'SLA missed!' when triggered by a task exceeding its SLA.

3. Final Answer:

   The sla_miss_callback function will print 'SLA missed!' when the SLA is missed. -> Option A

4. Quick Check:

   SLA miss triggers callback print = A [OK]

Hint: SLA miss triggers callback, not retries or failure [OK]

Common Mistakes:

• Thinking SLA causes retries automatically
• Assuming DummyOperator ignores SLA
• Believing DAG fails immediately on SLA miss

4. You wrote this code to notify on SLA misses but no notification happens:

def notify_sla_miss(dag, task_list, blocking_task_list, slas, session=None):
    send_email('[e-mail]', 'SLA Missed')

dag = DAG('sla_notify', start_date=days_ago(1), sla_miss_callback=notify_sla_miss)

task = DummyOperator(task_id='task', dag=dag, sla=timedelta(minutes=5))

What is the likely reason notifications are not sent?

medium

A. DummyOperator does not support SLA misses.

B. SLA miss callback must be set on the task, not the DAG.

C. The SLA time must be set in seconds, not minutes.

D. The send_email function is not defined or imported.

Solution

1. Step 1: Check the callback function implementation

   The callback calls send_email but no import or definition is shown.

2. Step 2: Understand Airflow's email sending

   Airflow requires proper email function or operator; undefined send_email causes failure silently.

3. Final Answer:

   The send_email function is not defined or imported. -> Option D

4. Quick Check:

   Undefined email function blocks notification = D [OK]

Hint: Ensure email functions are imported or use Airflow email operators [OK]

Common Mistakes:

• Setting callback on DAG instead of task (both valid but DAG preferred)
• Wrong SLA time units
• Assuming DummyOperator ignores SLA

5. You want to send a custom Slack message when any task misses its SLA in your DAG. Which approach correctly implements this?

hard

A. Set sla on tasks and add a SlackOperator as a downstream task to notify SLA misses.

B. Define a sla_miss_callback function that uses Slack API and assign it to the DAG's sla_miss_callback parameter.

C. Use on_failure_callback on tasks to send Slack messages when SLA is missed.

D. Configure Airflow to send Slack notifications by setting sla to the Slack channel name.

Solution

1. Step 1: Understand how SLA miss notifications work

   SLA misses trigger the sla_miss_callback function assigned to the DAG or task.

2. Step 2: Identify correct way to send Slack message on SLA miss

   Implement a callback function that calls Slack API and assign it to DAG's sla_miss_callback.

3. Final Answer:

   Define a sla_miss_callback function that uses Slack API and assign it to the DAG's sla_miss_callback parameter. -> Option B

4. Quick Check:

   SLA miss callback with Slack API = A [OK]

Hint: Use sla_miss_callback with Slack API for notifications [OK]

Common Mistakes:

• Using on_failure_callback instead of sla_miss_callback
• Trying to notify via downstream tasks
• Misconfiguring sla parameter as Slack channel