Task failure callbacks in Apache Airflow - Step-by-Step Execution

Process Flow - Task failure callbacks

Task starts running

↓

Task succeeds?

No→Trigger failure callback

↓

Callback runs (e.g., send alert)

↓

Task ends normally

The task runs and checks if it succeeds. If it fails, the failure callback triggers to handle the failure, like sending an alert.

Execution Sample

Apache Airflow

def on_failure_callback(context):
    print(f"Task {context['task_instance'].task_id} failed")

def my_task():
    raise Exception('Fail')

This code defines a failure callback that prints a message when the task fails, and a task function that raises an error to trigger the callback.

Process Table

Step	Action	Task Status	Callback Triggered	Output
1	Start task execution	Running	No	No output yet
2	Task raises Exception	Failed	Yes	Trigger failure callback
3	Run failure callback	Failed	Yes	Print: Task my_task failed
4	Task ends	Failed	No	Task failure handled

💡 Task failed at step 2, failure callback triggered to handle failure

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
task_status	Not started	Running	Failed	Failed	Failed
callback_triggered	False	False	True	True	True

Key Moments - 2 Insights

Why does the failure callback run only after the task fails?

Because the callback is set to trigger only when the task status changes to 'Failed' as shown in execution_table step 2 and 3.

Does the failure callback change the task status?

No, the task status remains 'Failed' after the callback runs, as seen in variable_tracker and execution_table steps 3 and 4.

Visual Quiz - 3 Questions

Look at the execution table, at which step is the failure callback triggered?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Check the 'Callback Triggered' column in the execution_table rows

According to variable_tracker, what is the value of 'task_status' after step 2?

A Failed

B Running

C Not started

D Succeeded

💡 Hint

Look at the 'task_status' row under 'After Step 2' in variable_tracker

If the task did not raise an exception, what would happen to the failure callback?

A It would still run after task ends

B It would run before the task starts

C It would never run

D It would run twice

💡 Hint

Refer to concept_flow where callback triggers only on failure

Concept Snapshot

Task failure callbacks run only when a task fails.
Define a function to handle failure (e.g., alert).
Assign it to the task's on_failure_callback.
When task fails, callback runs automatically.
Task status remains 'Failed' after callback.
Useful for notifications or cleanup on failure.

Full Transcript

In Airflow, a task runs and may succeed or fail. If it fails, a special function called a failure callback runs automatically. This callback can do things like send alerts or log errors. The task status changes to 'Failed' when an error happens. The failure callback does not change the task status but helps handle the failure. This flow ensures you know when tasks fail and can react accordingly.

Practice

1. What is the main purpose of on_failure_callback in an Airflow task?

easy

A. To run a function automatically when the task fails

B. To schedule the task to run at a specific time

C. To retry the task multiple times on failure

D. To log the task's successful completion

Solution

1. Step 1: Understand the role of on_failure_callback

   This parameter is designed to specify a function that runs automatically when a task fails.

2. Step 2: Differentiate from other task features

   Scheduling and retries are handled by other parameters, not on_failure_callback.

3. Final Answer:

   To run a function automatically when the task fails -> Option A

4. Quick Check:

   Failure callback = automatic function on failure [OK]

Hint: Failure callback runs code only when task fails [OK]

Common Mistakes:

• Confusing failure callback with scheduling
• Thinking it retries the task automatically
• Assuming it runs on task success

2. Which of the following is the correct way to assign a failure callback function named notify_failure to an Airflow task?

easy

A. task = PythonOperator(failure_callback=notify_failure, ...)

B. task = PythonOperator(on_failure_callback=notify_failure, ...)

C. task = PythonOperator(on_fail_callback=notify_failure, ...)

D. task = PythonOperator(callback_on_failure=notify_failure, ...)

Solution

1. Step 1: Recall the exact parameter name

   The correct parameter to assign a failure callback is on_failure_callback.

2. Step 2: Match the parameter with the options

   Only task = PythonOperator(on_failure_callback=notify_failure, ...) uses on_failure_callback correctly.

3. Final Answer:

   task = PythonOperator(on_failure_callback=notify_failure, ...) -> Option B

4. Quick Check:

   Correct parameter name = on_failure_callback [OK]

Hint: Use exact parameter name: on_failure_callback [OK]

Common Mistakes:

• Using incorrect parameter names like failure_callback
• Swapping words in the parameter name
• Misspelling the parameter

3. Given this code snippet, what will be printed if the task fails?

def alert(context):
    print(f"Task {context['task_instance'].task_id} failed")

task = PythonOperator(
    task_id='my_task',
    python_callable=lambda: 1/0,
    on_failure_callback=alert
)

medium

A. Task succeeded

B. ZeroDivisionError

C. No output, task succeeds

D. Task my_task failed

Solution

1. Step 1: Analyze the task's python_callable

   The lambda divides 1 by 0, causing a failure (ZeroDivisionError).

2. Step 2: Understand the failure callback behavior

   On failure, the alert function runs and prints the failure message with the task ID.

3. Final Answer:

   Task my_task failed -> Option D

4. Quick Check:

   Failure triggers alert print [OK]

Hint: Failure triggers callback print message [OK]

Common Mistakes:

• Expecting ZeroDivisionError to print directly
• Thinking task succeeds despite error
• Ignoring the callback print output

4. You wrote this failure callback function:

def fail_alert():
    print("Task failed")

task = PythonOperator(
    task_id='fail_task',
    python_callable=lambda: 1/0,
    on_failure_callback=fail_alert
)

Why does the callback not run when the task fails?

medium

A. The callback function must be decorated with @task

B. The callback function must return a value

C. The callback function must accept a context parameter

D. The callback function must be async

Solution

1. Step 1: Check callback function signature

   Failure callbacks receive a context dictionary argument; missing it causes errors.

2. Step 2: Understand Airflow callback requirements

   The callback must accept one parameter (context) to work properly.

3. Final Answer:

   The callback function must accept a context parameter -> Option C

4. Quick Check:

   Callback needs context argument [OK]

Hint: Callback must accept context argument [OK]

Common Mistakes:

• Defining callback without parameters
• Expecting return value from callback
• Making callback async unnecessarily

5. You want to send an email alert only when a task fails and the failure reason contains the word "timeout". How should you implement the on_failure_callback function?

hard

A. Define a function that checks context['exception'] for 'timeout' and sends email if true

B. Define a function that always sends email regardless of failure reason

C. Use on_failure_callback to retry the task on timeout

D. Set on_failure_callback to None and handle alerts elsewhere

Solution

1. Step 1: Understand the requirement

   Send email only if failure reason contains "timeout".

2. Step 2: Use context to check exception message

   The failure callback receives context with exception info; check if 'timeout' is in the exception string.

3. Step 3: Send email conditionally

   Send email alert only if the condition is met inside the callback function.

4. Final Answer:

   Define a function that checks context['exception'] for 'timeout' and sends email if true -> Option A

5. Quick Check:

   Check exception text in callback before alert [OK]

Hint: Check exception text in callback before alert [OK]

Common Mistakes:

• Sending email on all failures without filtering
• Trying to retry task in failure callback
• Ignoring exception details in context