Task failure callbacks in Apache Airflow - Step-by-Step Execution

Process Flow - Task failure callbacks

Task starts running

↓

Task succeeds?

No→Trigger failure callback

↓

Callback runs (e.g., send alert)

↓

Task ends normally

The task runs and checks if it succeeds. If it fails, the failure callback triggers to handle the failure, like sending an alert.

Execution Sample

Apache Airflow

def on_failure_callback(context):
    print(f"Task {context['task_instance'].task_id} failed")

def my_task():
    raise Exception('Fail')

This code defines a failure callback that prints a message when the task fails, and a task function that raises an error to trigger the callback.

Process Table

Step	Action	Task Status	Callback Triggered	Output
1	Start task execution	Running	No	No output yet
2	Task raises Exception	Failed	Yes	Trigger failure callback
3	Run failure callback	Failed	Yes	Print: Task my_task failed
4	Task ends	Failed	No	Task failure handled

💡 Task failed at step 2, failure callback triggered to handle failure

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
task_status	Not started	Running	Failed	Failed	Failed
callback_triggered	False	False	True	True	True

Key Moments - 2 Insights

Why does the failure callback run only after the task fails?

Because the callback is set to trigger only when the task status changes to 'Failed' as shown in execution_table step 2 and 3.

Does the failure callback change the task status?

No, the task status remains 'Failed' after the callback runs, as seen in variable_tracker and execution_table steps 3 and 4.

Visual Quiz - 3 Questions

Look at the execution table, at which step is the failure callback triggered?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Check the 'Callback Triggered' column in the execution_table rows

According to variable_tracker, what is the value of 'task_status' after step 2?

A Failed

B Running

C Not started

D Succeeded

💡 Hint

Look at the 'task_status' row under 'After Step 2' in variable_tracker

If the task did not raise an exception, what would happen to the failure callback?

A It would still run after task ends

B It would run before the task starts

C It would never run

D It would run twice

💡 Hint

Refer to concept_flow where callback triggers only on failure

Concept Snapshot

Task failure callbacks run only when a task fails.
Define a function to handle failure (e.g., alert).
Assign it to the task's on_failure_callback.
When task fails, callback runs automatically.
Task status remains 'Failed' after callback.
Useful for notifications or cleanup on failure.

Full Transcript

In Airflow, a task runs and may succeed or fail. If it fails, a special function called a failure callback runs automatically. This callback can do things like send alerts or log errors. The task status changes to 'Failed' when an error happens. The failure callback does not change the task status but helps handle the failure. This flow ensures you know when tasks fail and can react accordingly.