Task failure callbacks in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the purpose of a task failure callback in Airflow?

In Airflow, what is the main purpose of setting a task failure callback function?

A To automatically restart the entire DAG run from the beginning.

B To retry the failed task immediately without any delay.

C To execute custom code automatically when a task fails, such as sending alerts or cleaning resources.

D To skip the failed task and continue the workflow without any notification.

💡 Hint

Think about what you want to happen right after a task fails.

A task failure callback lets you run custom code when a task fails, like sending an email alert or logging extra info. It does not retry or skip tasks automatically.

💻 Command Output

intermediate

Output of a task failure callback function logging failure info

Given this Airflow task failure callback function, what will be printed when a task fails?

Apache Airflow

def failure_callback(context):
    task_instance = context['task_instance']
    print(f"Task {task_instance.task_id} failed at {task_instance.end_date}")

A Task <task_id> succeeded at <timestamp>

B Task <task_id> failed at <timestamp>

C Error: 'end_date' attribute not found

D No output is printed

💡 Hint

Look at what the print statement outputs using the context dictionary.

The callback prints the task ID and the time it ended, which is when it failed.

❓ Configuration

advanced

Correct way to assign a failure callback to an Airflow task

Which code snippet correctly assigns a failure callback function named notify_failure to an Airflow PythonOperator task?

A task = PythonOperator(task_id='task1', python_callable=my_func, on_failure_callback=notify_failure)

B task = PythonOperator(task_id='task1', python_callable=my_func, failure_callback=notify_failure)

C task = PythonOperator(task_id='task1', python_callable=my_func, on_fail_callback=notify_failure)

D task = PythonOperator(task_id='task1', python_callable=my_func, on_failure=notify_failure)

💡 Hint

Check the exact parameter name for failure callbacks in Airflow operators.

The correct parameter to assign a failure callback is on_failure_callback. Other options are invalid parameter names.

❓ Troubleshoot

advanced

Why is the failure callback not triggered on task failure?

You assigned a failure callback to a task, but it never runs when the task fails. What is the most likely reason?

A Failure callbacks only run if the DAG run succeeds.

B The task is marked as <code>retries=0</code>, so failure callbacks are skipped.

C The failure callback function must be decorated with <code>@task_failure</code> to work.

D The failure callback function is not passed as <code>on_failure_callback</code> parameter to the task.

💡 Hint

Check how the failure callback is assigned to the task.

If the failure callback is not assigned using the on_failure_callback parameter, Airflow will not know to run it on failure.

🔀 Workflow

expert

Order of execution when a task fails with a failure callback

Arrange the steps in the correct order when an Airflow task fails and a failure callback is set.

A 1,2,3,4

B 3,1,2,4

C 2,1,3,4

D 1,3,2,4

💡 Hint

Think about what happens first: failure, marking state, then callback.

First the task fails, then Airflow marks it failed, then it calls the failure callback, which performs actions.

Practice

1. What is the main purpose of on_failure_callback in an Airflow task?

easy

A. To run a function automatically when the task fails

B. To schedule the task to run at a specific time

C. To retry the task multiple times on failure

D. To log the task's successful completion

Solution

1. Step 1: Understand the role of on_failure_callback

   This parameter is designed to specify a function that runs automatically when a task fails.

2. Step 2: Differentiate from other task features

   Scheduling and retries are handled by other parameters, not on_failure_callback.

3. Final Answer:

   To run a function automatically when the task fails -> Option A

4. Quick Check:

   Failure callback = automatic function on failure [OK]

Hint: Failure callback runs code only when task fails [OK]

Common Mistakes:

• Confusing failure callback with scheduling
• Thinking it retries the task automatically
• Assuming it runs on task success

2. Which of the following is the correct way to assign a failure callback function named notify_failure to an Airflow task?

easy

A. task = PythonOperator(failure_callback=notify_failure, ...)

B. task = PythonOperator(on_failure_callback=notify_failure, ...)

C. task = PythonOperator(on_fail_callback=notify_failure, ...)

D. task = PythonOperator(callback_on_failure=notify_failure, ...)

Solution

1. Step 1: Recall the exact parameter name

   The correct parameter to assign a failure callback is on_failure_callback.

2. Step 2: Match the parameter with the options

   Only task = PythonOperator(on_failure_callback=notify_failure, ...) uses on_failure_callback correctly.

3. Final Answer:

   task = PythonOperator(on_failure_callback=notify_failure, ...) -> Option B

4. Quick Check:

   Correct parameter name = on_failure_callback [OK]

Hint: Use exact parameter name: on_failure_callback [OK]

Common Mistakes:

• Using incorrect parameter names like failure_callback
• Swapping words in the parameter name
• Misspelling the parameter

3. Given this code snippet, what will be printed if the task fails?

def alert(context):
    print(f"Task {context['task_instance'].task_id} failed")

task = PythonOperator(
    task_id='my_task',
    python_callable=lambda: 1/0,
    on_failure_callback=alert
)

medium

A. Task succeeded

B. ZeroDivisionError

C. No output, task succeeds

D. Task my_task failed

Solution

1. Step 1: Analyze the task's python_callable

   The lambda divides 1 by 0, causing a failure (ZeroDivisionError).

2. Step 2: Understand the failure callback behavior

   On failure, the alert function runs and prints the failure message with the task ID.

3. Final Answer:

   Task my_task failed -> Option D

4. Quick Check:

   Failure triggers alert print [OK]

Hint: Failure triggers callback print message [OK]

Common Mistakes:

• Expecting ZeroDivisionError to print directly
• Thinking task succeeds despite error
• Ignoring the callback print output

4. You wrote this failure callback function:

def fail_alert():
    print("Task failed")

task = PythonOperator(
    task_id='fail_task',
    python_callable=lambda: 1/0,
    on_failure_callback=fail_alert
)

Why does the callback not run when the task fails?

medium

A. The callback function must be decorated with @task

B. The callback function must return a value

C. The callback function must accept a context parameter

D. The callback function must be async

Solution

1. Step 1: Check callback function signature

   Failure callbacks receive a context dictionary argument; missing it causes errors.

2. Step 2: Understand Airflow callback requirements

   The callback must accept one parameter (context) to work properly.

3. Final Answer:

   The callback function must accept a context parameter -> Option C

4. Quick Check:

   Callback needs context argument [OK]

Hint: Callback must accept context argument [OK]

Common Mistakes:

• Defining callback without parameters
• Expecting return value from callback
• Making callback async unnecessarily

5. You want to send an email alert only when a task fails and the failure reason contains the word "timeout". How should you implement the on_failure_callback function?

hard

A. Define a function that checks context['exception'] for 'timeout' and sends email if true

B. Define a function that always sends email regardless of failure reason

C. Use on_failure_callback to retry the task on timeout

D. Set on_failure_callback to None and handle alerts elsewhere

Solution

1. Step 1: Understand the requirement

   Send email only if failure reason contains "timeout".

2. Step 2: Use context to check exception message

   The failure callback receives context with exception info; check if 'timeout' is in the exception string.

3. Step 3: Send email conditionally

   Send email alert only if the condition is met inside the callback function.

4. Final Answer:

   Define a function that checks context['exception'] for 'timeout' and sends email if true -> Option A

5. Quick Check:

   Check exception text in callback before alert [OK]

Hint: Check exception text in callback before alert [OK]

Common Mistakes:

• Sending email on all failures without filtering
• Trying to retry task in failure callback
• Ignoring exception details in context