Mapped tasks for parallel processing in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

Output of mapped task execution in Airflow

Given the following Airflow DAG snippet using task mapping, what will be the output logs of the mapped task?

Apache Airflow

from airflow import DAG
from airflow.decorators import task
from datetime import datetime

with DAG('mapped_task_example', start_date=datetime(2023, 1, 1), schedule_interval='@daily', catchup=False) as dag:
    @task
    def multiply_by_two(number):
        return number * 2

    numbers = [1, 2, 3]
    results = multiply_by_two.expand(number=numbers)

A Task multiply_by_two fails due to unsupported input type

B Task multiply_by_two runs once with input [1, 2, 3] and outputs [2, 4, 6]

C Task multiply_by_two runs 3 times but outputs are all 0

D Task multiply_by_two runs 3 times with inputs 1, 2, 3 and outputs 2, 4, 6 respectively

💡 Hint

Mapped tasks run once per item in the input list, producing one output per run.

In Airflow, using expand() on a task creates multiple task instances, each running with one item from the input list. Here, multiply_by_two runs 3 times with inputs 1, 2, and 3, producing outputs 2, 4, and 6 respectively.

🧠 Conceptual

intermediate

Understanding task mapping behavior in Airflow

Which statement correctly describes how Airflow handles mapped tasks when the input list is empty?

A The mapped task does not run at all

B The mapped task runs infinitely until stopped

C The mapped task runs once with a None input

D The mapped task raises a runtime error

💡 Hint

Think about what happens if there is nothing to process.

If the input list to a mapped task is empty, Airflow skips running that task entirely because there are no items to map over.

❓ Troubleshoot

advanced

Troubleshooting mapped task failure due to XCom size limit

You have a mapped task that returns a very large list as output, but the task fails with an XCom size limit error. What is the best way to fix this issue?

A Split the output into smaller chunks and map over those chunks instead

B Increase the XCom size limit in Airflow configuration

C Disable XCom for the task

D Return the large list as a string instead of a list

💡 Hint

Think about how to reduce the size of data passed between tasks.

Airflow has a limit on the size of data stored in XComs. Splitting the large output into smaller chunks and mapping over those chunks reduces the size of each XCom, avoiding the limit.

🔀 Workflow

advanced

Designing a workflow with mapped tasks and dependencies

You want to create an Airflow DAG where a mapped task processes a list of files, and after all mapped tasks finish, a final task aggregates the results. Which approach correctly ensures the final task runs only after all mapped tasks complete?

A Set the final task downstream of the mapped task using final_task >> mapped_task

B Set the final task downstream of the mapped task using final_task.set_downstream(mapped_task)

C Set the final task downstream of the mapped task using mapped_task >> final_task

D Set the final task downstream of the mapped task using final_task.set_upstream(mapped_task)

💡 Hint

Remember the direction of dependencies in Airflow DAGs.

In Airflow, the operator >> sets the left task upstream of the right task. So mapped_task >> final_task means final_task runs after all mapped tasks complete.

✅ Best Practice

expert

Best practice for handling dynamic task mapping with large input lists

When using Airflow's mapped tasks with a very large input list (thousands of items), what is the best practice to avoid scheduler overload and improve performance?

A Map over the entire large list at once to maximize parallelism

B Use batch processing by splitting the input list into smaller chunks and map over each chunk sequentially

C Disable task retries to reduce scheduler load

D Increase the number of scheduler workers without changing the DAG

💡 Hint

Think about balancing parallelism and system limits.

Mapping over thousands of items at once can overwhelm the scheduler. Splitting the input into smaller batches and processing them sequentially reduces load and improves stability.