Sharing data between tasks effectively in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

How does XCom share data between tasks?

In Apache Airflow, what is the primary mechanism used to share small pieces of data between tasks?

A Writing data to a shared file system manually.

B Using XCom to push and pull data between tasks.

C Passing data through environment variables automatically.

D Using global Python variables accessible by all tasks.

💡 Hint

Think about the built-in Airflow feature designed for inter-task communication.

XCom (short for cross-communication) is Airflow's built-in feature to share small amounts of data between tasks by pushing and pulling key-value pairs.

💻 Command Output

intermediate

Output of XCom pull with no matching key

What is the output of the following Airflow PythonOperator task code snippet when no XCom value exists for the given key?

Apache Airflow

ti.xcom_pull(key='nonexistent_key', task_ids='task_a')

A '' (empty string)

B Raises KeyError

C None

D 0

💡 Hint

Consider what Airflow returns when no XCom value is found for a key.

If no XCom value exists for the specified key and task, Airflow returns None instead of raising an error.

❓ Configuration

advanced

Correct XCom push syntax in a PythonOperator

Which code snippet correctly pushes a value to XCom inside a PythonOperator's callable function?

A

def task_func():
    xcom_push(key='result', value=42)

B

def task_func(ti):
    ti.xcom_push('result', 42)

C

def task_func(ti):
    ti.push_xcom(key='result', value=42)

D

def task_func(ti):
    ti.xcom_push(key='result', value=42)

💡 Hint

Remember the method name and parameters for pushing XCom data.

The correct method is ti.xcom_push with named parameters key and value.

🔀 Workflow

advanced

Best practice for sharing large data between tasks

Which approach is best for sharing large datasets between Airflow tasks?

A Store the data in an external storage like S3 or a database and pass the reference via XCom.

B Push the entire dataset directly via XCom between tasks.

C Serialize the data as a string and push it via XCom.

D Use global variables in the DAG file to share data.

💡 Hint

Think about XCom size limits and performance.

XCom is designed for small data. Large data should be stored externally and referenced by XCom to avoid performance issues.

❓ Troubleshoot

expert

Why does XCom pull return None unexpectedly?

You have two tasks: task_a pushes a value to XCom with key 'data', and task_b tries to pull it. But task_b always gets None. What is the most likely cause?

A task_b runs before task_a completes, so the XCom value is not yet available.

B The key used in task_b's xcom_pull is misspelled.

C XCom backend is disabled in Airflow configuration.

D task_a did not push any value to XCom.

💡 Hint

Consider task execution order and dependencies.

If task_b runs before task_a finishes, the XCom value won't exist yet, so xcom_pull returns None.