Sharing data between tasks effectively in Apache Airflow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is XCom in Airflow?

XCom (short for cross-communication) is a feature in Airflow that allows tasks to share small amounts of data between each other during a workflow run.

beginner

How do you push data to XCom in an Airflow task?

You can push data to XCom by using the task instance's xcom_push method inside a PythonOperator or by returning a value from a Python callable, which Airflow automatically pushes to XCom.

intermediate

What is the recommended size limit for data shared via XCom?

XCom is designed for small pieces of data like strings or small dictionaries. Large data should be stored externally (e.g., in a database or cloud storage) and referenced by XCom.

beginner

How can you pull data from XCom in a downstream task?

You can pull data from XCom using the xcom_pull method on the task instance, specifying the task ID and key of the data you want to retrieve.

intermediate

Why should you avoid sharing large data directly between tasks in Airflow?

Sharing large data directly can slow down your workflow and cause performance issues. Instead, store large data externally and share only references or metadata via XCom.

What does XCom stand for in Airflow?

A Cross-computation

B External communication

C Execution command

D Cross-communication

XCom stands for cross-communication, which is used for sharing data between tasks.

Which method is used to push data to XCom inside a task?

A push_data

B xcom_push

C xcom_send

D send_xcom

The xcom_push method is used to send data to XCom.

What is the best practice for sharing large data between Airflow tasks?

A Store data externally and share references via XCom

B Push large data directly via XCom

C Use environment variables

D Write data to local disk only

Large data should be stored externally and only references shared via XCom to avoid performance issues.

How do you retrieve data from XCom in a downstream task?

A Using xcom_pull method

B Using xcom_get method

C Using get_xcom method

D Using pull_xcom method

The xcom_pull method is used to get data from XCom.

What type of data is suitable for sharing via XCom?

A Database backups

B Large files like videos

C Small data like strings or dictionaries

D Entire datasets

XCom is designed for small data pieces, not large files or datasets.

Explain how Airflow tasks can share data effectively using XCom.

Think about how tasks send and receive small pieces of information.

Describe best practices for handling large data sharing between Airflow tasks.

Consider performance and data size limits.