Pipeline orchestration with Airflow in Apache Spark

Introduction

Airflow helps you run and manage data tasks in order. It makes sure each step happens at the right time.

You want to run a data cleaning job every day automatically.

You need to run a Spark job after a data file arrives.

You want to chain multiple data tasks, like extract, transform, and load.

You want to retry a task if it fails without manual work.

You want to see the status of your data jobs in one place.

Syntax

Apache Spark

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG('my_pipeline', start_date=datetime(2024, 1, 1), schedule_interval='@daily') as dag:
    task1 = BashOperator(
        task_id='run_spark_job',
        bash_command='spark-submit --class my.main.Class /path/to/app.jar'
    )

    task2 = BashOperator(
        task_id='cleanup',
        bash_command='rm -rf /tmp/data'
    )

    task1 >> task2

The DAG defines the workflow and schedule.

Tasks run in order using the '>>' operator.

Examples

This example runs a Spark job every day.

Apache Spark

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG('daily_spark_job', start_date=datetime(2024, 1, 1), schedule_interval='@daily') as dag:
    run_spark = BashOperator(
        task_id='run_spark',
        bash_command='spark-submit --master local /path/to/job.py'
    )

Two tasks run one after the other: extract then transform.

Apache Spark

task1 = BashOperator(
    task_id='extract_data',
    bash_command='python extract.py'
)

task2 = BashOperator(
    task_id='transform_data',
    bash_command='python transform.py'
)

task1 >> task2

Sample Program

This code defines two tasks: one simulates running a Spark job, the other cleans up files. They run in order.

Apache Spark

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG('spark_pipeline', start_date=datetime(2024, 1, 1), schedule_interval='@daily') as dag:
    run_spark = BashOperator(
        task_id='run_spark_job',
        bash_command='echo Running Spark job... && sleep 1 && echo Spark job done'
    )

    cleanup = BashOperator(
        task_id='cleanup_temp',
        bash_command='echo Cleaning up temporary files... && sleep 1 && echo Cleanup done'
    )

    run_spark >> cleanup

# Simulate running tasks by calling their bash_command manually
print("Task run_spark_job output:")
print("Running Spark job...\nSpark job done")
print("\nTask cleanup_temp output:")
print("Cleaning up temporary files...\nCleanup done")

Important Notes

Airflow uses DAGs (Directed Acyclic Graphs) to manage task order.

Each task can be a script, command, or Spark job.

You can see task status in Airflow's web interface.

Summary

Airflow helps automate and order data tasks.

Define tasks and their order using DAGs and operators.

It works well with Spark jobs and other scripts.

Practice

1. What is the main purpose of using Airflow in Apache Spark pipelines?

easy

A. To store large datasets

B. To write Spark SQL queries

C. To visualize Spark data

D. To schedule and manage the order of Spark jobs

Solution

1. Step 1: Understand Airflow's role

   Airflow is designed to schedule and orchestrate workflows, ensuring tasks run in the correct order.

2. Step 2: Compare with other options

   Writing Spark SQL, storing data, and visualizing data are not Airflow's main functions.

3. Final Answer:

   To schedule and manage the order of Spark jobs -> Option D

4. Quick Check:

   Airflow schedules pipelines = C [OK]

Hint: Airflow controls job order and timing [OK]

Common Mistakes:

• Confusing Airflow with Spark SQL or storage tools
• Thinking Airflow processes data directly
• Assuming Airflow is for data visualization

2. Which Airflow operator is commonly used to run a Spark job using spark-submit?

easy

A. BashOperator

B. PythonOperator

C. SparkOperator

D. EmailOperator

Solution

1. Step 1: Identify operator for shell commands

   BashOperator runs shell commands, perfect for spark-submit.

2. Step 2: Exclude other operators

   PythonOperator runs Python code, SparkOperator is not a default Airflow operator, EmailOperator sends emails.

3. Final Answer:

   BashOperator -> Option A

4. Quick Check:

   Run shell commands with BashOperator [OK]

Hint: Use BashOperator for shell commands like spark-submit [OK]

Common Mistakes:

• Choosing PythonOperator for shell commands
• Assuming SparkOperator exists by default
• Confusing EmailOperator with job execution

3. Given this Airflow DAG snippet, what is the order of task execution?

task1 = BashOperator(task_id='task1', bash_command='spark-submit job1.py', dag=dag)
task2 = BashOperator(task_id='task2', bash_command='spark-submit job2.py', dag=dag)
task3 = BashOperator(task_id='task3', bash_command='spark-submit job3.py', dag=dag)
task1 >> task2 >> task3

medium

A. task3 -> task2 -> task1

B. task1 -> task2 -> task3

C. All tasks run in parallel

D. task1 -> task3 -> task2

Solution

1. Step 1: Understand the dependency operator

   The operator >> means 'runs before', so task1 runs before task2, and task2 before task3.

2. Step 2: Determine execution order

   Tasks run sequentially: task1, then task2, then task3.

3. Final Answer:

   task1 -> task2 -> task3 -> Option B

4. Quick Check:

   Dependencies define order = D [OK]

Hint: Use >> to read task order left to right [OK]

Common Mistakes:

• Reading >> as reverse order
• Assuming tasks run in parallel without dependencies
• Mixing up task names and order

4. Identify the error in this Airflow DAG code snippet:

task1 = BashOperator(task_id='task1', bash_command='spark-submit job1.py', dag=dag)
task2 = BashOperator(task_id='task1', bash_command='spark-submit job2.py', dag=dag)
task1 >> task2

medium

A. Incorrect use of >> operator

B. Missing dag parameter in task2

C. Duplicate task_id 'task1' causes conflict

D. bash_command syntax is wrong

Solution

1. Step 1: Check task_id uniqueness

   Both tasks have the same task_id 'task1', which is not allowed in Airflow.

2. Step 2: Verify other parts

   dag parameter is present, >> operator is correctly used, bash_command syntax is valid.

3. Final Answer:

   Duplicate task_id 'task1' causes conflict -> Option C

4. Quick Check:

   Unique task_id needed = A [OK]

Hint: Each task_id must be unique in Airflow DAGs [OK]

Common Mistakes:

• Reusing task_id for different tasks
• Ignoring error messages about task_id
• Thinking >> operator causes error here

5. You want to run three Spark jobs in sequence using Airflow. Job 2 depends on Job 1, and Job 3 depends on Job 2. Which Airflow code snippet correctly sets this up?

hard

A. job1 = BashOperator(task_id='job1', bash_command='spark-submit job1.py', dag=dag)
job2 = BashOperator(task_id='job2', bash_command='spark-submit job2.py', dag=dag)
job3 = BashOperator(task_id='job3', bash_command='spark-submit job3.py', dag=dag)
job1.set_downstream(job2)
job2.set_downstream(job3)

B. job1 = BashOperator(task_id='job1', bash_command='spark-submit job1.py', dag=dag)
job2 = BashOperator(task_id='job2', bash_command='spark-submit job2.py', dag=dag)
job3 = BashOperator(task_id='job3', bash_command='spark-submit job3.py', dag=dag)
job3.set_downstream(job2)
job2.set_downstream(job1)

C. job1 = BashOperator(task_id='job1', bash_command='spark-submit job1.py', dag=dag)
job2 = BashOperator(task_id='job2', bash_command='spark-submit job2.py', dag=dag)
job3 = BashOperator(task_id='job3', bash_command='spark-submit job3.py', dag=dag)
job3 >> job2 >> job1

D. job1 = BashOperator(task_id='job1', bash_command='spark-submit job1.py', dag=dag)
job2 = BashOperator(task_id='job2', bash_command='spark-submit job2.py', dag=dag)
job3 = BashOperator(task_id='job3', bash_command='spark-submit job3.py', dag=dag)
job1 >> job3 >> job2

Solution

1. Step 1: Understand dependency direction

   job1 must run before job2, and job2 before job3, so job1 is upstream of job2, job2 upstream of job3.

2. Step 2: Check each option's dependency setup

   job1 = BashOperator(task_id='job1', bash_command='spark-submit job1.py', dag=dag)
   job2 = BashOperator(task_id='job2', bash_command='spark-submit job2.py', dag=dag)
   job3 = BashOperator(task_id='job3', bash_command='spark-submit job3.py', dag=dag)
   job1.set_downstream(job2)
   job2.set_downstream(job3)

   uses set_downstream correctly: job1 -> job2 -> job3. The other snippets reverse or mix the order incorrectly.

3. Final Answer:

   job1.set_downstream(job2); job2.set_downstream(job3) -> Option A

4. Quick Check:

   Use set_downstream for correct order = A [OK]

Hint: Use set_downstream or >> left to right for order [OK]

Common Mistakes:

• Reversing dependency direction
• Mixing task order in >> chains
• Confusing set_downstream with set_upstream