Pipeline orchestration with Airflow in Apache Spark - Time & Space Complexity

Time Complexity: Pipeline orchestration with Airflow

O(n)

Understanding Time Complexity

We want to understand how the time to run a data pipeline changes as the amount of data or tasks grows.

How does adding more steps or data affect the total execution time?

Scenario Under Consideration

Analyze the time complexity of the following Airflow DAG that runs Spark jobs sequentially.

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

def spark_job(task_id, script):
    return BashOperator(
        task_id=task_id,
        bash_command=f"spark-submit {script}",
        dag=dag
    )

dag = DAG('spark_pipeline', start_date=datetime(2024,1,1))

job1 = spark_job('job1', 'job1.py')
job2 = spark_job('job2', 'job2.py')
job3 = spark_job('job3', 'job3.py')

job1 >> job2 >> job3

This code defines three Spark jobs run one after another using Airflow orchestration.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Running each Spark job one after another.
• How many times: Once per job, here 3 times, but can grow with more jobs.

How Execution Grows With Input

As you add more Spark jobs to the pipeline, the total time grows roughly by adding each job's time.

Input Size (number of jobs)	Approx. Operations (job runs)
3	3 Spark job runs
10	10 Spark job runs
100	100 Spark job runs

Pattern observation: The total execution time grows linearly as you add more jobs.

Final Time Complexity

Time Complexity: O(n)

This means the total time grows directly in proportion to the number of jobs in the pipeline.

Common Mistake

[X] Wrong: "Adding more jobs won't increase total time because they run in parallel automatically."

[OK] Correct: In this setup, jobs run one after another, so each job adds to total time. Parallelism must be explicitly set.

Interview Connect

Understanding how pipeline steps add up in time helps you design efficient workflows and explain your reasoning clearly in interviews.

Self-Check

"What if we changed the pipeline to run some jobs in parallel? How would the time complexity change?"

Practice

1. What is the main purpose of using Airflow in Apache Spark pipelines?

easy

A. To store large datasets

B. To write Spark SQL queries

C. To visualize Spark data

D. To schedule and manage the order of Spark jobs

Solution

1. Step 1: Understand Airflow's role

   Airflow is designed to schedule and orchestrate workflows, ensuring tasks run in the correct order.

2. Step 2: Compare with other options

   Writing Spark SQL, storing data, and visualizing data are not Airflow's main functions.

3. Final Answer:

   To schedule and manage the order of Spark jobs -> Option D

4. Quick Check:

   Airflow schedules pipelines = C [OK]

Hint: Airflow controls job order and timing [OK]

Common Mistakes:

• Confusing Airflow with Spark SQL or storage tools
• Thinking Airflow processes data directly
• Assuming Airflow is for data visualization

2. Which Airflow operator is commonly used to run a Spark job using spark-submit?

easy

A. BashOperator

B. PythonOperator

C. SparkOperator

D. EmailOperator

Solution

1. Step 1: Identify operator for shell commands

   BashOperator runs shell commands, perfect for spark-submit.

2. Step 2: Exclude other operators

   PythonOperator runs Python code, SparkOperator is not a default Airflow operator, EmailOperator sends emails.

3. Final Answer:

   BashOperator -> Option A

4. Quick Check:

   Run shell commands with BashOperator [OK]

Hint: Use BashOperator for shell commands like spark-submit [OK]

Common Mistakes:

• Choosing PythonOperator for shell commands
• Assuming SparkOperator exists by default
• Confusing EmailOperator with job execution

3. Given this Airflow DAG snippet, what is the order of task execution?

task1 = BashOperator(task_id='task1', bash_command='spark-submit job1.py', dag=dag)
task2 = BashOperator(task_id='task2', bash_command='spark-submit job2.py', dag=dag)
task3 = BashOperator(task_id='task3', bash_command='spark-submit job3.py', dag=dag)
task1 >> task2 >> task3

medium

A. task3 -> task2 -> task1

B. task1 -> task2 -> task3

C. All tasks run in parallel

D. task1 -> task3 -> task2

Solution

1. Step 1: Understand the dependency operator

   The operator >> means 'runs before', so task1 runs before task2, and task2 before task3.

2. Step 2: Determine execution order

   Tasks run sequentially: task1, then task2, then task3.

3. Final Answer:

   task1 -> task2 -> task3 -> Option B

4. Quick Check:

   Dependencies define order = D [OK]

Hint: Use >> to read task order left to right [OK]

Common Mistakes:

• Reading >> as reverse order
• Assuming tasks run in parallel without dependencies
• Mixing up task names and order

4. Identify the error in this Airflow DAG code snippet:

task1 = BashOperator(task_id='task1', bash_command='spark-submit job1.py', dag=dag)
task2 = BashOperator(task_id='task1', bash_command='spark-submit job2.py', dag=dag)
task1 >> task2

medium

A. Incorrect use of >> operator

B. Missing dag parameter in task2

C. Duplicate task_id 'task1' causes conflict

D. bash_command syntax is wrong

Solution

1. Step 1: Check task_id uniqueness

   Both tasks have the same task_id 'task1', which is not allowed in Airflow.

2. Step 2: Verify other parts

   dag parameter is present, >> operator is correctly used, bash_command syntax is valid.

3. Final Answer:

   Duplicate task_id 'task1' causes conflict -> Option C

4. Quick Check:

   Unique task_id needed = A [OK]

Hint: Each task_id must be unique in Airflow DAGs [OK]

Common Mistakes:

• Reusing task_id for different tasks
• Ignoring error messages about task_id
• Thinking >> operator causes error here

5. You want to run three Spark jobs in sequence using Airflow. Job 2 depends on Job 1, and Job 3 depends on Job 2. Which Airflow code snippet correctly sets this up?

hard

A. job1 = BashOperator(task_id='job1', bash_command='spark-submit job1.py', dag=dag)
job2 = BashOperator(task_id='job2', bash_command='spark-submit job2.py', dag=dag)
job3 = BashOperator(task_id='job3', bash_command='spark-submit job3.py', dag=dag)
job1.set_downstream(job2)
job2.set_downstream(job3)

B. job1 = BashOperator(task_id='job1', bash_command='spark-submit job1.py', dag=dag)
job2 = BashOperator(task_id='job2', bash_command='spark-submit job2.py', dag=dag)
job3 = BashOperator(task_id='job3', bash_command='spark-submit job3.py', dag=dag)
job3.set_downstream(job2)
job2.set_downstream(job1)

C. job1 = BashOperator(task_id='job1', bash_command='spark-submit job1.py', dag=dag)
job2 = BashOperator(task_id='job2', bash_command='spark-submit job2.py', dag=dag)
job3 = BashOperator(task_id='job3', bash_command='spark-submit job3.py', dag=dag)
job3 >> job2 >> job1

D. job1 = BashOperator(task_id='job1', bash_command='spark-submit job1.py', dag=dag)
job2 = BashOperator(task_id='job2', bash_command='spark-submit job2.py', dag=dag)
job3 = BashOperator(task_id='job3', bash_command='spark-submit job3.py', dag=dag)
job1 >> job3 >> job2

Solution

1. Step 1: Understand dependency direction

   job1 must run before job2, and job2 before job3, so job1 is upstream of job2, job2 upstream of job3.

2. Step 2: Check each option's dependency setup

   job1 = BashOperator(task_id='job1', bash_command='spark-submit job1.py', dag=dag)
   job2 = BashOperator(task_id='job2', bash_command='spark-submit job2.py', dag=dag)
   job3 = BashOperator(task_id='job3', bash_command='spark-submit job3.py', dag=dag)
   job1.set_downstream(job2)
   job2.set_downstream(job3)

   uses set_downstream correctly: job1 -> job2 -> job3. The other snippets reverse or mix the order incorrectly.

3. Final Answer:

   job1.set_downstream(job2); job2.set_downstream(job3) -> Option A

4. Quick Check:

   Use set_downstream for correct order = A [OK]

Hint: Use set_downstream or >> left to right for order [OK]

Common Mistakes:

• Reversing dependency direction
• Mixing task order in >> chains
• Confusing set_downstream with set_upstream