Bird
Raised Fist0
Apache Airflowdevops~5 mins

Creating a basic DAG file in Apache Airflow - Performance & Efficiency

Choose your learning style10 modes available
LearnWhyDeepVisualTryChallengeProjectRecallTime
Time Complexity: Creating a basic DAG file
O(n)
Understanding Time Complexity

When we create a basic DAG file in Airflow, we want to know how the time it takes to run grows as we add more tasks.

We ask: How does adding more tasks affect the total work Airflow does?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

default_args = {'start_date': datetime(2024, 1, 1)}

dag = DAG('basic_dag', default_args=default_args, schedule_interval='@daily')

task1 = BashOperator(task_id='task1', bash_command='echo Hello', dag=dag)
task2 = BashOperator(task_id='task2', bash_command='echo World', dag=dag)

task1 >> task2

This code creates a DAG with two tasks that run simple shell commands in order.

Identify Repeating Operations
  • Primary operation: Defining tasks and their dependencies in the DAG.
  • How many times: Each task is defined once; no loops or recursion in this snippet.
How Execution Grows With Input

As you add more tasks, the number of operations grows roughly in a straight line.

Input Size (n tasks)Approx. Operations
1010 task definitions
100100 task definitions
10001000 task definitions

Pattern observation: Adding tasks increases work evenly, one by one.

Final Time Complexity

Time Complexity: O(n)

This means the time to create the DAG grows directly with the number of tasks.

Common Mistake

[X] Wrong: "Adding more tasks will make the DAG creation time grow much faster, like squared or exponential."

[OK] Correct: Each task is defined once without nested loops, so the work grows evenly, not faster.

Interview Connect

Understanding how task count affects DAG creation helps you design workflows that scale well and run smoothly.

Self-Check

"What if we added nested loops to create tasks dynamically? How would the time complexity change?"

Practice

(1/5)
1. What is the main purpose of a DAG file in Apache Airflow?
easy
A. To configure the Airflow webserver settings
B. To store logs generated by tasks
C. To install Python packages required by tasks
D. To define tasks and their schedule for workflow automation

Solution

  1. Step 1: Understand what a DAG file represents

    A DAG file in Airflow is a Python script that defines a Directed Acyclic Graph, which represents tasks and their dependencies.

  2. Step 2: Identify the role of the DAG file

    The DAG file schedules and organizes tasks to automate workflows, not for logging, configuration, or package installation.

  3. Final Answer:

    To define tasks and their schedule for workflow automation -> Option D

  4. Quick Check:

    DAG file purpose = Define tasks and schedule [OK]
Hint: DAG files define workflows, not configs or logs [OK]
Common Mistakes:
  • Confusing DAG file with Airflow config files
  • Thinking DAG files store logs
  • Assuming DAG files install packages
2. Which of the following is the correct way to import the DAG class in an Airflow DAG file?
easy
A. import DAG from airflow.models
B. import airflow.DAG
C. from airflow.models import DAG
D. from airflow import DAG

Solution

  1. Step 1: Recall the correct import syntax in Python

    Python imports use 'from module import class' syntax for specific classes.

  2. Step 2: Match with Airflow's module structure

    The recommended import used in official Airflow documentation is 'from airflow.models import DAG'.

  3. Final Answer:

    from airflow.models import DAG -> Option C

  4. Quick Check:

    Correct import syntax = from airflow.models import DAG [OK]
Hint: Use 'from airflow.models import DAG' to import DAG class [OK]
Common Mistakes:
  • Using incorrect import syntax
  • Trying to import DAG directly from airflow
  • Using invalid Python import statements
3. Given the following DAG snippet, what will be the task_id of the BashOperator task? 
from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

default_args = {'start_date': datetime(2024, 1, 1)}
with DAG('example_dag', default_args=default_args, schedule_interval='@daily') as dag:
    task = BashOperator(task_id='print_date', bash_command='date')
medium
A. print_date
B. task
C. example_dag
D. bash_command

Solution

  1. Step 1: Identify the task_id parameter in BashOperator

    The BashOperator is created with task_id='print_date', which names the task.

  2. Step 2: Understand task_id usage

    task_id is the unique identifier for the task in the DAG, so the task's id is 'print_date'.

  3. Final Answer:

    print_date -> Option A

  4. Quick Check:

    task_id value = print_date [OK]
Hint: Look for task_id argument in operator constructor [OK]
Common Mistakes:
  • Confusing variable name with task_id
  • Using DAG id as task_id
  • Mistaking bash_command as task_id
4. Identify the error in this DAG file snippet: 
from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG('my_dag', start_date=datetime(2024, 1, 1), schedule_interval='@daily') as dag:
    task1 = BashOperator(task_id='task1', bash_command='echo Hello')
    task2 = BashOperator(task_id='task2', bash_command='echo World')
    task1 >> task2

print('DAG created')
medium
A. No error, DAG is correctly defined
B. Missing default_args dictionary for start_date
C. Incorrect import of BashOperator
D. Using print statement inside DAG context

Solution

  1. Step 1: Check imports and DAG definition

    Imports are correct, DAG is created with start_date and schedule_interval directly in constructor.

  2. Step 2: Verify task creation and dependencies

    Tasks are created with valid task_ids and bash commands, and dependency is set with 'task1 >> task2'.

  3. Step 3: Confirm print statement usage

    Print outside DAG context is allowed and does not affect DAG creation.

  4. Final Answer:

    No error, DAG is correctly defined -> Option A

  5. Quick Check:

    DAG syntax correct = No error [OK]
Hint: Check if DAG and tasks follow Airflow syntax rules [OK]
Common Mistakes:
  • Assuming default_args is mandatory
  • Thinking print inside DAG causes error
  • Confusing import paths for operators
5. You want to create a DAG that runs a task every Monday at 7 AM starting from March 1, 2024. Which schedule_interval and start_date combination is correct in the DAG definition?
hard
A. schedule_interval='@weekly', start_date=datetime(2024, 3, 4)
B. schedule_interval='0 7 * * 1', start_date=datetime(2024, 3, 1)
C. schedule_interval='@daily', start_date=datetime(2024, 3, 1, 7, 0)
D. schedule_interval='0 7 * * 0', start_date=datetime(2024, 3, 1)

Solution

  1. Step 1: Determine the cron expression for Monday 7 AM

    Cron '0 7 * * 1' runs every Monday at 7:00 AM (1=Monday).

  2. Step 2: Validate start_date

    March 1, 2024 (Friday) is before the first Monday (March 4), so first run on March 4, 7 AM.

  3. Step 3: Eliminate other options

    A: '@weekly'='0 0 * * 0' (Sunday midnight), wrong day/time. C: daily. D: Sunday (0).

  4. Final Answer:

    schedule_interval='0 7 * * 1', start_date=datetime(2024, 3, 1) -> Option B

  5. Quick Check:

    Monday 7AM cron = '0 7 * * 1' [OK]
Hint: Use cron '0 7 * * 1' for Mon 7AM [OK]
Common Mistakes:
  • Setting start_date on wrong weekday
  • Confusing cron day numbers (0=Sunday, 1=Monday)
  • Using daily schedule for weekly task
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard