Why best practices prevent technical debt in Apache Airflow - Test Your Understanding

Practice - 5 Tasks

1fill in blank

easy

Complete the code to import the Airflow DAG class.

Apache Airflow

from airflow.[1] import DAG

Drag options to blanks, or click blank then click option'

Ahooks

Boperators

Cmodels

Dutils

Attempts:

3 left

💡 Hint

Common Mistakes

Importing DAG from airflow.operators instead of airflow.models.

The DAG class is imported from airflow.models to define workflows.

2fill in blank

medium

Complete the code to set the default arguments for the DAG.

Apache Airflow

default_args = {
    'owner': 'airflow',
    'start_date': [1]
}

Drag options to blanks, or click blank then click option'

Adatetime(2023, 1, 1)

Bdatetime.now()

Cdatetime.today()

Ddatetime.utcnow()

Attempts:

3 left

💡 Hint

Common Mistakes

Using datetime.now() causes DAG to run repeatedly from current time.

Using a fixed start_date like datetime(2023, 1, 1) ensures consistent DAG runs and avoids unexpected behavior.

3fill in blank

hard

Fix the error in the task definition by completing the operator name.

Apache Airflow

task1 = [1](
    task_id='print_date',
    dag=dag
)

Drag options to blanks, or click blank then click option'

APythonOperator

BDummyOperator

CEmailOperator

DBashOperator

Attempts:

3 left

💡 Hint

Common Mistakes

Using PythonOperator without defining a python_callable.

BashOperator runs bash commands, which fits a task named 'print_date'.

4fill in blank

hard

Fill both blanks to create a dictionary comprehension that filters tasks with retries greater than 1.

Apache Airflow

filtered_tasks = {task.task_id: task for task in tasks if task.[1] [2] 1}

Drag options to blanks, or click blank then click option'

Aretries

B>

C<

Dtask_id

Attempts:

3 left

💡 Hint

Common Mistakes

Using task_id instead of retries for filtering.

This comprehension selects tasks where retries is greater than 1, helping identify tasks needing attention.

5fill in blank

hard

Fill all three blanks to create a dictionary of task_ids and owners for tasks that are active.

Apache Airflow

active_tasks = {task.[1]: task.[2] for task in tasks if task.[3] == True}

Drag options to blanks, or click blank then click option'

Atask_id

Bowner

Cis_active

Dstart_date

Attempts:

3 left

💡 Hint

Common Mistakes

Using start_date instead of is_active for filtering.

This comprehension collects task IDs and owners only for tasks marked as active, helping manage workflow ownership.

Practice

1. Why is it important to follow best practices when creating Airflow workflows?

easy

A. To avoid using any external libraries

B. To prevent technical debt and make workflows easier to maintain

C. To make the code run faster regardless of readability

D. To ensure workflows never fail

Solution

1. Step 1: Understand the purpose of best practices

   Best practices help keep code clean, organized, and easier to update.

2. Step 2: Connect best practices to technical debt

   Following best practices prevents technical debt, which is the extra work caused by poor code quality.

3. Final Answer:

   To prevent technical debt and make workflows easier to maintain -> Option B

4. Quick Check:

   Best practices = prevent technical debt [OK]

Hint: Best practices reduce future problems and save time [OK]

Common Mistakes:

• Thinking speed is the only goal
• Ignoring code readability
• Assuming workflows never fail if best practices are followed

2. Which of the following is a correct way to name an Airflow DAG following best practices?

easy

A. 123dag

B. MyDAG

C. dag_123

D. data_pipeline_daily

Solution

1. Step 1: Identify naming conventions

   Best practice is to use clear, descriptive, and lowercase names with underscores.

2. Step 2: Evaluate options

   data_pipeline_daily uses lowercase letters and underscores describing the DAG purpose and schedule.

3. Final Answer:

   data_pipeline_daily -> Option D

4. Quick Check:

   Clear descriptive names = data_pipeline_daily [OK]

Hint: Use lowercase and underscores for clear DAG names [OK]

Common Mistakes:

• Using uppercase letters or spaces
• Starting names with numbers
• Using vague or short names

3. Given this Airflow DAG snippet, what is the main benefit of using modular tasks with clear dependencies?

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def task_a():
    print('Task A')

def task_b():
    print('Task B')

def task_c():
    print('Task C')

with DAG('modular_dag', start_date=datetime(2024, 1, 1), schedule_interval='@daily') as dag:
    a = PythonOperator(task_id='task_a', python_callable=task_a)
    b = PythonOperator(task_id='task_b', python_callable=task_b)
    c = PythonOperator(task_id='task_c', python_callable=task_c)

    a >> b >> c

medium

A. It makes the DAG harder to read and maintain

B. It allows tasks to run in parallel without order

C. It improves clarity and makes debugging easier

D. It prevents the DAG from running on schedule

Solution

1. Step 1: Understand modular tasks and dependencies

   Breaking workflows into small tasks with clear order helps understand and fix issues.

2. Step 2: Analyze the DAG structure

   The tasks run in sequence (a then b then c), which is clear and easy to follow.

3. Final Answer:

   It improves clarity and makes debugging easier -> Option C

4. Quick Check:

   Modular tasks + clear order = easier debugging [OK]

Hint: Clear task order helps find and fix problems fast [OK]

Common Mistakes:

• Thinking modular means tasks run randomly
• Assuming modular code is more complex
• Believing dependencies slow down the DAG

4. You have an Airflow DAG with duplicated task IDs causing errors. What is the best way to fix this technical debt?

medium

A. Rename duplicated task IDs to unique names

B. Delete all tasks and start over

C. Ignore the error and run the DAG anyway

D. Add more tasks with the same IDs

Solution

1. Step 1: Identify the cause of error

   Duplicated task IDs cause Airflow to fail because each task must have a unique ID.

2. Step 2: Apply best practice to fix

   Renaming tasks to unique IDs removes the conflict and prevents technical debt buildup.

3. Final Answer:

   Rename duplicated task IDs to unique names -> Option A

4. Quick Check:

   Unique task IDs fix duplication errors [OK]

Hint: Each task ID must be unique to avoid errors [OK]

Common Mistakes:

• Ignoring errors and hoping for the best
• Deleting all tasks unnecessarily
• Adding more duplicates instead of fixing

5. You want to prevent technical debt in your Airflow project by improving testing. Which approach best follows best practices?

hard

A. Write unit tests for individual tasks and integration tests for the DAG

B. Only test the DAG by running it manually once

C. Skip testing to save time and fix bugs later

D. Test tasks only after deployment to production

Solution

1. Step 1: Understand testing levels in Airflow

   Unit tests check single tasks; integration tests check how tasks work together in the DAG.

2. Step 2: Choose the best testing approach

   Writing both unit and integration tests catches errors early and prevents technical debt.

3. Final Answer:

   Write unit tests for individual tasks and integration tests for the DAG -> Option A

4. Quick Check:

   Unit + integration tests = less technical debt [OK]

Hint: Test small parts and whole DAG to catch bugs early [OK]

Common Mistakes:

• Skipping tests to save time
• Testing only after deployment
• Relying on manual runs instead of automated tests