What is Apache Airflow - Practice Questions & Exercises

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Apache Airflow's Purpose

What is the primary purpose of Apache Airflow in a DevOps environment?

A To automate and schedule complex workflows and data pipelines

B To serve as a web server for hosting websites

C To monitor network traffic and detect intrusions

D To manage container orchestration like Kubernetes

💡 Hint

Think about what Airflow helps automate and organize.

Apache Airflow is designed to automate, schedule, and monitor workflows, especially data pipelines, making complex tasks easier to manage.

💻 Command Output

intermediate

Airflow Webserver Startup Output

What output will you see when you successfully start the Apache Airflow webserver using the command airflow webserver?

Apache Airflow

airflow webserver

A Error: command not found

B

Starting the web server on port 8080
Listening on http://localhost:8080

C Database connection failed

D Kubernetes cluster not found

💡 Hint

Look for a message indicating the webserver is running and the port number.

When the Airflow webserver starts successfully, it listens on port 8080 by default and shows a message with the URL.

❓ Configuration

advanced

Configuring Airflow Executor

Which configuration setting in airflow.cfg correctly sets the executor to use CeleryExecutor?

Apache Airflow

[core]
executor = SequentialExecutor

A

[core]
executor = CeleryExecutor

B

[core]
executor = KubernetesExecutor

C

[core]
executor = SequentialExecutor

D

[core]
executor = LocalExecutor

💡 Hint

The executor setting defines how tasks are run in Airflow.

Setting executor = CeleryExecutor enables distributed task execution using Celery workers.

🔀 Workflow

advanced

Airflow DAG Scheduling Behavior

Given a DAG scheduled with schedule_interval='@daily' and a start date of 2024-06-01, when will the first DAG run be triggered?

A At midnight on 2024-06-01

B Immediately when the DAG is deployed

C At midnight on 2024-06-02

D At noon on 2024-06-01

💡 Hint

Airflow schedules runs after the period ends.

Airflow triggers the first run at the end of the first schedule interval, so the run for 2024-06-01 is triggered at midnight on 2024-06-02.

❓ Troubleshoot

expert

Diagnosing Airflow Task Failure

You notice a task in your Airflow DAG is stuck in the 'queued' state indefinitely. Which is the most likely cause?

A The Airflow webserver is down

B The DAG file has syntax errors

C The task has already succeeded

D No available worker to pick up the task

💡 Hint

Think about what controls task execution after queuing.

If no worker is available, tasks remain queued and do not start running.

Practice

1. What is the main purpose of Apache Airflow?

easy

A. To monitor network traffic

B. To store large amounts of data

C. To create user interfaces for websites

D. To automate and schedule tasks in a specific order

Solution

1. Step 1: Understand Apache Airflow's role

   Apache Airflow is designed to automate and schedule tasks, not to store data or create interfaces.

2. Step 2: Identify the correct purpose

   It manages workflows by running tasks in a defined order automatically.

3. Final Answer:

   To automate and schedule tasks in a specific order -> Option D

4. Quick Check:

   Automation and scheduling = A [OK]

Hint: Airflow schedules and runs tasks automatically [OK]

Common Mistakes:

• Confusing Airflow with data storage tools
• Thinking Airflow builds websites
• Assuming Airflow monitors networks

2. Which of the following is the correct way to define a DAG in Apache Airflow using Python?

easy

A. dag = DAG('my_dag', schedule_interval='@daily')

B. dag = DAG('my_dag' schedule_interval='@daily')

C. dag = DAG['my_dag', schedule_interval='@daily']

D. dag = DAG{'my_dag', schedule_interval='@daily'}

Solution

1. Step 1: Check Python syntax for function calls

   Function calls use parentheses with arguments separated by commas.

2. Step 2: Identify correct syntax

   dag = DAG('my_dag', schedule_interval='@daily') uses parentheses and commas correctly; others have missing commas or wrong brackets.

3. Final Answer:

   dag = DAG('my_dag', schedule_interval='@daily') -> Option A

4. Quick Check:

   Function call syntax = A [OK]

Hint: Use parentheses and commas for function arguments [OK]

Common Mistakes:

• Missing commas between arguments
• Using square or curly brackets instead of parentheses
• Incorrect argument separation

3. Given the following Airflow DAG snippet, what will be the order of task execution?

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

dag = DAG('example_dag', start_date=datetime(2023,1,1))

t1 = DummyOperator(task_id='task1', dag=dag)
t2 = DummyOperator(task_id='task2', dag=dag)
t3 = DummyOperator(task_id='task3', dag=dag)

t1 >> t2 >> t3

medium

A. All tasks run at the same time

B. task3 runs, then task2, then task1

C. task1 runs, then task2, then task3

D. task2 runs first, then task1, then task3

Solution

1. Step 1: Understand the '>>' operator in Airflow

   The '>>' operator sets task dependencies, meaning left task runs before right task.

2. Step 2: Trace the task order

   t1 >> t2 >> t3 means task1 runs first, then task2, then task3.

3. Final Answer:

   task1 runs, then task2, then task3 -> Option C

4. Quick Check:

   Dependency chain = task1 -> task2 -> task3 [OK]

Hint: The '>>' operator means 'runs before' [OK]

Common Mistakes:

• Thinking tasks run in reverse order
• Assuming tasks run in parallel without dependencies
• Misreading the '>>' operator

4. You wrote this Airflow DAG code but it fails to run:

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

dag = DAG('my_dag', start_date=datetime(2023,1,1))

t1 = DummyOperator(task_id='task1')
t2 = DummyOperator(task_id='task2')

t1 >> t2

What is the error?

medium

A. Task IDs must be integers

B. Missing DAG parameter in DummyOperator tasks

C. start_date must be a string

D. Wrong import for DummyOperator

Solution

1. Step 1: Check task definitions

   Tasks need to be linked to a DAG by passing the 'dag' parameter.

2. Step 2: Identify missing 'dag' argument

   t1 and t2 are missing 'dag=dag', so Airflow doesn't know which DAG they belong to.

3. Final Answer:

   Missing DAG parameter in DummyOperator tasks -> Option B

4. Quick Check:

   Tasks need DAG assigned = D [OK]

Hint: Always assign tasks to a DAG with dag=dag [OK]

Common Mistakes:

• Forgetting to pass dag=dag to tasks
• Incorrect imports causing errors
• Wrong data types for start_date or task_id

5. You want to create an Airflow workflow that retries a task up to 3 times if it fails. Which DAG task parameter should you set?

hard

A. retries=3

B. retry_count=3

C. max_retry=3

D. retry_limit=3

Solution

1. Step 1: Recall Airflow retry parameter

   Airflow uses 'retries' to specify how many times to retry a failed task.

2. Step 2: Match parameter name

   The correct parameter is 'retries', not retry_count, max_retry, or retry_limit.

3. Final Answer:

   retries=3 -> Option A

4. Quick Check:

   Retries parameter controls retry count = B [OK]

Hint: Use 'retries' to set retry attempts in Airflow tasks [OK]

Common Mistakes:

• Using incorrect parameter names like retry_count
• Confusing retry count with timeout
• Not setting retries at all