What is Apache Airflow - CLI Guide

Introduction

Apache Airflow helps you organize and run tasks automatically in a specific order. It solves the problem of managing many steps that depend on each other, like a recipe for a cake but for computer jobs.

When you want to run data processing steps one after another without doing it manually.

When you need to schedule tasks to run at certain times, like every day or every hour.

When you want to see if your tasks worked or failed and get alerts.

When you have many tasks that depend on each other and want to manage their order easily.

When you want to reuse and share task workflows with your team.

Commands

This command sets up the database Airflow uses to keep track of tasks and workflows.

Terminal

airflow db init

Expected Output

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.

INFO  [alembic.runtime.migration] Will assume transactional DDL.

INFO  [alembic.runtime.migration] Running upgrade head

INFO  [alembic.runtime.migration] Upgrade successful

Creates an admin user so you can log into the Airflow web interface and manage workflows.

Terminal

airflow users create --username admin --firstname Admin --lastname User --role Admin --email [e-mail] --password admin123

Expected Output

User created successfully

→

--username - Set the login name

→

--role - Set user permissions

→

--password - Set the login password

Starts the Airflow web interface on port 8080 so you can see and control your workflows in a browser.

Terminal

airflow webserver --port 8080

Expected Output

[2024-06-01 12:00:00,000] {webserver.py:123} INFO - Starting web server on port 8080

→

--port - Choose the port number for the web interface

Starts the scheduler that runs your tasks at the right time and in the right order.

Terminal

airflow scheduler

Expected Output

[2024-06-01 12:00:05,000] {scheduler.py:456} INFO - Starting the scheduler

Key Concept

If you remember nothing else, remember: Apache Airflow lets you automate and monitor complex task workflows easily.

Common Mistakes

Not initializing the database before starting Airflow

Airflow needs the database to store task info; without it, Airflow won't work.

Always run 'airflow db init' before starting the webserver or scheduler.

Forgetting to create a user before accessing the web interface

Without a user, you cannot log in to manage workflows.

Create at least one user with 'airflow users create' before starting the webserver.

Running the webserver and scheduler in the same terminal without backgrounding

One process will block the other, so both won't run properly.

Run them in separate terminals or use background processes.

Summary

Initialize Airflow's database with 'airflow db init' to prepare the system.

Create a user to access the Airflow web interface using 'airflow users create'.

Start the webserver to view and control workflows in a browser.

Run the scheduler to execute tasks automatically in the right order.

Practice

1. What is the main purpose of Apache Airflow?

easy

A. To monitor network traffic

B. To store large amounts of data

C. To create user interfaces for websites

D. To automate and schedule tasks in a specific order

Solution

1. Step 1: Understand Apache Airflow's role

   Apache Airflow is designed to automate and schedule tasks, not to store data or create interfaces.

2. Step 2: Identify the correct purpose

   It manages workflows by running tasks in a defined order automatically.

3. Final Answer:

   To automate and schedule tasks in a specific order -> Option D

4. Quick Check:

   Automation and scheduling = A [OK]

Hint: Airflow schedules and runs tasks automatically [OK]

Common Mistakes:

• Confusing Airflow with data storage tools
• Thinking Airflow builds websites
• Assuming Airflow monitors networks

2. Which of the following is the correct way to define a DAG in Apache Airflow using Python?

easy

A. dag = DAG('my_dag', schedule_interval='@daily')

B. dag = DAG('my_dag' schedule_interval='@daily')

C. dag = DAG['my_dag', schedule_interval='@daily']

D. dag = DAG{'my_dag', schedule_interval='@daily'}

Solution

1. Step 1: Check Python syntax for function calls

   Function calls use parentheses with arguments separated by commas.

2. Step 2: Identify correct syntax

   dag = DAG('my_dag', schedule_interval='@daily') uses parentheses and commas correctly; others have missing commas or wrong brackets.

3. Final Answer:

   dag = DAG('my_dag', schedule_interval='@daily') -> Option A

4. Quick Check:

   Function call syntax = A [OK]

Hint: Use parentheses and commas for function arguments [OK]

Common Mistakes:

• Missing commas between arguments
• Using square or curly brackets instead of parentheses
• Incorrect argument separation

3. Given the following Airflow DAG snippet, what will be the order of task execution?

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

dag = DAG('example_dag', start_date=datetime(2023,1,1))

t1 = DummyOperator(task_id='task1', dag=dag)
t2 = DummyOperator(task_id='task2', dag=dag)
t3 = DummyOperator(task_id='task3', dag=dag)

t1 >> t2 >> t3

medium

A. All tasks run at the same time

B. task3 runs, then task2, then task1

C. task1 runs, then task2, then task3

D. task2 runs first, then task1, then task3

Solution

1. Step 1: Understand the '>>' operator in Airflow

   The '>>' operator sets task dependencies, meaning left task runs before right task.

2. Step 2: Trace the task order

   t1 >> t2 >> t3 means task1 runs first, then task2, then task3.

3. Final Answer:

   task1 runs, then task2, then task3 -> Option C

4. Quick Check:

   Dependency chain = task1 -> task2 -> task3 [OK]

Hint: The '>>' operator means 'runs before' [OK]

Common Mistakes:

• Thinking tasks run in reverse order
• Assuming tasks run in parallel without dependencies
• Misreading the '>>' operator

4. You wrote this Airflow DAG code but it fails to run:

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

dag = DAG('my_dag', start_date=datetime(2023,1,1))

t1 = DummyOperator(task_id='task1')
t2 = DummyOperator(task_id='task2')

t1 >> t2

What is the error?

medium

A. Task IDs must be integers

B. Missing DAG parameter in DummyOperator tasks

C. start_date must be a string

D. Wrong import for DummyOperator

Solution

1. Step 1: Check task definitions

   Tasks need to be linked to a DAG by passing the 'dag' parameter.

2. Step 2: Identify missing 'dag' argument

   t1 and t2 are missing 'dag=dag', so Airflow doesn't know which DAG they belong to.

3. Final Answer:

   Missing DAG parameter in DummyOperator tasks -> Option B

4. Quick Check:

   Tasks need DAG assigned = D [OK]

Hint: Always assign tasks to a DAG with dag=dag [OK]

Common Mistakes:

• Forgetting to pass dag=dag to tasks
• Incorrect imports causing errors
• Wrong data types for start_date or task_id

5. You want to create an Airflow workflow that retries a task up to 3 times if it fails. Which DAG task parameter should you set?

hard

A. retries=3

B. retry_count=3

C. max_retry=3

D. retry_limit=3

Solution

1. Step 1: Recall Airflow retry parameter

   Airflow uses 'retries' to specify how many times to retry a failed task.

2. Step 2: Match parameter name

   The correct parameter is 'retries', not retry_count, max_retry, or retry_limit.

3. Final Answer:

   retries=3 -> Option A

4. Quick Check:

   Retries parameter controls retry count = B [OK]

Hint: Use 'retries' to set retry attempts in Airflow tasks [OK]

Common Mistakes:

• Using incorrect parameter names like retry_count
• Confusing retry count with timeout
• Not setting retries at all