Airflow UI overview - Mini Project: Build & Apply

Airflow UI Overview

📖 Scenario: You are a data engineer starting to use Apache Airflow to manage workflows. You want to understand how to interact with Airflow's web interface to monitor and control your workflows easily.

🎯 Goal: Build a simple Airflow DAG and learn how to navigate the Airflow UI to see DAG status, task details, and logs.

📋 What You'll Learn

Create a basic Airflow DAG with two tasks

Set a DAG configuration variable for scheduling

Use Airflow's PythonOperator to define tasks

Print the DAG run status in the Airflow UI

💡 Why This Matters

🌍 Real World

Airflow is widely used to automate and monitor workflows in data engineering and software projects. Understanding the UI helps you track and troubleshoot your workflows easily.

💼 Career

Knowing how to create DAGs and use the Airflow UI is essential for roles like Data Engineer, DevOps Engineer, and Workflow Automation Specialist.

1

Create a basic Airflow DAG

Create a Python file with a DAG named example_dag using Airflow's DAG class. Set start_date to 2024-01-01 and schedule_interval to @daily. Import necessary modules from airflow and datetime.

Apache Airflow

# Import DAG and datetime modules
# Create a DAG named example_dag with start_date 2024-01-01 and schedule_interval '@daily'
# Your code here

Hint

Use DAG class from airflow and datetime module to set start date.

2

Add a configuration variable for retries

Add a dictionary called default_args with a key 'retries' set to 1. Pass default_args=default_args when creating the DAG example_dag.

Apache Airflow

from airflow import DAG
from datetime import datetime

# Create default_args dictionary with retries set to 1
# Your code here

dag = DAG(
    'example_dag',
    start_date=datetime(2024, 1, 1),
    schedule_interval='@daily',
    default_args=default_args
)

Hint

Define default_args before the DAG and pass it as a parameter.

3

Define two PythonOperator tasks

Import PythonOperator from airflow.operators.python. Define two tasks named task1 and task2 using PythonOperator with the DAG example_dag. Each task should run a Python function that prints 'Task 1 executed' and 'Task 2 executed' respectively.

Apache Airflow

from airflow import DAG
from datetime import datetime
from airflow.operators.python import PythonOperator

# Define functions task1_func and task2_func that print messages
# Define task1 and task2 as PythonOperator tasks using example_dag
# Your code here

Hint

Define Python functions first, then create PythonOperator tasks with task_id and python_callable.

4

Print the DAG run status

Add a print statement that outputs 'DAG example_dag is ready to run' to confirm the DAG setup.

Apache Airflow

from airflow import DAG
from datetime import datetime
from airflow.operators.python import PythonOperator

# Print confirmation message
# Your code here

Hint

Use print() to show the message confirming DAG setup.

Practice

1. What is the primary purpose of the Airflow UI?

easy

A. To store data files for workflows

B. To write Python code for tasks

C. To monitor and manage workflows visually

D. To configure network settings

Solution

1. Step 1: Understand Airflow UI role

   The Airflow UI is designed as a web interface to help users see and control workflows easily.

2. Step 2: Identify main function

   It shows task status, logs, and workflow structure, making monitoring and management simple.

3. Final Answer:

   To monitor and manage workflows visually -> Option C

4. Quick Check:

   Airflow UI = Monitor & Manage Workflows [OK]

Hint: Airflow UI is for watching workflows, not coding [OK]

Common Mistakes:

• Confusing UI with coding environment
• Thinking UI stores data files
• Assuming UI manages network

2. Which section in the Airflow UI shows the status of each task in a workflow?

easy

A. Admin panel

B. DAGs tab

C. Logs tab

D. Graph view

Solution

1. Step 1: Identify UI sections

   The Graph view in Airflow UI visually displays tasks and their statuses in a workflow.

2. Step 2: Compare other options

   DAGs tab lists workflows, Logs tab shows task logs, Admin panel manages users/settings.

3. Final Answer:

   Graph view -> Option D

4. Quick Check:

   Task status = Graph view [OK]

Hint: Task status is shown in Graph view, not Logs [OK]

Common Mistakes:

• Confusing Logs tab with task status
• Thinking DAGs tab shows task details
• Assuming Admin panel shows task status

3. If a task in Airflow UI shows a red color in the Graph view, what does it mean?

medium

A. Task failed

B. Task is running

C. Task succeeded

D. Task is skipped

Solution

1. Step 1: Understand color codes in Graph view

   In Airflow UI, red color indicates a failed task.

2. Step 2: Recall other colors meaning

   Green means success, yellow means running, grey means skipped.

3. Final Answer:

   Task failed -> Option A

4. Quick Check:

   Red = Failed task [OK]

Hint: Red means failure in Airflow UI Graph view [OK]

Common Mistakes:

• Mixing red with running or success
• Confusing skipped with failed
• Ignoring color legend in UI

4. You notice that the Logs tab in Airflow UI shows no logs for a task that ran. What is the likely cause?

medium

A. Task never started or failed before logging

B. Logs tab is broken and needs UI restart

C. Task succeeded without logs

D. Logs are stored in a different UI section

Solution

1. Step 1: Analyze missing logs scenario

   If no logs appear, usually the task did not start or failed before producing logs.

2. Step 2: Eliminate other options

   Logs tab usually works; success tasks still have logs; logs are not in other UI sections.

3. Final Answer:

   Task never started or failed before logging -> Option A

4. Quick Check:

   No logs = Task not started or early failure [OK]

Hint: No logs usually means task didn't start properly [OK]

Common Mistakes:

• Assuming UI bug without checking task status
• Thinking success tasks have no logs
• Looking for logs in wrong UI tabs

5. You want to quickly check the order of tasks and their dependencies in a DAG using the Airflow UI. Which view should you use and why?

hard

A. Tree view, because it shows tasks in a timeline with dependencies

B. Graph view, because it visually displays tasks and their dependencies

C. Calendar view, because it shows task schedules

D. Code view, because it shows the Python code defining tasks

Solution

1. Step 1: Identify views for task order and dependencies

   The Graph view visually shows tasks as nodes connected by arrows representing dependencies.

2. Step 2: Compare other views

   Tree view shows task runs over time, Code view shows code but not visual dependencies, Calendar shows schedules.

3. Final Answer:

   Graph view, because it visually displays tasks and their dependencies -> Option B

4. Quick Check:

   Task order & dependencies = Graph view [OK]

Hint: Use Graph view to see task dependencies clearly [OK]

Common Mistakes:

• Confusing Tree view with dependency graph
• Looking at code instead of visual graph
• Using Calendar view for dependencies