Airflow UI overview - Mini Project: Build & Apply

Airflow UI Overview

📖 Scenario: You are a data engineer starting to use Apache Airflow to manage workflows. You want to understand how to interact with Airflow's web interface to monitor and control your workflows easily.

🎯 Goal: Build a simple Airflow DAG and learn how to navigate the Airflow UI to see DAG status, task details, and logs.

📋 What You'll Learn

Create a basic Airflow DAG with two tasks

Set a DAG configuration variable for scheduling

Use Airflow's PythonOperator to define tasks

Print the DAG run status in the Airflow UI

💡 Why This Matters

🌍 Real World

Airflow is widely used to automate and monitor workflows in data engineering and software projects. Understanding the UI helps you track and troubleshoot your workflows easily.

💼 Career

Knowing how to create DAGs and use the Airflow UI is essential for roles like Data Engineer, DevOps Engineer, and Workflow Automation Specialist.

1

Create a basic Airflow DAG

Create a Python file with a DAG named example_dag using Airflow's DAG class. Set start_date to 2024-01-01 and schedule_interval to @daily. Import necessary modules from airflow and datetime.

Apache Airflow

# Import DAG and datetime modules
# Create a DAG named example_dag with start_date 2024-01-01 and schedule_interval '@daily'
# Your code here

Need a hint?

Use DAG class from airflow and datetime module to set start date.

2

Add a configuration variable for retries

Add a dictionary called default_args with a key 'retries' set to 1. Pass default_args=default_args when creating the DAG example_dag.

Apache Airflow

from airflow import DAG
from datetime import datetime

# Create default_args dictionary with retries set to 1
# Your code here

dag = DAG(
    'example_dag',
    start_date=datetime(2024, 1, 1),
    schedule_interval='@daily',
    default_args=default_args
)

Need a hint?

Define default_args before the DAG and pass it as a parameter.

3

Define two PythonOperator tasks

Import PythonOperator from airflow.operators.python. Define two tasks named task1 and task2 using PythonOperator with the DAG example_dag. Each task should run a Python function that prints 'Task 1 executed' and 'Task 2 executed' respectively.

Apache Airflow

from airflow import DAG
from datetime import datetime
from airflow.operators.python import PythonOperator

# Define functions task1_func and task2_func that print messages
# Define task1 and task2 as PythonOperator tasks using example_dag
# Your code here

Need a hint?

Define Python functions first, then create PythonOperator tasks with task_id and python_callable.

4

Print the DAG run status

Add a print statement that outputs 'DAG example_dag is ready to run' to confirm the DAG setup.

Apache Airflow

from airflow import DAG
from datetime import datetime
from airflow.operators.python import PythonOperator

# Print confirmation message
# Your code here

Need a hint?

Use print() to show the message confirming DAG setup.