Why production Airflow needs careful setup - Why It Works

Introduction

Airflow helps run tasks automatically on a schedule. In production, it needs careful setup to avoid failures and keep workflows running smoothly.

When you want to run data pipelines reliably every day without manual work

When multiple people or teams share the same Airflow system and need clear task management

When your workflows depend on each other and must run in a specific order

When you want to monitor task success and get alerts if something breaks

When you need to scale Airflow to handle many tasks at once without slowing down

Commands

Initializes the Airflow database to store task and workflow metadata. This is the first step to prepare Airflow for use.

Terminal

airflow db init

Expected Output

DB: Initialized the metadata database

Creates an admin user to access the Airflow web interface securely.

Terminal

airflow users create --username admin --firstname Admin --lastname User --role Admin --email [e-mail] --password admin123

Expected Output

User created successfully

→

--role - Assigns the user role for permissions

→

--password - Sets the login password

Starts the Airflow webserver on port 8080 so you can view and manage workflows in a browser.

Terminal

airflow webserver --port 8080

Expected Output

[2024-06-01 12:00:00,000] {webserver.py:123} INFO - Starting webserver on port 8080

→

--port - Specifies the port for the webserver

Starts the scheduler that triggers tasks based on their schedule and dependencies.

Terminal

airflow scheduler

Expected Output

[2024-06-01 12:00:05,000] {scheduler.py:456} INFO - Scheduler started

Lists all tasks in the example_dag to verify the workflow is loaded correctly.

Terminal

airflow tasks list example_dag

Expected Output

task_1

task_2

task_3

Key Concept

If you remember nothing else from this pattern, remember: production Airflow needs proper database setup, user management, and running both webserver and scheduler to work reliably.

Common Mistakes

Skipping database initialization before starting Airflow

Airflow cannot store task states or schedules without the database, so it will fail to run workflows.

Always run 'airflow db init' before starting the webserver or scheduler.

Running only the webserver without the scheduler

The webserver shows the UI but does not trigger any tasks, so workflows will not run automatically.

Run both 'airflow webserver' and 'airflow scheduler' to have a working system.

Not creating users and leaving the web interface open

Without user accounts, anyone can access and change workflows, risking security and accidental errors.

Create admin and user accounts with strong passwords before exposing Airflow.

Summary

Initialize the Airflow database to store workflow data.

Create users to secure access to the Airflow web interface.

Run both the webserver and scheduler to manage and execute workflows.

Verify tasks are loaded correctly to ensure workflows are ready.