Installing Airflow locally - Step-by-Step CLI Walkthrough

Introduction

Airflow helps you schedule and run tasks automatically. Installing it locally lets you try it out and build workflows on your own computer without needing a server.

When you want to learn how Airflow works before using it on a real server

When you need to test your task workflows quickly on your laptop

When you want to develop and debug your data pipelines without internet

When you want to experiment with scheduling tasks without affecting others

When you want to run simple automation jobs on your own machine

Commands

Create a clean virtual environment to keep Airflow and its packages separate from other Python projects.

Terminal

python3 -m venv airflow_env

Expected Output

No output (command runs silently)

Activate the virtual environment so that Python commands use the isolated packages inside it.

Terminal

source airflow_env/bin/activate

Expected Output

No output (command runs silently)

Install Airflow version 2.7.1 with the exact package versions it needs for Python 3.8 to avoid compatibility issues.

Terminal

pip install apache-airflow==2.7.1 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.7.1/constraints-3.8.txt

Expected Output

Collecting apache-airflow==2.7.1

  Downloading apache_airflow-2.7.1-py3-none-any.whl (1.2 MB)

...

Successfully installed apache-airflow-2.7.1 ...

→

==2.7.1 - Specifies the exact Airflow version to install

→

--constraint - Ensures compatible package versions for Airflow

Initialize the Airflow database which stores task and workflow information.

Terminal

airflow db init

Expected Output

INFO  [alembic.runtime.migration] Context impl PostgresImpl.

INFO  [alembic.runtime.migration] Will assume transactional DDL.

INFO  [alembic.runtime.migration] Running upgrade head

INFO  [alembic.runtime.migration] Upgrade successful

Create an admin user to log into the Airflow web interface.

Terminal

airflow users create --username admin --firstname Admin --lastname User --role Admin --email [e-mail] --password admin

Expected Output

User 'admin' created successfully

→

--role Admin - Gives full access to Airflow features

Start the Airflow web interface on port 8080 so you can open it in your browser.

Terminal

airflow webserver --port 8080

Expected Output

[2024-06-01 12:00:00,000] {webserver.py:123} INFO - Starting web server on port 8080

[2024-06-01 12:00:00,100] {webserver.py:456} INFO - Web server started

→

--port 8080 - Sets the webserver to listen on port 8080

Start the scheduler that runs your tasks at the right times.

Terminal

airflow scheduler

Expected Output

[2024-06-01 12:00:05,000] {scheduler.py:789} INFO - Starting scheduler

[2024-06-01 12:00:05,100] {scheduler.py:790} INFO - Scheduler started

Key Concept

If you remember nothing else, remember: Airflow needs a database initialized and a user created before you can use its web interface and scheduler.

Common Mistakes

Skipping the virtual environment and installing Airflow globally

This can cause package conflicts with other Python projects or system tools.

Always create and activate a virtual environment before installing Airflow.

Not using the constraints file when installing Airflow

Airflow depends on specific package versions; ignoring constraints can cause errors or crashes.

Use the --constraint flag with the official constraints file matching your Airflow version and Python version.

Trying to access the web interface before creating a user

Airflow requires a user to log in; without one, you cannot access the UI.

Run the airflow users create command to add an admin user before starting the webserver.

Summary

Create and activate a Python virtual environment to isolate Airflow installation.

Install Airflow with the correct version and constraints to avoid package conflicts.

Initialize the Airflow database and create an admin user for access.

Start the Airflow webserver and scheduler to run and monitor workflows locally.

Practice

1. What is the first step to install Apache Airflow locally on your machine?

easy

A. Create an Airflow user

B. Start the Airflow webserver

C. Create and activate a Python virtual environment

D. Initialize the Airflow database

Solution

1. Step 1: Set up isolated Python environment

   Creating and activating a virtual environment ensures Airflow and its dependencies do not interfere with other Python projects.

2. Step 2: Prepare for Airflow installation

   Once the virtual environment is active, you can safely install Airflow using pip.

3. Final Answer:

   Create and activate a Python virtual environment -> Option C

4. Quick Check:

   First step = virtual environment setup [OK]

Hint: Always start with a virtual environment for Python projects [OK]

Common Mistakes:

• Skipping virtual environment and installing globally
• Trying to start Airflow before installation
• Initializing database before installing Airflow

2. Which command correctly installs Apache Airflow with pip inside a virtual environment?

easy

A. pip install airflow

B. pip install airflow-webserver

C. pip install airflow==latest

D. pip install apache-airflow

Solution

1. Step 1: Use the official package name

   The correct package name for Airflow on PyPI is 'apache-airflow'.

2. Step 2: Avoid incorrect or incomplete package names

   Other options either use wrong names or non-existent packages.

3. Final Answer:

   pip install apache-airflow -> Option D

4. Quick Check:

   Correct package name = apache-airflow [OK]

Hint: Use 'apache-airflow' as the pip package name [OK]

Common Mistakes:

• Using 'airflow' instead of 'apache-airflow'
• Adding invalid version specifiers
• Trying to install non-existent sub-packages

3. After installing Airflow, what is the output of running airflow db init?

medium

A. Initializes the Airflow metadata database

B. Creates a new Airflow user

C. Starts the Airflow webserver

D. Displays Airflow version information

Solution

1. Step 1: Understand the command purpose

   The command 'airflow db init' sets up the metadata database needed for Airflow to track workflows and tasks.

2. Step 2: Differentiate from other commands

   Starting the webserver, creating users, or showing version info are done with different commands.

3. Final Answer:

   Initializes the Airflow metadata database -> Option A

4. Quick Check:

   'airflow db init' = database initialization [OK]

Hint: Init command sets up Airflow's database, not the server [OK]

Common Mistakes:

• Confusing db init with webserver start
• Expecting user creation from db init
• Thinking it shows version info

4. You run airflow webserver but get an error saying the database is not initialized. What should you do to fix this?

medium

A. Run airflow db init to initialize the database

B. Create a new Airflow user first

C. Restart the machine and try again

D. Install Airflow again using pip

Solution

1. Step 1: Identify the cause of the error

   The error indicates the Airflow metadata database is missing or not set up.

2. Step 2: Initialize the database

   Running 'airflow db init' creates the necessary database tables and schema.

3. Final Answer:

   Run airflow db init to initialize the database -> Option A

4. Quick Check:

   Database error fixed by db init command [OK]

Hint: Initialize database before starting webserver [OK]

Common Mistakes:

• Skipping database initialization
• Trying to create user before db init
• Restarting machine without fixing db

5. Which sequence of commands correctly installs Airflow, initializes the database, creates an admin user, and starts the webserver and scheduler locally?

hard

A. pip install apache-airflow; airflow users create --username admin; airflow db init; airflow webserver; airflow scheduler

B. python -m venv venv; source venv/bin/activate; pip install apache-airflow; airflow db init; airflow users create --username admin --firstname Admin --lastname User --role Admin --email [e-mail] --password admin; airflow webserver & airflow scheduler

C. source venv/bin/activate; airflow db init; pip install apache-airflow; airflow webserver; airflow scheduler

D. python -m venv venv; pip install apache-airflow; airflow webserver; airflow db init; airflow scheduler

Solution

1. Step 1: Set up virtual environment and install Airflow

   Create and activate a virtual environment, then install Airflow with pip.

2. Step 2: Initialize database and create admin user

   Run 'airflow db init' to prepare the database, then create an admin user with the correct command.

3. Step 3: Start Airflow services

   Start the webserver and scheduler to run Airflow locally.

4. Final Answer:

   python -m venv venv; source venv/bin/activate; pip install apache-airflow; airflow db init; airflow users create --username admin --firstname Admin --lastname User --role Admin --email [e-mail] --password admin; airflow webserver & airflow scheduler -> Option B

5. Quick Check:

   Correct full setup sequence = python -m venv venv; source venv/bin/activate; pip install apache-airflow; airflow db init; airflow users create --username admin --firstname Admin --lastname User --role Admin --email [e-mail] --password admin; airflow webserver & airflow scheduler [OK]

Hint: Follow install, init, user create, then start services [OK]

Common Mistakes:

• Installing Airflow after db init
• Starting webserver before creating user
• Not activating virtual environment