Installing Airflow locally - Try It Yourself

Installing Airflow locally

📖 Scenario: You want to run Apache Airflow on your own computer to practice scheduling and managing workflows. This project will guide you step-by-step to install Airflow locally using the recommended commands.

🎯 Goal: By the end, you will have Apache Airflow installed and ready to use on your local machine.

📋 What You'll Learn

Have Python 3.7 or higher installed

Have pip installed

Use a terminal or command prompt

💡 Why This Matters

🌍 Real World

Developers and data engineers often install Airflow locally to test and develop workflow automation before deploying to production.

💼 Career

Knowing how to install and run Airflow is a key skill for roles involving data pipeline management and workflow orchestration.

1

Create a new directory for Airflow

Open your terminal and create a new directory called airflow_home using the command mkdir airflow_home. Then change into this directory using cd airflow_home.

Apache Airflow

# Use mkdir airflow_home and cd airflow_home commands here

Hint

Think of this like making a new folder on your computer and then opening it.

2

Set the Airflow home environment variable

Set the environment variable AIRFLOW_HOME to the current directory by running export AIRFLOW_HOME=$(pwd) on Linux/macOS or set AIRFLOW_HOME=%cd% on Windows.

Apache Airflow

mkdir airflow_home
cd airflow_home
# Set AIRFLOW_HOME environment variable here

Hint

This tells Airflow where to store its files, like setting a workspace.

3

Install Apache Airflow using pip

Run the command pip install apache-airflow==2.7.1 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.7.1/constraints-3.7.txt to install Airflow version 2.7.1 with the correct dependencies.

Apache Airflow

mkdir airflow_home
cd airflow_home
export AIRFLOW_HOME=$(pwd)
# Install Apache Airflow with pip here

Hint

This command installs Airflow and ensures all needed packages match the version.

4

Initialize the Airflow database and start the webserver

Run airflow db init to initialize the Airflow database. Then start the Airflow webserver by running airflow webserver --port 8080. This will start the Airflow interface on your local machine.

Apache Airflow

mkdir airflow_home
cd airflow_home
export AIRFLOW_HOME=$(pwd)
pip install apache-airflow==2.7.1 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.7.1/constraints-3.7.txt
# Initialize Airflow DB and start webserver here

Hint

Initializing the database sets up Airflow's internal storage. Starting the webserver lets you see Airflow in your browser.

Practice

1. What is the first step to install Apache Airflow locally on your machine?

easy

A. Create an Airflow user

B. Start the Airflow webserver

C. Create and activate a Python virtual environment

D. Initialize the Airflow database

Solution

1. Step 1: Set up isolated Python environment

   Creating and activating a virtual environment ensures Airflow and its dependencies do not interfere with other Python projects.

2. Step 2: Prepare for Airflow installation

   Once the virtual environment is active, you can safely install Airflow using pip.

3. Final Answer:

   Create and activate a Python virtual environment -> Option C

4. Quick Check:

   First step = virtual environment setup [OK]

Hint: Always start with a virtual environment for Python projects [OK]

Common Mistakes:

• Skipping virtual environment and installing globally
• Trying to start Airflow before installation
• Initializing database before installing Airflow

2. Which command correctly installs Apache Airflow with pip inside a virtual environment?

easy

A. pip install airflow

B. pip install airflow-webserver

C. pip install airflow==latest

D. pip install apache-airflow

Solution

1. Step 1: Use the official package name

   The correct package name for Airflow on PyPI is 'apache-airflow'.

2. Step 2: Avoid incorrect or incomplete package names

   Other options either use wrong names or non-existent packages.

3. Final Answer:

   pip install apache-airflow -> Option D

4. Quick Check:

   Correct package name = apache-airflow [OK]

Hint: Use 'apache-airflow' as the pip package name [OK]

Common Mistakes:

• Using 'airflow' instead of 'apache-airflow'
• Adding invalid version specifiers
• Trying to install non-existent sub-packages

3. After installing Airflow, what is the output of running airflow db init?

medium

A. Initializes the Airflow metadata database

B. Creates a new Airflow user

C. Starts the Airflow webserver

D. Displays Airflow version information

Solution

1. Step 1: Understand the command purpose

   The command 'airflow db init' sets up the metadata database needed for Airflow to track workflows and tasks.

2. Step 2: Differentiate from other commands

   Starting the webserver, creating users, or showing version info are done with different commands.

3. Final Answer:

   Initializes the Airflow metadata database -> Option A

4. Quick Check:

   'airflow db init' = database initialization [OK]

Hint: Init command sets up Airflow's database, not the server [OK]

Common Mistakes:

• Confusing db init with webserver start
• Expecting user creation from db init
• Thinking it shows version info

4. You run airflow webserver but get an error saying the database is not initialized. What should you do to fix this?

medium

A. Run airflow db init to initialize the database

B. Create a new Airflow user first

C. Restart the machine and try again

D. Install Airflow again using pip

Solution

1. Step 1: Identify the cause of the error

   The error indicates the Airflow metadata database is missing or not set up.

2. Step 2: Initialize the database

   Running 'airflow db init' creates the necessary database tables and schema.

3. Final Answer:

   Run airflow db init to initialize the database -> Option A

4. Quick Check:

   Database error fixed by db init command [OK]

Hint: Initialize database before starting webserver [OK]

Common Mistakes:

• Skipping database initialization
• Trying to create user before db init
• Restarting machine without fixing db

5. Which sequence of commands correctly installs Airflow, initializes the database, creates an admin user, and starts the webserver and scheduler locally?

hard

A. pip install apache-airflow; airflow users create --username admin; airflow db init; airflow webserver; airflow scheduler

B. python -m venv venv; source venv/bin/activate; pip install apache-airflow; airflow db init; airflow users create --username admin --firstname Admin --lastname User --role Admin --email [e-mail] --password admin; airflow webserver & airflow scheduler

C. source venv/bin/activate; airflow db init; pip install apache-airflow; airflow webserver; airflow scheduler

D. python -m venv venv; pip install apache-airflow; airflow webserver; airflow db init; airflow scheduler

Solution

1. Step 1: Set up virtual environment and install Airflow

   Create and activate a virtual environment, then install Airflow with pip.

2. Step 2: Initialize database and create admin user

   Run 'airflow db init' to prepare the database, then create an admin user with the correct command.

3. Step 3: Start Airflow services

   Start the webserver and scheduler to run Airflow locally.

4. Final Answer:

   python -m venv venv; source venv/bin/activate; pip install apache-airflow; airflow db init; airflow users create --username admin --firstname Admin --lastname User --role Admin --email [e-mail] --password admin; airflow webserver & airflow scheduler -> Option B

5. Quick Check:

   Correct full setup sequence = python -m venv venv; source venv/bin/activate; pip install apache-airflow; airflow db init; airflow users create --username admin --firstname Admin --lastname User --role Admin --email [e-mail] --password admin; airflow webserver & airflow scheduler [OK]

Hint: Follow install, init, user create, then start services [OK]

Common Mistakes:

• Installing Airflow after db init
• Starting webserver before creating user
• Not activating virtual environment